why we built an e2e testing tool on top of Playwright

When we started Octomind, we wanted to build a better way of testing for ourselves and our community. Our ideal end-to-end testing tool would achieve three things:

1. Help build high-quality web apps, fast
2. Eliminate the pain of writing and maintaining e2e tests
3. Use AI to make it all happen

Right at the beginning, we had to make a critical choice - we could either write a new test automation framework or build our tooling on top of an established e2e testing library. 

Wasting developer hours for building a framework while there are open source options on the market that have broad adoption sounded like a serious case of reinventing the wheel.

We decided to spend our time and energy on what we do best: AI.

Microsoft launched Playwright in January 2020 as an “End-to-end testing tool for modern websites”. It has become a major player in E2E testing in this short time: its GitHub repository has 59.2k stars and 3.3k forks as of February 2024, while weekly NPM downloads average 3.3 million. These numbers surpass Cypress, the other leader in E2E testing and Selenium, the OG test automation framework.

‍

Despite its newness and relatively small community, Playwright’s evangelists were loud enough to trigger an avalanche. Drop by any dev forum and you’ll find lively discussions on whether Playwright is better than Cypress (mostly yes) or whether to migrate from Cypress to Playwright (not necessarily). One area of consensus? Playwright is the best choice for new projects, for reasons I’ll get into later.

Playwright is supported by the developer community. Playwright is open source and free to use (so far). And an important point for us - Playwright is here to stay.

‍

"With Playwright being owned by Microsoft, there are still unknowns but when I look at the tooling they have built and provided for the developer community, I would put my bet on Microsoft. When I look at Typescript, VS Code, .Net Core, etc, I don't have many fears about the future of Playwright.”

Butch Mayhew

But at the end of the day, the technology needs to live up to the hype. Here are some other reasons we chose Playwright:

• State-of-the-art testing for modern browsers: Playwright supports today’s browser architecture and user behaviors, allowing testing across domains, browser tabs, origins, and users. You can also include multiple test scenarios in a single test, decreasing testing time.
• Cross-everything support: It offers cross-browser testing (all modern rendering engines including Chromium, WebKit, and Firefox), cross-platform testing (Windows, Linux, and macOS, locally or on CI, headless or headed), and cross-language support (TypeScript, JavaScript, Python, .NET, and Java). 
• Speed: The last thing anyone wants to do is waiting for their test suites to run. Playwright offers fast test execution speeds plus parallel testing.
• Flakiness prevention: Playwright offers resiliency-building features like auto-wait, web-first assertions, and tracing to fight flaky tests. Still not on the level we’d like to have it, but it has some tricks to increase the confidence in the test outcomes.
• Playwright trace viewer - a GUI tool, which lets you play back visual recordings of your tests to pinpoint where specific interactions occurred. A functionality we adopted in our test reports almost from the beginning.

We are big Playwright believers, but we’ve built our tooling in a way that makes it possible to expand the framework coverage in the future. If our users and the market need us to. 

‍

Every ever so good technology can find its challenger. Software engineers testing their apps are also not a monolith, they have different needs and preferences. Different frameworks fit different use cases. That’s why we want to keep real Octomind open to a real possibility of working on top of other testing frameworks.