Monday, Jul 7, 2025

Automated Software Testing: Episode 2
So, you’ve written a few tests. Maybe even built a little test suite. The CI is green, the report looks decent, and everything feels like it’s heading in the right direction. Until it isn’t. A test fails randomly. Another breaks because the app changed slightly. Suddenly, you find yourself debugging tests more than the code they’re supposed to test. Welcome to the real world of test automation.
This is where automated testing gets real. It’s no longer just about writing tests — it’s about writing the right ones. Tests that are useful. Tests that catch bugs before your users do.
What to Automate (and What Not To)
Not everything needs a test.
Your login flow? Absolutely. The core business logic? Hell yes. But testing whether a button is blue or green? Probably not worth your time. Focus on what’s critical to the user experience and business value.
Here’s a rough guide I go by:
- High-risk = High-priority. Payment flows, user authentication, data manipulation.
- Frequently used paths. Anything often touched by users deserves protection.
- Things that broke before. Past bugs make great test cases.
- Brittle UI tweaks? Probably not. Snapshot testing has its place, but don’t overdo it.
Think of your test suite like a garden. You can’t plant everything, so pick the stuff that feeds your application.
Write Tests That Actually Help
A good test should do three things:
- Tell you what’s broken.
- Fail for the right reasons.
- Be easy to fix or update.
If your test breaks because your locator used div[3]
and the frontend team refactored the layout, that’s not useful. Use stable selectors, meaningful test data, and make assertions on behavior — not implementation.
Don’t assert for the sake of asserting. The more assertions you have, the harder the test is to read and maintain. Keep them scoped and purposeful.
Lessons from the Land of Flakes
Ah, flaky tests. The ghosts in the CI. The rerun lottery. The thing that makes engineers scream into the void.
Here’s what I learned the hard way:
- Time-based waits are your enemy. If your test says
sleep 5
to wait for a modal, you’re just praying it shows up in time. Use proper wait conditions. - External dependencies suck. That third-party service you’re hitting? It will go down during your deploy. Stub it out in tests.
- Your CI environment is not your local machine. Things are slower, noisier, and less predictable. Design your tests with that in mind.
You’ll also need to build habits:
- Rerun flaky tests locally. Reproduce it. Tweak it. Figure out what it’s trying to tell you.
- Don’t accept flaky tests as normal. Comment them out if you must, but don’t let them stay flaky.
- Invest in test tooling. Retry logic, better logs, visual test runners — it all adds up.
Maintenance Is a Feature
Tests are not write-once, run-forever. They evolve with your code. If you’re not maintaining your test suite, it’s going to rot.
Some things I now do regularly:
- Audit tests every sprint. What failed? What’s slow? What’s still useful?
- Tag and group your tests. This lets you run smoke tests quickly or isolate long runners.
- Refactor tests like you refactor code. When they fail, clean them up.
Closing Thoughts
Sometimes you won’t be able to make stable locators. Sometimes tests will run a minute or two longer than you’d like. Some tests will be painfully flaky and you won’t see a clear path to deflake them — and that’s okay.
Writing automated tests that actually help takes time and intention. Don’t aim for perfection. Aim for usefulness.
It’s okay to have some manual steps. It’s okay to delete a test that’s more pain than gain. The goal is not to automate everything — it’s to automate well.
In the next episode, we’ll go into:
- The tools and frameworks I’ve tried (and which ones I stuck with)
- My test environment setup and what makes it fast and reliable
- Why observability matters for test failures
Until then: keep your tests lean, your assertions sharp, and your CI logs clean.
You’re still figuring it out one bug at a time — and that’s exactly how it should be.
As a small bonus for those who want to know more about writing robust code in general, here’s a warm invitation to explore the Tiger Style philosophy.