It's so annoying to see developers calling failing tests "flaky".
Last week I found a factory that assigned a random age to a user instance, so no wonder a test that used the age in some bit of logic failed sometimes. That's not a "flaky test", that's a bug in the factory.
Today I saw a date parser bug be called a "flaky test". It was written on the 21st of April and worked fine until it began to fail today, the 1st of May.
@zenspider@gd inability to repro definitely adds a dimension of difficulty. In some ways I found "briefly hidden" flaky test bugs more frustrating because they were fairly easy to reproduce, but led people to just rerun the tests a few times until they pass, rather than fix the underlying bugs.
Add comment