@anderseknert I would look at the runtime of my unit tests as a function of the size of the codebase. Unit tests running for 1 hour for a single-page app that plays Solitaire and nothing else? Probably would look into improving that runtime. Unit tests running for 1 hour for a fully featured online banking platform? Maybe that's to be expected, carry on.
@anderseknert 20 seconds, but that is only for unit tests that could possibly fail based on the code that changed. If I do a full rebuild (which CI does every time) tests should take 5-10 minutes (we are a multi-repo org - which has pros and cons but overall I think is better than single repo for us) - anything else means the repo is too small and so overhead of working with another repo is too high, anything more and the repo is too large and should be split because of the overhead of a large repo is too high.
I don't really believe in a distinction between unit and integration tests. However I do see value in long test which I would not run as part of my normal TDD cycle, instead I would let my CI system deal with them. I have a dozen different builds on my CI system (embedded system where we target several different CPUs, plus we run things like clang-tidy on CI that developers don't normally run locally)
@anderseknert This is something where I do think there is a right answer actually... around 2 mins. Anything longer should either be refactored into smaller, more granular tests or the whole test should be moved to a less frequently run stage downstream.
There are always exceptions to the rule but this is a good guideline in my experience
@elejia Yeah, it's happened to me on a few occasions that a slow running test suite led to some great refactoring, and even that parts of a project spun off into separate libraries, which of course could be tested independently. Creating libraries from commonly used code is often a good way to force one to consider public interfaces vs. internal, and a bunch of other nice side effects (like the aforementioned test suite run time)
@anderseknert of course! i would also add, that acceptable should probably be considered together with feasible. if achieving faster tests is feasible, then having slow tests is unacceptable 😅
@anderseknert in my Erlang projects, I'm happy with somewhere under 5 minutes for unit tests. For integration or end-to-end tests, maybe 20-25 minutes.
@anderseknert for inner-loop while programming, probably <30s at the most. I do enjoy liberal use of tagging tests that are slow. testing.Short in golang, bazel's native "test timeout" tagging, or whatever works with the toolchain at hand.
And on pre-commit, or once in a while on-demand, run the entire test suite.
Also, all manner of caching to only run tests affected by changed code helps tremendously. gotestsum makes me happy for golang projects
@anderseknert asfaik there's no more to it that the ability to (manually) tag up tests as "short or not" :D. I'm guessing there's more to the story internally at goog.
The only similar way I know of tagging tests in Go is to use build tags, which works alright for separating integration tests and messier stuff like that.
@anderseknert I don’t think I’ve ever worked somewhere where tests took less than 10–20 minutes or similar, so I have no idea what would be reasonable to expect that’s lower than that.
Faster would obviously be preferable but it’s impossible to say this is unacceptable.
If it was hours, then that would definitely be too much.
@benjamineskola Did you never start on a greenfield project? It's interesting doing that, and see how all the things you'd normally curse in legacy projects eventually grow in your own creation. Long test times certainly included!
@anderseknert Not at work, no — personal projects yes, but they’re all much smaller of course. And I can definitely imagine it would mean either making all the same mistakes, or, by avoiding them, making a whole new different set of mistakes.
@benjamineskola Haha, yes, that. But they're at least initially "interesting" mistakes. Not so much once the novelty's worn off. It's a humbling experience though.
@anderseknert I've worked on systems that took hours to build completely (roughly 15 minute change one file compile link cycle), and we had test suites that ran for tens of hours. And were super helpful (eg: replay keystrokes while shuffling memory with every allocator call).
I love fast turnaround, but it's fine to have tests that take hours or days...
@anderseknert it would be interesting to see variation by language/platform. I know that the C++ folks I used to work with were more tolerant of long-ish build/test cycles.
@rogerlipscombe Totally! Once a project grows beyond a certain size, I normally just run the tests related to my changes locally, and the full test suite only in CI. Mostly hacking on Golang, and the tests are generally fast, but there are lots of them and I'm impatient 😅
Add comment