Benchmarks should be like a scientific paper: they should describe all the choices made and why for the configurations. At least that will show if the people doing it really understand what they’re comparing.
Have you ever read a paper? You can consider yourself lucky if they have error bars and repeated their measurements more than once. The quality of “benchmarking papers” is comically bad (on average).
Add comment