You ever get to the end of running benchmarks, maybe a long running one, and realize… “Oh no. I forgot to set that important option, and these results are useless”
Yeah. I have. Too many times.
So I’ve added --benchmark-mode and --strict-benchmark-mode to SpiderMonkey.
These options configure the shell for benchmarking, taking the wisdom of the team and boiling multiple shell options down to a single --benchmark-mode flag, and in --strict-benchmark-mode will abort the run if the shell is configured in a way where effective benchmarking is unlikely to be possible (e.g. benchmarking a debug build!)
The nice thing about nailing this down is that this is something we can point anyone to and know that their shell is following the rules any of us would follow.
The general design philosophy of benchmark mode is to disable things you wouldn’t see enabled in Firefox in normal configuration, as well as debugging code that maybe makes sense for test suites but doesn’t make sense for a benchmark.
Hopefully this is the end of me realizing that I forgot to pass --no-async-stacks yet again.