07.08.09

Public service announcement: name test files descriptively, not just with bug numbers

Mozilla’s tests are spread through a large number of directories. One of these directories, layout/reftests/bugs, contains a thousand or so reftests. Each reftest file is named by the number of the corresponding bug: for example, 169749-1.html and 169749-1-ref.html. This name-the-test-by-the-bug pattern is not specific to this particular directory, and if you skim through the directories in layout/reftests you’ll find numerous other tests named this way.

The problem? These names are effectively useless for determining the extent of test coverage. (Assume arguendo that all these files were properly located in directories specific to a feature being tested, for example in a hypothetical layout/reftests/list-item directory for the two aforementioned tests, to make it reasonable to scan the coverage of a particular feature.) If I want to know what corner cases are being tested, I have to read each test and corresponding bug individually, a tedious task.

The solution? Name your test files descriptively. For example, in the above case, a better name might perhaps be marker-position-for-list-item-containing-block-with-margins.html.

Okay, this will make it easier for random readers at unknown points in the future, but is there anything in it for you, the test author? There is: you may endear yourself to reviewers. For example, dbaron decided, based upon the names I gave to test files in the recent patch I wrote to implement background-size (and upon a handful of comments in the list of tests itself), that my patch’s tests were adequate and didn’t need a long, tedious review of each one:

The reftest.list looks good based on the names of the tests and the
comments; I don't see the need to review the tests themselves.

In conclusion: DON’T name your test files based upon bug numbers, and DO name them based upon the particular scenario that they test. Readers will and reviewers may thank you.