Comments on: Public service announcement: name test files descriptively, not just with bug numbers

By: David Baron

David Baron — Sun, 09 Aug 2009 17:43:04 +0000

I think one practices is right in some cases, and the other in others.

In particular, if you’re trying to write a thorough set of tests for a feature, use descriptive names.

But if you’re landing a testcase for a particular bug that’s not part of a set of thorough tests, just to make sure that one particular case doesn’t regress, use the bug number, since a “named” test could mislead people into thinking whatever was described by that test name was actually being tested adequately.

[In the latter case it seems to me an appropriate amount of vagueness is in order. Enough of a hint from the name will point the way for someone interested in determining whether something's been tested, even if it doesn't identify exactly what's being tested. Having a handful of files with plausible names is better than having the same handful with just numbers to go by.]

By: Boris

Boris — Sun, 09 Aug 2009 17:12:00 +0000

Having “what the test tests” in the name doesn’t help that much with coverage either, necessarily… and it not feasible anyway for tests that test more than one thing (most of them, actually). Using a non-numbered name also means you have to figure out _where_ to put the test, unless you think we should dump random non-numbered names into bugs/.

Having the bug# is nice just in terms of figuring out whether a particular bug has a test (since some people aren’t so good at setting in-testsuite flags).

It’s also nice in terms of figuring out what the test is _trying_ to test, if you discover that the test is bogus in the course of working on some patch (happens reasonably often, sadly).

In the end, neither one provides all the metadata one wants for a test, but putting all that metadata into the test increases the test-writing burden tremendously. Not sure there are any good solutions here.

By: Robert Kaiser

Robert Kaiser — Sun, 09 Aug 2009 13:24:31 +0000

A descriptive name is helpful also in case of determining what a failure might be about, but a bug number is _very_ helpful in speedily finding the history of what a test was introduced for – IMHO, both should be in there, but too long filenames should also be avoided as I think they still give headaches with some filesystems (e.g. possibly FAT32).

[Last I recall, FAT32 didn't have problems with pretty generous path lengths, but I might be mistaken.]

By: bc

bc — Sat, 08 Aug 2009 09:33:50 +0000

I agree with Adam that retaining the bug number in the test file name is good practice. Coming up with a common format for bug number and test description in the test file name is also important. Consider the use case of taking a list of tests and determining the list of bugs where the tests were created. As long as you keep that simple, adding more descriptive text to the file name is fine.

By: Mossop

Mossop — Sat, 08 Aug 2009 09:27:30 +0000

I’ve seen long-named tests in the tree and they’ve bugged me everytime because it is harder to identify the test in the list (because frequently in the directory view the end is chopped off) and even then they tend to be pretty obscure because they try to keep the length shorter.

Also I really hope we aren’t judging whether we have good test coverage on things based on the naming alone.

Instead I’d be more inclined to have a README or something in the directory that includes a short description of what each test covers.

[No serious judgment, but it's great for a birds-eye view. A README could work, particularly since MXR might display it (or only some of it?), but that requires some duplication of information. (Comments in the test itself, I expect, would be less duplicative because they are likely to be more of an explanatory flavor than a descriptive flavor.)]

By: mnandigama

mnandigama — Sat, 08 Aug 2009 06:32:17 +0000

I concur with your reply. But at the end of the day, one way or the other, we should be able to understand the purpose of the test with out reading the entire test code for each test case. Be it annotations or descriptive names.

By: mnandigama

mnandigama — Sat, 08 Aug 2009 06:21:00 +0000

I respectfully disagree with your suggestion. The test case functionality has to be annotated in the test case in the form of a comment instead of trying to cram it into the test case name itself. Consider testing scenarios where we end up with very deep nested directory structures coupled with long test names and end up failing running tests because of a long path problem. A test case name having a bug in it serves a specific purpose. It acts as a regression test case. If a bug number named test case fails , we know that it is causing a regression just by looking at the list of failed tests log. Naming test to indicate the purpose and annotating the bug for which the test case is written inside or vice-versa serves same functional purpose if suitable parsers are written to grep for annotations or test case names. However, bug numbered tests provide shorter path lengths while executing tests for fennec, just for example. [I'm not suggesting it be described exclusively by the test name; a comment in the test is also well worth doing. Further, as Adam notes, description and number are not exclusive. By the way, you can't assume a test is a regression just because it's a number; I'm sure I could point to many testcases named with numbers which are not themselves regressions. I don't buy shorter path lengths as a valid goal. I don't think we should hack around deficient platforms that don't support long paths by handicapping ourselves in the source tree. If that's really a concern, there could be a renaming step included in the test-execution process to specifically address that platform.]

By: Adam Hauner

Adam Hauner — Sat, 08 Aug 2009 06:17:35 +0000

Is anything wrong with names combining both ways? For example:
169749-1-marker-position-for-list-item-containing-block-with-margins.html

[That seems reasonable.]