08.12.11

Party like it’s 1999: <stdint.h> comes to Mozilla!

tl;dr

Need an integer type with guaranteed size? If you’re not defining a cross-file interface, #include "mozilla/StdInt.h"#include "mozilla/StandardInteger.h" and use uint32_t or any other type defined by <stdint.h>. (If you are defining an interface, use PRUint32 and similar — for now.) mozilla/StdInt.hmozilla/StandardInteger.h is a cross-platform implementation of <stdint.h>‘s functionality usable in any code.

Embedders may find that the mozilla/StdInt.hmozilla/StandardInteger.h typedefs conflict with ones they have already been using. To work around this conflict, write a stdint.h compatible with the embedding’s typedefs (more likely: adapt an existing implementation), then set the preprocessor variable MOZ_CUSTOM_STDINT_H to a quoted path to that reimplementation when mozilla/StdInt.hmozilla/StandardInteger.h is included. It may be simplest to add this to command line flags when invoking the compiler.

Fixed-size integer types

Fixed-size integer types are signed or unsigned types with exactly N bits. They contrast with the built-in C and C++ types (char, short, int, &c.) with compiler-dependent sizes. Fixed-size integer types are quite useful:

  • They work well when serializing an object to a sequence of bytes, where the size of a serialized item must be constant for correctness.
  • They minimize memory use in classes and structs, also making padding-based waste more obvious.
  • They work well in cross-platform APIs, eliminating the challenge of implementing correct behavior when types have different sizes across platforms.

Fixed-size integer types are useful in the same way size_t, off_t, and other non-built-in types are: they fit some problem domains better than built-in types.

C99 and C++11 finally standardized fixed-size integer types in <stdint.h> and <cstdint>. They define {u,}int{8,16,32,64}_t types, plus useful constants for their limits (INT8_MIN, INT8_MAX, UINT8_MAX, INT16_MIN, &c.). One would expect projects to quickly use these types, but it didn’t happen.

Old projects predating <stdint.h> have been particularly slow to adopt it. Many such projects already rolled their own non-<stdint.h>-named fixed-size integer types; switching would be a hassle. And not all compilers shipped <stdint.h>: Visual Studio didn’t have it until 2010! (ಠ_ಠ) Projects implementing their own <stdint.h>-compatible types posed another problem, because different projects’ implementations might be incompatible.

New projects fare better, but not always. Sometimes their dependence on old projects anchors them to the old, pre-<stdint.h> world.

Fixed-size integer types in Mozilla

Mozilla sits squarely in the old-project category, facing every rationale noted above for using its own types. These have long been NSPR‘s PR{Ui,I}nt{8,16,32,64} and SpiderMonkey’s {u,}int{8,16,32,64} types. But recently the landscape has changed.

As Mozilla has imported more external code, fixed-size integer types have proliferated. Most imported code uses <stdint.h>, with fallback typedefs for Visual Studio; some code (IPC code from Chromium) defines and uses {u,}int{8,16,32,64}. As type definitions have proliferated, surprising problems have arisen.

The woes of multiplicity

#include-order issues are the simplest problem. For example, the IPC uint32-style definitions are incompatible with SpiderMonkey’s definitions with some compilers. #include IPC and SpiderMonkey headers in the wrong order, and the compiler will error on incompatible typedefs. This problem is easy to diagnose but harder to resolve, and sometimes it causes considerable pain. Mass refactorings that add #includes have fallen afoul of this, with the least bad solution usually being to fix the first error, recompile, and repeat until done (twenty-odd times in one instance).

Worse than #include mis-ordering and conflict are linking problems. int and long could be 32-bit integers yet appear different when linked. Suppose a method taking an int32 argument defined int32 = int during compilation, but a user of it saw int32 = long during compilation. Each alone would compile. But beneath typedefs they’d be incompatible and wouldn’t link.

As we’ve imported more code in Mozilla, more and more developers have been bitten by these problems. We’ve reached a breaking point. We could use PRUint32, JSUint32, and other types which never trample upon each other. Yet no one likes them given the standardized types, and it’s not possible to change “upstream” code to such a scheme. Thus a second solution: use <stdint.h> definitions for everything.

Switching to <stdint.h>

Using the <stdint.h> types in Mozilla code is now as simple as #include "mozilla/StdInt.h"#include "mozilla/StandardInteger.h". mozilla/StdInt.hmozilla/StandardInteger.h implements the <stdint.h> interface even in the edge cases: for compilers not supporting it, and for embedders who can’t use the regular definitions. It works as follows:

  1. If the preprocessor definition MOZ_CUSTOM_STDINT_H is defined, then #include MOZ_CUSTOM_STDINT_H. Embedders who can’t use the default definitions should use this to adapt. (MOZ_CUSTOM_STDINT_H may also be passed into the Mozilla build system using an environment variable. Note that while the preprocessor definition must be a quoted path, the environment variable must be an unquoted path.)
  2. Otherwise, if the compiler doesn’t provide <stdint.h>, use a custom implementation. This is currently limited to Visual Studio prior to 2010, using an implementation imported from msinttypes on Google Code.
  3. Otherwise use <stdint.h>.

We’re only providing these types now, but shortly we’ll start switching code using non-<stdint.h> fixed-size integer types to use them. Adding mozilla/StdInt.hmozilla/StandardInteger.h is merely the first step toward removing the other fixed-size integer types (except when they’re necessary to interact with external libraries).

Conclusion

It’s been a dozen years since <stdint.h> was standardized. Now it finally comes to Mozilla. Let’s lower a barrier to hackability in Mozilla and start using it.

11 Comments »

  1. So who is going to do this mass switchover? 😉

    Comment by Kyle Huey — 08.12.11 @ 13:38

  2. Nice. This stint away from your normal JS hacking time is time well-spent!

    Comment by Justin Dolske — 08.12.11 @ 13:42

  3. Note there’s a flaw in the scheme. Files under mfbt/ are installed in dist/include/mozilla when building js/src. Which means anything built before that (and there a some low-level bits there) can’t #include “mozilla/something.h”. I should probably file a bug.

    Comment by glandium — 08.12.11 @ 13:44

  4. I’m going to switch the JS engine…roughly. There are some holdouts in the JS pit that think that we should still use uint32 internally in SpiderMonkey code. So at least for now, unless those of us on the side of Truth, Justice, and the American Way prevail, the JSAPI will use the <stdint.h> types, and other SpiderMonkey code (internal or should-be-internal headers, non-header files) will use uint32 and such. I’m not giving up hope in the longer run — or perhaps in the shorter run as those holdouts see the error of their ways (having to decide which type to use each time it’s typed in a header is going to be pretty bothersome). But for now I am focusing on the positive of the massive win on the one hand and ignoring the negative of the one holdout on the other.

    I am not going to switch Mozilla outside the JS engine. Someone else with more experience with the rewriting tools can do that. 🙂 Plus there’s the question of how to make this work with the NSPR types, exactly, if the two need to intermingle (which they probably will in some cases when calling NSPR functions, where exact typedef aliases may still matter).

    Comment by Jeff — 08.12.11 @ 13:49

  5. I wouldn’t call it a stint, so much — just picking off the occasional thing as I need it. 🙂 It’s definitely nice doing stuff that provides benefits more broadly than to just the JS engine.

    Comment by Jeff — 08.12.11 @ 13:52

  6. <stint.h>

    Comment by Justin Dolske — 08.12.11 @ 17:56

  7. Subtle is always good, but. 🙂 I’m willing to say that one’s too subtle. (Although it did occur to me you might have been driving at that.)

    Comment by Jeff — 08.12.11 @ 18:26

  8. Which version of Mozilla did this land in? FF8, FF9, etc.

    Comment by Alex Vincent — 09.12.11 @ 09:04

  9. It’s in mozilla-central now, which I think corresponds to Firefox 11.

    Comment by Jeff — 09.12.11 @ 11:46

  10. I was having trouble finding this file.
    It appears it was renamed to: “StandardInteger.h”
    Is located at: “mfbt/StandardInteger.h”
    And is included as: “mozilla/StandardInteger.h”

    Comment by jgilbert — 04.04.12 @ 18:19

  11. Oops, yes, the file had to be renamed due to case-insensitivity considerations and the arcane way includes in Mozilla are processed. 🙁

    Comment by Jeff — 13.04.12 @ 23:29

RSS feed for comments on this post. TrackBack URI

Leave a comment

HTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>