06.03.11

JavaScript change in Firefox 5 (not 4), and in other browsers: regular expressions can’t be called like functions

Callable regular expressions

Way back in the day when Netscape implemented regular expressions in JavaScript, it made them callable. If you slapped an argument list after a regular expression, it’d act as if you called RegExp.prototype.exec on it with the provided arguments.

var r = /abc/, res;

res = r("abc");
assert(res.length === 1);

res = r("def");
assert(res === null);

Why? Beats me. I’d have thought .exec was easy enough to type and clearer to boot, myself. Hopefully readers familiar with the history can explain in comments.

Problems

Callable regular expressions present one immediate problem to a “naive” implementation: their behavior with typeof. According to ECMAScript, the typeof for any object which is callable should be "function", and Netscape and Mozilla for a long time faithfully implemented this. This tended to cause much confusion in practice, so browsers that implemented callable regular expressions eventually changed typeof to arguably “lie” for regular expressions and return "object". In SpiderMonkey the “fix” was an utterly inelegant hack which distinguished callables as either regular expressions or not, to determine typeof behavior.

Past this, callable regular expressions complicate implementing callability and optimizations of it. Implementations supporting getters and setters (once purely as an extension, now standardized in ES5) must consider the case where the getter or setter is a regular expression and do something appropriate. And of course they must handle regular old calls, qualified (/a/()) and unqualified (({ p: /a/ }).p()) both. Mozilla’s had a solid trickle of bugs involving callable regular expressions, almost always filed as a result of Jesse‘s evil fuzzers (and not due to actual sites breaking).

It’s also hard to justify callable regular expressions as an extension. While ECMAScript explicitly permits extensions, it generally prefers extensions to be new methods or properties of existing objects. Regular expression callability is neither of these: instead it’s adding an internal hook to regular expressions to make them callable. This might not technically be contrary to the spec, but it goes against its spirit.

Regular expressions won’t be callable in Firefox 5

No one’s ever really used callable regular expressions. They’re non-standard, not all browsers implement them, and they unnecessarily complicate implementations. So, in concert with other browser engines like WebKit, we’re making regular expressions non-callable in Firefox 5. (Regular expressions are callable in Firefox 4, but of course don’t rely on this.)

You can experiment with a version of Firefox with these changes by downloading a TraceMonkey nightly build. Trunk’s still locked down for Firefox 4, so it won’t pick up the change until Firefox 4 branches and trunk reopens for changes targeted at the next release. (Don’t forget to use the profile manager if you want to keep the settings you use with your primary Firefox installation pristine.)

26.02.11

The proper way to call parseInt (tl;dr: parseInt(str, radix))

Introduction

Allen Wirfs-Brock recently discussed the impedance mismatch when functions accepting optional arguments are incompatibly combined, considering in particular combining parseInt and Array.prototype.map. In doing so he makes this comment:

The most common usage of parseInt passes only a single argument

Code-quality systems like JSLint routinely warn about parseInt without an explicit radix. Most uses might well pass only a single argument, but I could easily imagine this going the other way.

This raises an interesting question: why do lints warn about using parseInt without a radix?

parseInt and radixes

Like much of JavaScript, parseInt tries to be helpful when called without an explicitly specified radix. It attempts to guess a suitable radix:

assertEq(parseInt("+17"), 17);
assertEq(parseInt("42"), 42);
assertEq(parseInt("-0x42"), -66);
// assertEq(parseInt("0755"), ???); // we'll get back to this

If the string (after optional leading whitespace and + or -) starts with a non-zero decimal digit, it’s parsed as decimal. But if the string begins with 0, things get wacky. If the next character is x or X, the number is parsed in base-16: hexadecimal. Last, if the next character isn’t x or X…hold that thought. I’ll return to it in a moment.

Thus the behavior of parseInt without a radix depends not just on the numeric contents of the string but also upon its internal structure, entirely separate from its contents. This alone is reason enough to always specify a radix: specify a radix 2 ≤ r ≤ 36 and it will be used, no guessing, no uncertain behavior in the face of varying strings. (Although, to be sure, there’s still a very slight wrinkle: if r === 16 and your string begins with 0x or 0X, they’ll be skipped when determining the integer to return. But this is a pretty far-out edge case where you might want to parse a hexadecimal string without a prefix and would also want to process one with a prefix as just 0.)

But wait! There’s more

Beyond cuteness lies another concern. Let’s return to the leading-zero-but-not-hexadecimal case:

parseInt("0755");

In some programming languages a leading zero (that’s not part of a hexadecimal prefix) means the number is base-8: octal. So maybe JavaScript infers this to be an octal number, returning 7 × 8 × 8 + 5 × 8 + 5 === 493.

On the other hand, as I noted in Mozilla’s ES5 strict mode documentation, there’s some evidence that people use leading zeroes as alignment devices, thinking they have no semantic effect. So maybe leading zero is decimal instead.

The wonderful thing about standards is that there are so many of them to choose from

According to ES3, a leading zero with no explicit radix is either decimal or, if octal extensions have been implemented, octal. So what happens depends on who wrote the ES3 implementation, and what choice they made. But what if it’s an ES5 implementation? ES5 explicitly forbids octal and says this is interpreted as decimal. Therefore, parseInt("0755") is 755 in bog-standard ES3 implementations, 493 in ES3 implementations which have implemented octal extensions, and 755 in conforming ES5 implementations. Isn’t it great?

What do browsers actually do?

On the web everyone implements the octal extensions, so you’ll have to look hard to find an ES3 browser that doesn’t make parseInt("0755") === 493. But ES3 is old and busted, and ES5 is the new hotness. What do ES5 implementations do, especially as the change in ES5 isn’t backwards-compatible?

Surprisingly browsers aren’t all playing chicken here, waiting to see that they can change without breaking sites. On this particular point IE9 leads the way (in standards mode code only), implementing parseInt("0755") === 755 despite having implemented parseInt("0755") === 493 in the past. Before I saw IE9 implemented this (although I hasten to note they have not shipped a release with it yet), I expected no browser would implement it due to the possibility of breaking sites. After seeing IE9′s example, I’m less certain. Hopefully their experience will shed light on the decision for the other browser vendors.

Conclusion

Precise details of browser and specification inconsistencies aside, the point remains that parseInt(str) tries to be cute when parsing str. That cuteness can make parseInt(str) unpredictable if inputs vary sufficiently. Avoid edge-case bugs and use parseInt(str, radix) instead.

03.02.11

Working on the JS engine, Episode IV

A testcase submitted to us today:

([][(![]+[])[!+[]+!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[]+[][[]])[+!+[]+[+[]]]+(![]+[])[!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+!+[]]])[+!+[]+[+[]]]+(!![]+[])[+!+[]]+(!![]+[])[+[]]][([][(![]+[])[+[]]+(![]+[]+[][[]])[+!+[]+[+[]]]+(![]+[])[!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+!+[]]]+[])[!+[]+!+[]+!+[]]+(![]+[])[+!+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[!+[]+!+[]]]()+[])[!+[]+!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[]+[][[]])[+!+[]+[+[]]]+(![]+[])[!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+!+[]]])[+!+[]+[+[]]]+([][(![]+[])[!+[]+!+[]+!+[]]+(!![]+[][(![]+[])[+[]]+(![]+[]+[][[]])[+!+[]+[+[]]]+(![]+[])[!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+!+[]]])[+!+[]+[+[]]]+(!![]+[])[+!+[]]+(!![]+[])[+[]]][([][(![]+[])[+[]]+(![]+[]+[][[]])[+!+[]+[+[]]]+(![]+[])[!+[]+!+[]]+(!![]+[])[+[]]+(!![]+[])[!+[]+!+[]+!+[]]+(!![]+[])[+!+[]]]+[])[!+[]+!+[]+!+[]]+(![]+[])[+!+[]]+(![]+[])[!+[]+!+[]]+(![]+[])[!+[]+!+[]]]()+[])[!+[]+!+[]]

The result according to ES3, plus a common implementation-specific behavior, is the string "job".

The result according to ES5, plus a common implementation-specific behavior, is a thrown TypeError.

24.01.11

New ES5 strict mode requirement: function statements not at top level of a program or function are prohibited

Function statements in ECMAScript

What’s the effect of this program according to ECMAScript?

function foo() { }

If you said that it defines a function as a property of the global object, congratulations! You’ve mastered a basic part of JavaScript syntax.

Let’s go a little trickier: what is the effect of the function defined in this program according to ECMAScript?

function foo()
{
  return g;
  function g() { }
}

This function, when called, defines a local variable g whose value is the specified function. Then it returns that function as the value of that variable. If you knew this as well, give yourself a gold star.

Now let’s try something even harder: what’s the effect of these programs?

if (true)
  function bar() { }
function g() { }
function foo()
{
  if (true)
    function g() { }
  return g;
}

Shenanigans!

Trick question! They fail to run due to syntax errors.

ECMAScript permits function statements in exactly two places: directly within the list of statements that make up a program, and directly within the list of statements that make up the contents of a function body. These are the first two examples. (A function statement also looks like an expression, but if it appears in expression context it’s a function expression, not a function statement.) Engines which permit a function statement anywhere else — as the child of a block statement enclosed by curly braces, as the child of a loop or condition, as the child of a with, or as the child of a case or default in a switch statement — do so by extending ES5.

Spec requirements aside, what are the semantics of extensionland function statements?

Now you’re just messing with me

Which semantics?

Browsers all implement extensionland function statements differently, with different semantics. Use them just so and they’ll work the same way across browsers. Use them in any way where the function statement conditionally executes, or where you start capturing the binding for the function in different locations, and you’ll find any semblance of cross-browser compatibility disappears. This example by Rich Dougherty, used with permission, demonstrates some of the incompatibilities (and I wonder whether function statements in with might present more):

var result = [];
result.push(f());
function f() { return 1; }
result.push(f());
if (1)
{
  result.push(f());
  function f() { return 2; }
  result.push(f());
}
result.push(f());
function y()
{
  result.push(g());
  function g() { return 3; }
  result.push(g());
  if (1)
  {
    result.push(g());
    function g() { return 4; }
    result.push(g());
  }
  result.push(g());
}
y();
print(result);

Results in different browsers vary a fair bit, although there’s a little more consensus on behavior now than at the time this example was originally written:

Browser Output
Firefox 1.5 and 2 1,1,1,2,2,3,3,3,3,3
Firefox 4 1,1,1,2,2,3,3,3,4,4
Opera 2,2,2,2,2,4,4,4,4,4
Internet Explorer 7 2,2,2,2,2,4,4,4,4,4
Safari 3 1,1,2,2,2,3,3,4,4,4
Safari 4 2,2,2,2,2,4,4,4,4,4
Chrome 2,2,2,2,2,4,4,4,4,4

Why not specify semantics?

Blindly specifying some particular behavior won’t work. Many sites these days (and different browser-specific implementations of those sites) rely on engine-specific behavior with user-agent-conditioned code. Changing browser behavior breaks that pretty hard. Specification will break any browsers not already implementing it at time of specification.

A way forward

The next version of ECMAScript would like to specify semantics for this case — quite possibly semantics not implemented by any browser. How to do it, if implementations irreconcilably disagree? The solution comes in two parts. First, “ES6” will require affirmative opt-in to enable new syntax and semantics, including for currently-nonstandard function statements. Second, in anticipation of that change, the ECMA committee recommends that non-standard function statements be forbidden in strict mode code, to open up a future path down which ES6 can walk.

To permit ES6 to standardize semantics, the ECMA committee recommends forbidding non-standard function statements in strict mode code. Thus these examples are syntax errors:

"use strict";
{
  function foo() { }
}
"use strict";
if (true)
  function bar() { }
"use strict";
with (obj)
  function foo() { }
"use strict";
for (;;)
  function foo() { }
"use strict";
switch (v)
{
  case 10:
    function bar() { }
  default:
    function baz() { }
}

Both Firefox and WebKit now implement this restriction, and other engines will follow as they too implement strict mode.

Conclusion

In order for future versions of ECMAScript to be able to define semantics for extensionland functions, strict mode “clears the deck” and forbids them entirely. Instead, assign functions to variables, a la var f = function() { };. Semantics for this are completely defined and compatibly implemented across browsers.

You can experiment with a version of Firefox with these changes by downloading a nightly build. (Don’t forget to use the profile manager if you want to keep the settings you use with your primary Firefox installation pristine.)

10.01.11

New ES5 strict mode support: new vars created by strict mode eval code are local to that code only

tl;dr

Ideally you shouldn’t use eval, because it inhibits many optimizations and makes code run slower. new Function(argname1, argname2, ..., code) doesn’t inhibit optimizations, so it’s a better way to dynamically generate code. (This may require passing in arguments instead of using names for local variables. That’s the price you pay to play nice with compilers that could optimize but for eval.)

Nevertheless, it’s still possible to use eval in ES5. eval of normal code behaves as it always has. But eval of strict mode code behaves differently: any variables created by the code being evaluated affect only that code, not the enclosing scope. (The enclosing scope’s variables are still visible in the eval code if they aren’t shadowed.) Firefox now correctly binds variables created by strict mode eval code, completing the last major component of strict mode (though bugs remain). These programs demonstrate the idea:

var x = 2, y = 3;
print(eval("var x = 9; x"));               // prints 9
print(x);                                  // prints 9
print(eval("'use strict'; var x = 5; x")); // prints 5
print(eval("'use strict'; var x = 7; x")); // prints 7
print(eval("'use strict'; y"));            // prints 3
print(x);                                  // prints 9
"use strict";
var x = 2, y = 3;
// NB: Strictness propagates into eval code evaluated by a
//     direct call to eval — a call occurring through an
//     expression of the form eval(...).
print(eval("var x = 5; x")); // prints 5
print(eval("var x = 7; x")); // prints 7
print(eval("y"));            // prints 3
print(x);                    // prints 2

This partially defangs eval. But even strict mode eval inhibits optimizations, so you are still better off avoiding it.

eval is a double-edged sword

eval is one of the most powerful parts of JavaScript: it enables runtime code generation. You can compile code to perform specific operations, avoiding unnecessary general-purpose overhead — a powerful concept. (But you’d be better off using new Function(argname1, argname2, ..., code), which doesn’t inhibit optimizations and still enables code generation, at loss of the ability to capture the local scope. Code using eval may see considerable speedups: for example, roc’s CPU emulator sped up ~14% switching from eval to Function. Less beefy code won’t see that magnitude of win, yet why give up performance when you have a ready alternative?)

Yet at the same time, eval is too powerful. As inline assembly is to C or C++ (at least without the information gcc‘s asm syntax requires), so is eval to JavaScript. In both instances a powerful construct inhibits many optimizations. Even if you don’t care about optimizations or performance, eval‘s ability to introduce and delete bindings makes code that uses it much harder to reason about.

eval arbitrarily mutates variables

At its simplest, eval can change the value of any variable:

function test(code)
{
  var v = 1;
  eval(code);
  return v;
}
assert(test("v = 2") === 2);

Thus you can’t reorder or constant-fold assignments past eval: eval forces everything to be “written to disk” so that the eval code can observe it, and likewise it forces everything to be read back “from disk” when needed next. Without costly analysis you can’t store v in a register across the call.

eval can insert bindings after compilation

eval‘s ability to add bindings is worse. This can make it impossible to say what a name refers to until runtime:

var v;
function test(code)
{
  eval(code);
  return v;
}

Does the v in the return statement mean the global variable? You can’t know without knowing the code eval will compile and run. If that code is "var v = 17;" it refers to a new variable. If that code is "/* psych! */" it refers to the global variable. eval in a function will deoptimize any name in that function which refers to a variable in an enclosing scope. (And don’t forget that the name test itself is in an enclosing scope: if the function returned test instead of v, you couldn’t say whether that test referred to the enclosing function or to a new variable without knowing code.)

eval can remove bindings added after compilation

You can also delete bindings introduced by eval (but not any other variables):

var v = 42;
function test(code)
{
  eval(code);
  function f(code2)
  {
    eval(code2);
    return function g(code3) { eval(code3); return v; };
  }
  return f;
}
var f = test("var v = 17;");
var g = f("var v = 8675309;");
assert(g("/* nada */") === 8675309);
assert(g("var v = 5;") === 5);
assert(g("delete v") === 17);
assert(g("delete v") === 42);
assert(g("delete v") === 42); // can't delete non-eval var (thankfully)

So not only can you not be sure what binding a name refers to given eval, you can’t even be sure what binding it refers to over time! (Throw generators into the game and you also have to account for a scope without a binding containing that binding even after a function has “returned”.)

eval can affect enclosing scopes

Worst, none of these complications (and I’ve listed only a few) are limited to purely local variables. eval can affect any variable it can see at runtime, whether in its immediate function or in any enclosing function or globally. eval is the fruit of the poisonous tree: it taints not just the scope containing it, but all scopes containing it.

Save us, ES5!

ES5 brings some relief from this madness: strict mode eval can no longer introduce or delete bindings. (Normal eval remains unchanged.) Deleting a binding is impossible in strict mode because delete name is a syntax error. And instead of introducing bindings in the calling scope, eval of strict mode code introduces bindings for that code only:

var x = 2, y = 3;
print(eval("var x = 9; x"));               // prints 9
print(x);                                  // prints 9
print(eval("'use strict'; var x = 5; x")); // prints 5
print(eval("'use strict'; var x = 7; x")); // prints 7
print(eval("'use strict'; y"));            // prints 3
print(x);                                  // prints 9

This works best if you have strict mode all the way down, so that eval can never affect the bindings of any scope (and so you don’t need "use strict" at the start of every eval code):

"use strict";
var x = 2, y = 3;
// NB: Strictness propagates into eval code evaluated by a
//     direct call to eval — a call occurring through an
//     expression of the form eval(...).
print(eval("var x = 5; x")); // prints 5
print(eval("var x = 7; x")); // prints 7
print(eval("y"));            // prints 3
print(x);                    // prints 2

Names in strict mode code can thus be associated without having to worry about eval in strict mode code altering bindings, preserving additional optimization opportunities.

Firefox now correctly implements strict mode eval code binding semantics (modulo bugs, of course).

So if I write strict mode code, should I use eval?

eval‘s worst aspects are gone in strict mode, but using it still isn’t a good idea. It can still change variables in ways the JavaScript compiler can’t detect, so strict mode eval still generally forces every variable to be saved before it occurs and to be reloaded when needed. This deoptimization is unavoidable if runtime code generation can affect dynamically-determined local variables. It’s still better to use Function than to use eval.

Also, as a temporary SpiderMonkey-specific concern, we don’t perform many of the binding optimizations strict mode eval enables. Binding semantics might (I haven’t tested, and it’s entirely possible the extra work is unnoticeable in practice) slow down strict eval compared to normal eval. Strict mode eval performance in SpiderMonkey won’t be much better than that of regular eval, and it might be slightly worse. We’ll fix this over time, but for now don’t expect strict mode eval to improve performance. (If you really need performance, don’t use eval.)

Conclusion

eval is powerful — arguably too powerful. ES5′s strict mode blunts eval‘s sharpest corners to simplify it and permit typical optimizations in code using it. But while strict mode eval is better than regular eval, Function is still the best way to generate code at runtime. If you must use eval, consider using strict mode eval for a simpler binding model and eventual performance benefits.

You can experiment with a version of Firefox with these changes by downloading a nightly build. (Don’t forget to use the profile manager if you want to keep the settings you use with your primary Firefox installation pristine.)

« NewerOlder »