Test distance - testing at the right level of abstraction

So here's a metric for you:

Test distance - the measure of how far from app code your test code is.

To see what I mean by it, let's take a journey from Galactic distances down to the very microscopic ones...

1. In a separate project (Galactical distance)

Sometimes tests are kept in a completely separate repository from the main source code tree. Looks crazy? Indeed... Most likely the project manager was clueless. But there are specific situations where this kind of arrangement might make sense. For example when your tests apply for many projects, like the ACID tests for browsers.

Advantages:

• Keeps the tests independent from a specific project.

Disadvantages:

• Very hard to keep tests in sync (if possible at all).
• Inconvenient to even run the tests.

2. In a separate directory (Flying distance)

The most widespread practice is to have your app code and test code as a separate directory-trees which closely mimic each-other:

src/
  foo.rb
  bar.rb
test/
  foo_test.rb
  bar_test.rb

But the common practice is an odd one. It's tedious to rename foo_test.rb every time you rename foo.rb, and it's inconvenient to look up the corresponding test file from a completely separate directory.

The actual big advantage of this strategy is the ability to have your tests structured completely differently from code. For example you could have a full directory of tests to ensure one very complex function works as needed, or you could have one test file to cover the behavior of a group of classes.

Advantages:

• Keeps the tests file-hierarchy independent from the code.
• Easy to exclude tests from production version.
• Well-supported by tools.

Disadvantages:

• Easy to forget about updating the tests when you update code.
• Tedious to keep the structure of src/ and test/ dir in sync.
• Tedious to look up the test from a separate directory.

3. In a separate file (Driving distance)

If you really want to have a test file for every code file, then here's a possible structure for you:

src/
  foo.rb
  foo_test.rb
  bar.rb
  bar_test.rb

I've never really seen it used in practice. Most devs like to keep some distance between the code and tests and this seems to cross that line. It also has a bit of a clobbered feeling, and the number of files in your src/ dir will double.

But this way it's much harder to forget about the tests. You don't have to take on a journey to a different directory tree to get to your tests - they are right there.

Advantages:

• Easy to look up the corresponding test file of a normal source file.
• Easy to keep the file structure of tests and code in sync.

Disadvantages:

• Clobbers the main src/ dir with tests.

4. Inside a file (Grabbing distance)

To really solve the syncronisation of code and test files we need to get rid of the test files. The solution: put tests inside the code. For example let's have the test code at the end of the file:

function foo() {
}
function bar() {
}
...
function test_foo() {
}
function test_bar() {
}

All the file-management problems now go away. When you open up the code, you simultaneously open up the tests.

I've seen this practice sometimes used in Haskell. Also @test annotations in Java allow for a relatively convenient way to achieve this. In a compiled languages this whole approach makes much more sense - the code for test functions is simply eliminated at compile time.

Though, if you put the tests at the end of the file, you might need to scroll up and down between the code and tests, which could end up being really tedious. While if you put the test methods right next to real methods, your code might become way too messy.

Advantages:

• There's no file structure of tests to keep in sync, only the tests themselves.

Disadvantages:

• Mixed up code and tests.
• Some build system needed to remove tests from production version.
• Inconvenience of switching between code and tests.

5. Right next to the function (Microscopic distance)

To avoid creating indistinguishable mix of app code and test code, we can embed the tests inside comments. Here's a style example from Functional JS libarary which uses that approach throughout its code:

/**
 * Returns its argument coerced to a function.
 * >> lambda('1+')(2) -> 3
 * >> lambda(function(n){return n+1})(2) -> 3
 */
Functional.lambda = function(object) {
    return object.toFunction();
}

This takes on a literal-programming feel as tests will become an actual part of the documentation.

Advantages:

• Very little effort needed in keeping tests in sync.
• Convenient for writing the simplest tests.
• Tests form a part of the documentation.

Disadvantages:

• A tool needed to extract tests from comments.
• Only suitable for writing very simple tests.
• Tests might clutter the comments.

To sum up

There are some general tendencies here. When testing at hight distance you have a lot of flexibility in organizing your tests, but it's quite hard to keep your tests in sync with your app code. When you go down to smaller distances, it becomes much easier to keep things in sync, but you start to lose flexibility up to the point where you can only write very limited assertion-like tests.

Additionally as you go to smaller distances, you will need more specific tools to help you in keeping the app code and test code separate.

But I'm not here to tell you what the right distance is. You might be just fine with the usual flying distance. Or maybe the microscopic approach suits you the best. You could also test from several different distances at the same time. Like always, don't be slave to any methodology and knowing your options choose what suits you the best.

But make sure you don't fall entirely into the zeroth category that I didn't mention (but I think you can quess what that is).

Live preview of JavaScript code

After watching this amazing talk of Bret Victor I felt an immediate need to implement some of his demos in JavaScript.

After a bunch of trial and error I came up with this prototype.

It's far from what Bret would want us to aim for, but toying around with this primitive tool made me feel what an amazing effect immediate feedback can have on your coding. The initial code you see there is the direct result of me playing around with the tool... and I really wouldn't have written this script without having a really-really fast feedback loop. The whole process felt much more like playing than programming.

A word about implementation

The main trick is to avoid flickering of the preview area. Like when I'm at the middle of typing I don't want the preview to blank out because of a syntax error.

This is accomplished by having two iframes. The first iframe contains the latest working preview. When code is modified a new hidden iframe is created and the code is executed inside it. Only when the execution succeeds will the old iframe be replaced with new one.

Basic computer graphics really - you draw in the buffer and then repaint.

Taking it further

So this was nice, but toying around with plain JavaScript is not particularly interesting. So what I wanted to do instead was to use this technique for building Sencha Touch UI-s. This is obviously a very graphical task, so the approach should suit it well.

Here's my attempt: Sencha Touch live prototyping.

But there are problems...

Most importantly Sencha Touch is not as lightweight as plain JavaScript. It takes a fraction of a second to load the whole framework, but it's a large enough fraction to be noticable, and the worst part is that it blocks. It blocks, it blocks, and it blocks...

Even when JavaScript runs inside an iframe it still blocks everything on the whole page. And when the page freezes while you are typing, it's really offputting.

I tried, I tried, and I tried... but there seems to be no way to make it not block.

Of course there is one way to run JavaScipt in non-blocking way - web workers. But web workers can't access DOM, so for our purposes they are pretty much useless.

This leaves us with cheating...

The first obvious thing would be to avoid loading the whole Touch framework every single time. Just clean up the DOM and run the code again. It's far from being a correct approach, variables might get left lying around in the global namespace, but if it works 99% of time it might be good enough.

Unfortunately Touch doesn't like if you run its setup code multiple times. Naah! But there has to be a workaround...

It can be made faster...

There's hope...

Always...

The Magic of IE, Chapter 385: Conditional Compilation

Recently Rob Hogan reported in Sencha forum, that the uncompressed source code of Ext JS fails to load in IE. For some strange reason IE chokes at the following code:

someObj = {
    //@private
    someProp: "foo"
}

Giving an error message Expected ':'. And the strangest thing is, when you add just one space to comment:

someObj = {
    // @private
    someProp: "foo"
}

IE will parse it just fine.

Doesn't it just make you cheer to the wonderful and magical IE?

Thankfully Rob was kind enough to track down the issue to another library he was using. That happened to use another amazing feature of IE: Conditional Compilation (CC). Most people know about IE conditional comments for HTML, this is pretty much the same, but for JavaScript.

The greatness is turned off by default. You need to enable it with:

//@cc_on

After this there's no turning back - all the comments following this declaration will be subject to CC.

The thing supports simple if statements and some variables:

/*@cc_on
  @set @version = @_jscript_version
  @if (@_win32)
    document.write("You are running 32 bit IE " + @version);
  @elif (@win_16)
    document.write("You are running 16 bit IE " + @version);
  @else @*/
    document.write("You are running another browser or an old IE.");
/*@end @*/

That's pretty much all the syntax there is. But this doesn't quite explain why //@private should cause any trouble as there is no @private statement.

Still, there is a bit more syntax:

var isMSIE = /*@cc_on!@*/false;

That's some nifty JavaScript from Dean Edwards which evaluates to true if browser is IE. When CC is already turned on, it can be abbreviated to:

var isMSIE = /*@!@*/false;

Nice. So now I have a theory of what //@private does: it outputs the identifier private into the document. Let's test that:

//@ alert
("Hello world");

I run the program and get a nice "Hello world" popup. Great! Now let's try without the space:

//@alert
("Hello world");

I run it and... oh... IE says: Function expected.

Oh man! IE just had us again. What kind of amazing logic is he following?

After some more reading about conditional compilation I come to the conclusion that @private is a variable. But those CC variables can have only two types of values: Boolean or Number.

So let's see what kind of value //@private yields:

var foo = //@private
alert(foo);

And it outputs... NaN.

Isn't it just neat how all undefined variables are initialized to NaN. Now it all makes perfect sense. This code:

someObj = {
    //@private
    someProp: "foo"
}

would be seen by IE as:

someObj = {
    NaN someProp: "foo"
}

And NaN happens to be a valid key in object literal, because the key can be either identifier, String or Number... and NaN (Not a Number) obviously is a Number. Well... obviously if you have been programming in JavaScript for way too long.

Thank you all for reading. This was chapter 385 from my book "The Magic of IE" which I will never write.