The JMAP Test Suite

If you’ve been a regular follower of the Fastmail blog, you’ve heard of JMAP. If you haven’t, go check out that link. You’ll get to see Neil and Bron telling you why it’s cool. If you’re really into the idea, you might also like reading the spec, which is quite easy to read and presented in soothing Fastmail Blue.

JMAP! Does it work?

If you can’t be bothered, though, it’s like this: JMAP is a way of accessing your email in the cloud. It’s like IMAP, but better. (It’s often joked that JMAP is one better than IMAP, but that’s not quite right. Then it would just be IMAQ. JMAP is 1,000 better than IMAP.)

JMAP looks really good in theory, but until somebody implements it, it’s not all that useful. Fortunately, Fastmail and others are hard at work implementing it. That means there are a number of partly-implemented JMAP servers, meaning that there are also a bunch of only partially-satisified JMAP consumers, like me. I’m quite pleased to count myself among the earliest users of JMAP, because it means I’ll end up with the most stories about the ways in which early implementations failed horribly. To help gather those stories, I’ve been working on the beginnings of a generic test suite for JMAP compliance, to check any given JMAP server against a rigorous set of torture tests. (This also ends up helping JMAP implementors make sure they’ve gotten it right, and I guess that’s why Fastmail is paying me to do it, but it’s certainly not the part that makes it fun for me!)

Writing the JMAP test suite has required writing a number of new pieces of software, and I’m going to give you a quick tour through them. If the idea of seeing still-half-baked Perl 5 code fills you with an indescribable sense of dread, maybe you should stop here and go read about revolutionary mail technology of the future (of the past) instead.

Perl is Funny

So, I write a lot of Perl, and it’s a great language, and I wouldn’t want it to get the wrong idea about how much I love it, but it’s got a bit of a huge blind spot. It’s just terrible at telling numbers and strings apart. Given these two variables in Perl…

my $x =  5 ;
my $y = "5";

…Perl is pretty much not going to be able to tell you that they’re not identical. Sure, there are some tricks that work some of the time, but there’s no really great solution. This is fine as long as your program doesn’t talk to anybody else’s programs, but when you’re testing a network server, program-to-program interaction is rather the order of the day.

JMAP uses JSON for data interchange and JSON has a strict distinction between types. These two hunks of JSON are quite different:

{ "mailboxId":  123  }
{ "mailboxId": "123" }

If your server hands back the wrong 123, your client is going to have a bad day. This meant we had two options: write our JMAP test system in something other than Perl or add a number/string distinction to Perl. We decided to do the latter. See, Perl? That’s how much we love you. This got us JSON::Typist, a type annotator for JSON data. When applied to a freshly JSON-decoded hunk of data, it indelibly marks numbers and strings as such:

my $data  = decode_json(q<{ "count": 5, "id": "5" }>);
my $typed = JSON::Typist->new->apply_types( $data );

$typed->{id   }->isa('JSON::Typist::String'); # true
$typed->{count}->isa('JSON::Typist::Number'); # true

$typed->{count}->isa('JSON::Typist::String'); # false
$typed->{id   }->isa('JSON::Typist::Number'); # false

This lays the foundation for testing the types of JSON data in a fairly typeless language. The next layer is Test::Deep::JType, which extends Perl 5’s excellent Test::Deep system to account for maybe-typed data. You can write:

jcmp_deeply(
  $jmap_response,
  [
    [ messagesSet => { updated => superhashof({ id => jnum(123) }) } ],
  ],
);

…and assert that when the JMAP server says it has updated message 123, it has returned the number 123, not the string. This is vital, not only because downstream clients will be confused by the wrong type, but also because Neil will give you a withering look once he realizes how you’ve screwed up. He might forgive you, though, if you do penance by implementing a test framework to prevent future mistakes of that sort. Here’s hoping, anyway, right?

Enough Perl, Let’s JMAP

Now that we can actually inspect JMAP responses to make sure that we’ve gotten the right things, the next thing we want is to actually get some JMAP responses! For this, we need a JMAP client. JMAP clients can come in many levels of complexity — one of JMAP’s strengths! Everybody’s favorite command line browser, curl, can be an adequate client for some things. For users who just want to read their mail, FastMail’s in-browser JMAP client is a much more satisfying experience. You want the right tool for the job, and the right tool for testing JMAP is JMAP::Tester.

JMAP::Tester is an HTTP client build to do only four kinds of things: authenticate with a JMAP server, upload and download binary content to it, and execute JMAP methods against it. All of these are trivial operations where curl would really suffice, but JMAP::Tester provides rich result objects that make it easy to inspect responses for just what you expect without flailing about every time. For example:

subtest "can we bake pies?" => sub {
  my $res = $jmap_tester->request([
    [ setPies  => {
      create => { shoofly => { delicious => jtrue(), origin => jstr(18018) } }
    } ],
  ]);

  my ($pies) = $res->single_sentence('pies')->as_set;

  isnt($pies->old_state, $pies->new_state, "the pies collection changed");

  jcmp_deeply(
    $pies->created->{shoofly},
    superhashof({ id => jstr }),
  );
};

We can make a request (ensuring we send the right types of data), assert that we got exactly one line of response, which is some kind of setFoos call, assert that our server state changed, and that we got back a record with an id string. Meanwhile, if we fail a critical assertion, like getting a pies response rather than an error, the entire subtest will cleanly abort and move on to the next subtest.

Testing JMAP Mail

Some of you may now be wondering why there’s now delicious Pennsylvanian molasses pie smeared all over your shiny new mail protocol. Well, JMAP isn’t just for mail. It’s also a generic client/server protocol that can be used for all sorts of things, including cloud-based bakeries. Most of the excitement about JMAP, though, is about JMAP Mail, which deals in messages and mailboxes more than cakes and pies.

This is where we (finally!!) reach the still very much under-construction JMAP::TestSuite, which layers JMAP Mail helpers on top of JMAP::Tester:

subtest "basic nested folder creation" => sub {
  my $batch = $context->create_batch(mailbox => {
    x => { name => "Folder X" },
    y => { name => undef },
    z => { name => "Folder Z", parentId => '#x' },
  });

  batch_ok($batch);

  ok( ! $batch->is_entirely_successful, "something failed");
  ok(   $batch->result_for('y')->is_error, 'y failed');

  my $x = $batch->result_for('x');
  my $z = $batch->result_for('z');

  ok(! $x->is_error, 'x succeeded')
  ok(! $z->is_error, 'z succeeded');

  if ($x->is_error or $z->is_error) {
    abort("without x and z, further testing pointless);
  }

  jcmp_deeply($z->parentId, $x->id, "z.parentId == x.id");
};

This is the fundamental unit of testing in the JMAP test suite: one hunk of code creating, updating, or destroying batches of data and then inspecting the results with Perl’s best-in-class testing libraries. Here, we create a batch of mailboxes. Each one is associated with the id (x, y, or z) by which its result can be found, which mirrors the way that creations and results are correlated in JMAP itself. With this system, any time a bug is found in a JMAP implementation, it should be trivial to safely add a new test for that bug and run it against every server.

In the next few iterations of the test suite, we’ll be adding generators for valid-but-random test data and improved debugging output so that “just add print” for debugging can be done automatically. This helps prevent the ever-embarrassing git commit, “remove 692 data dumps accidentally left in after debugging,” which appears eleven times in my commit history for the last week.

Pointing the Suite at your Server

Different JMAP servers have different capabilities. JMAP specifies an authentication mechanism, but not every server uses it. JMAP doesn’t specify how to create new users, so different servers will have different mechanisms for creating test users. (Or maybe, too, you want to re-use a test user over and over.) To abstract all this away, the JMAP test suite offers the idea of a “server adapter” that handles authenticating a JMAP::Tester to a server. We’ve got adapters for Cyrus, “standard JMAP,” and the JMAP Proxy. To point the whole test suite at your server, you just tell it which adapter to use:

~/code/JMAP-TestSuite$ cat cyrus.json
{
  "adapter"   : "Cyrus",
  "base_uri"  : "https://jmap.local/",
  "credentials": [ { "auth_token": "cmpic0BmYXN0bWFpbC5jb206c29ycnlhZ2Fpbm5laWw=" } ]
}

~/code/JMAP-TestSuite$ JMAP_SERVER_ADAPTER_FILE=cyrus.json perl -I lib t/mailboxes.t
ok 1 - one sentence of response to getMailboxes
ok 2 - batch has results for every creation id and nothing more
ok 3 - no unknown properties in batch results
ok 4 - something failed
ok 5 - y failed
ok 6 - x succeeded
not ok 7 - z succeeded
#   Failed test 'z succeeded'
#   at t/basic.t line 45.
ok 8 - our upload succeeded (496148e64f1c930e8ac75912f24dada211ba54db)
ok 9 - batch has results for every creation id and nothing more
not ok 10 - some batch results have unknown properties

…and look at that! A bug in Cyrus, discovered by the JMAP test suite! I look forward to finding ever more and weirder bugs in JMAP implementations through the perpetual expansion of the test suite… but also to implementations made increasingly robust by the test suite. Even better for the authors of JMAP servers, they can run the test suite in the privacy of their own offices and, unlike me, it won’t giggle maniacally at each new bug. There are still a few features left to add to the test suite, and a lot of tests left to write, but it’s a promising beginning.