Monday, 1 July 2013

On custom languages for unit tests

I'm reading "Testing in Scala" by Daniel Hinojosa [] at the moment, trying to get a handle on the bewildering array of choices for how to write unit tests for Scala code. A large part of this book is devoted to talking about how test conditions themselves are expressed. Both test frameworks discussed, ScalaTest and Specs2, seem to have picked up the trend from Business Driven Development (BDD) frameworks like Cucumber that suggests tests should be written in a language that looks vaguely like English, instead of simply being expressed in code. In other words, they've created Domain Specific Languages (DSLs) to express test cases and conditions. I'm puzzled by the adoption of this strategy in unit test frameworks - let me try to explain why:

One example is the following set of simple tests, written using ScalaTest:

 string should not endWith "My favorite friend, the end"
 string should not include ("Great balls of fire")
 string should endWith regex ("h.{4}r")
 string should include regex ("flames?")

This basically invents a new API for string operations ("endWith Regex" etc) in addition to the checking/assertion constructs needed for tests ("should", "should not"). ScalaTest also provides other "matchers", i.e. similar APIs, for floating point numbers, references, sequences, Maps etc. The developer writing tests like this is thus forced to learn a whole new set of operations in addition to the ones that the corresponding classes already provide.

Another level of complexity is introduced when test conditions consist of more complex operations, as in this example:

 redHotChiliPeppers should (contain("Anthony Kiedis") and
 (not contain ("John Frusciante")
 or contain("Dave Navarro")))

Here, new terms "and" and "not" have been introduced, but they don't behave as Scala's && and ! operators - their logic isn't short circuited, for example. This means yet more considerations for the poor developer to learn and keep in mind.

I'm puzzled by all this. Why wouldn't you want to write your unit tests in the same language as your code is written in? It's not like languages like Scala are lacking in expressivity, so why layer another language on top of it to create test conditions? And why write your test code in a different way than your code under test? One may say that the pseudo-English tests are easier to read for non-technical people but why would they be reading unit tests?

If instead we were to write some of the above examples in plain old Scala, they might look like this:

 assert(!string.endsWith("My favorite friend, the end"))
 assert((redHotChiliPeppers intersect Set("John Frusciante",
     "Dave Navarro", "Anthony Kiedis")) == Set("Anthony Kiedis"))

While not pretending to be English, this isn't exactly hard to read, and is only limited by the underlying langauge itself in terms of expressivity.

I'm sure the authors of these frameworks had the best intentions in mind and wanted to make it easier to write tests. And it may work OK when getting non-technical project stakeholders to write tests and specifications, BDD style, though I have yet to meet someone who's ever made that work in practice. But as a strategy for developer-authored unit tests, I think the DSL approach is misguided.

Maybe I'm missing something here (and feel free to point this out!) but until convinced otherwise, I'll write my tests in the good old xUnit style.


  1. You should really try ScalaCheck! It's not a DSL, but it's very different from xUnit-style testing.

  2. Thanks for the tip, I'll definitely check that out (no pun intended!).

  3. actually english is spoken throughout the world because it is one of the most important languages and alot of countries make thier students learn it.

    My IELTS Experience


Real Time Web Analytics