Spock and JUnit - a comparison
In the book ‘Working effectively with legacy code’ Micheal Feathers defines legacy code as code without tests. While some might not agree with this definition I do think it showcases how important testing is. It is very difficult to maintain and develop quality software without proper test automation. I would go so far as to say that it is near impossible to effectively maintain an application without tests over a longer period of time.
In his book Mr Feathers also talks about the “feedback loop”, a term also thrown around in Agile and craftsman circles. The feedback loop boils down to that developers want feedback quickly. The quicker we get feedback the quicker we can respond to said feedback. We want to know whether or not we are making correct and working software. The best tool for getting feedback is through automated tests.
This post won’t cover the whole test pyramid, but instead we will take a look at the most common way of testing software: unit tests. Defining what a unit test is a post in of itself, but for now let’s go with Roy Osherove’s definition:
“A unit test is an automated piece of code that invokes a unit of work in the system and then checks a single assumption about the behavior of that unit of work.”
In the Java ecosystem the most popular form of unit test framework is JUnit. JUnit is used to verify millions upon millions of lines of code every day and is one of the most renowned unit test frameworks out there. JUnit is, by far, the unit test framework I have the most experience with.
The other framework we will take a look at is Spock, which has a smaller community than JUnit, but also widely used and extremely capable. I would even argue that if we take both JUnit and Spock at face value then Spock definitely provides more functionality than JUnit. Spock also has a very different approach than JUnit which we will take a look at later in the post.
The idea of this post is not to figure out the “best” framework. As with most things there’s no “best”, just different. One cannot go wrong with either of these frameworks, but that doesn’t mean we cannot have preferences and make informed decisions when picking a framework.
We will compare these frameworks by writing tests for a single class. This will hopefully allow us to do some meaningful 1-to-1 comparisons. We will take a look at the class we’re supposed to test, and at the end will contain the whole test suite for both JUnit and Spock. If you are interested the source code can be found on GitHub as well.
Content
Different ideologies
Before we even start looking at any code we should probably discuss the different ideologies behind each framework.
JUnit
JUnit is a very traditional xUnit testing framework. First and foremost JUnit is a tool for developers designed to be used while developing. JUnit is a very unopinionated framework and doesn’t care how tests are written. If you have read in a book, or looked at unit tests online, the chances are that it will look very similar to how a JUnit test looks.
Spock
Unlike JUnit, Spock has strong opinions on what a test is and how it should be written. This can be seen from the language the framework uses to its design. For example what JUnit calls a “test class” Spock wants to call a “Specification”. The framework wants us to consider the tests as specifications for our code, echoing ideas from BDD tools such as Cucumber. Spock wants us to elevate our tests from something that verifies our code to something which dictates the correct behavior of our code. This doesn’t mean that we cannot think of JUnit tests the same way, but the design of the JUnit framework does not force these opinions on the developers.
Spock takes it even further and calls tests for “Features”. Therefore we are no longer writing tests for our code, we are writing features in our specification. Spock even specifies how a test should be written and refuses to compile or execute tests which don't follow this structure.
I have said it before, and I’d say it again. Frameworks which have strong opinions and enforce those opinions are simply more interesting and fun. It doesn’t make such frameworks better, as that always comes down to the opinions it has, but the feeling of working with a framework which agrees with you is definitely a great one. Opinionated frameworks sets themselves apart from other frameworks and allows developers to choose tools which agree with them and have their opinions enforced by the framework.
How do we compare these frameworks?
While researching this post I found plenty of websites with examples from both JUnit and Spock. The examples I found seemed to be specifically designed to show off a given feature in each of the frameworks, which always worries me. Designed problems to show off features often makes the various frameworks look way better than they actually are, so I set out to make a more realistic comparison between the two. A comparison which tests production-like code. By using real, non-perfect, code we can get a feeling of how a framework might serve us in a real project.
This is why I decided to write unit tests for this class from the openmrs. The choice of this class is not random, but based on a few criteria:
The class has relatively few dependencies and doesn’t rely too heavily on global values
The logic is relatively clear, even for one which doesn’t know the domain
The class has a relatively small size
The class is about 10 years old, which means it is a solid piece of production code
In hindsight I wish this example had more complex test cases where we had to be even more creative with our tests.
UserValidatorImpl.java
package john.mikael.gundersen.healthcare;
import org.springframework.util.StringUtils;
import org.springframework.validation.BindException;
import org.springframework.validation.Errors;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.regex.PatternSyntaxException;
/**
* Code copied and rewritten from the OPENMRS project:
* https://github.com/openmrs/openmrs-core/blob/1c21cb563496bbfeb58133b0ad2ea8b39121783e/api/src/main/java/org/openmrs/validator/UserValidator.java
*/
public class UserValidatorImpl implements UserValidator {
private boolean emailAsUsername;
private EmailValidator emailValidator;
public UserValidatorImpl(boolean emailAsUsername, EmailValidator emailValidator) {
this.emailAsUsername = emailAsUsername;
this.emailValidator = emailValidator;
}
public void setEmailAsUsername(boolean emailAsUsername) {
this.emailAsUsername = emailAsUsername;
}
/**
* Checks that all required parameters for a user are filled out
*
* @param user to validate
* @return Errors which contains all the invalid information for the given user
*/
public Errors validate(User user) {
Errors errors = new BindException(user, user.getClass().getName());
if (user.isRetired() && StringUtils.isEmpty(user.getRetireReason()))
errors.rejectValue("retireReason", "error.null");
errors.addAllErrors(validatePerson(user));
errors.addAllErrors(validateUsername(user));
return errors;
}
private Errors validatePerson(User user) {
Errors errors = new BindException(user, user.getClass().getName());
Person person = user.getPerson();
if (user.getPerson() == null) {
errors.rejectValue("person", "error.null");
return errors;
}
if (person.getGender() == null)
errors.rejectValue("person.gender", "error.null");
if (person.getDead() == null)
errors.rejectValue("person.dead", "error.null");
if (person.getVoided() == null)
errors.rejectValue("person.voided", "error.null");
if (person.getPersonName() == null || person.getPersonName().isEmpty())
errors.rejectValue("person", "Person.names.length");
return errors;
}
private Errors validateUsername(User user) {
Errors errors = new BindException(user, user.getClass().getName());
if (emailAsUsername) {
boolean isValidUserName = isUserNameAsEmailValid(user.getUsername());
if (!isValidUserName)
errors.rejectValue("username", "error.username.email");
} else {
boolean isValidUserName = isUserNameValid(user.getUsername());
if (!isValidUserName)
errors.rejectValue("username", "error.username.pattern");
}
if (!StringUtils.isEmpty(user.getEmail()) && !isEmailValid(user.getEmail()))
errors.rejectValue("email", "error.email.invalid");
return errors;
}
private boolean isUserNameValid(String username) {
String expression = "^[\\w][\\Q_\\E\\w-\\.]{1,49}$";
if (StringUtils.isEmpty(username))
return true;
try {
Pattern pattern = Pattern.compile(expression, Pattern.CASE_INSENSITIVE);
Matcher matcher = pattern.matcher(username);
return matcher.matches();
} catch (PatternSyntaxException pex) {
return false;
}
}
public boolean isUserNameAsEmailValid(String username) {
return emailValidator.isValid(username);
}
private boolean isEmailValid(String email) {
return emailValidator.isValid(email);
}
}
To bring this class into our project and into a test harness I decided to do a few alteration from the original file:
Dropping the logger
Dropping global Context
Moving most of the dependencies to be injected through the constructor rather than using spring or global variables. This way we don’t have to have a bunch of spring dependencies in our project
Methods now return an Error object rather than changing the one sent in as parameters. This makes sense in openmrs as this is how the Spring validator works, but since we aren’t bothering with Spring at this point I set the methods to return an Error object instead. This won’t meaningfully change the scope of the tests, but at least we avoid having to instantiate the Error class in every test.
Split up the methods for better readability
Replaced apache’s EmailValidator with an interface, mostly so that we can have a look at mocking
Removed most comments
Note that we are not here to criticize the original source code or openmrs in any way. This is a very good example of the type of logic often found in production code. Validation classes are magnets for all kinds of iffy logic, yet openmrs have kept the class pretty tidy considering that the class has been around for about 10 years.
The functionality of UserValidatorImpl isn’t too complex:
Mostly the class checks whether various values are null
We have a check which verifies that an email is in the correct format by using the injected “EmailValidator” interface
We have a regex checks which verifies that the username has the correct format, but only if the username is not the email
Setting up the frameworks
Groovy & Java
Cards on the table - before using Spock I had a very strained relationship with Groovy. For that I blame Jenkins, as it uses a somewhat restrictive version of Groovy with a fair amount of quirks. Having pipelines fail due to calls like toString() being considered “illegal and a security threat” left a bad taste.
I’m happy to report that this is no longer the case. Groovy turned out to be a very pleasant language to work with, and I found very little that bothered me with it. I won’t call myself a Groovy developer just yet, but suffice to say that there were a few handy calls which did make the Spock tests so much prettier.
It is also thanks to Groovy that we get some of the excellent futures Spock provides. One example is what usually is a bitshift operator: <<, but considering that Groovy supports operator overloading means that Spock can take that operator and use it for something that is, in context, more sensible.
Java is Java. It is its own verbose self, but extremely powerful. Personally I am not a huge fan of Java, despite being employed as a Java developer. It is verbose, it is not very elegant and it loves piling annotations on top of other annotations. Note that I am not saying that Java is bad, but my feelings on the language is probably a post for another day.
One thing to consider is that Java is, to nobody's surprise, by far the most popular JVM language. This means that when writing Spock tests we will, most of the time, test code written in Java. Realistically we should then consider that most of the developers know, and are comfortable with, Java. It might be a hard sell to get them to write Groovy tests rather than using ye ol’ Java with JUnit. To me Groovy wasn’t a huge hurdle and its syntax is pretty compatible with Java’s syntax. I like Groovy and it was pleasant to use, though that might not be true for everyone else.
Object creation
When we talk about languages we should probably talk about one of the biggest differences, at least from what we see in our example. This might not be directly related to either JUnit or Spock, but it is still a difference coming with either of these frameworks.
Creating objects, especially in unit tests, is messy. Especially when we have big and complicated classes which we have to build for every test. Our User and Person class isn’t that big, but we would have had to repeat them for about 14 tests. To create a valid object we’re looking around 6 lines of code, which means that we’re looking at 84 lines of code where we only create these objects. We could probably get that down as not every test requires every single line, but considering that the current UserValidatorImplTest file has ~150 lines of code means that object creation would become a significant chunk of the code.
For this example I decided to go with a method which creates these objects with some default values. This way one test cannot affect the other test as the object doesn’t share instances between the tests.
In an attempt to make this even more pretty I had to utilize a few language features and libraries to get the tests looking how I want them to look
JUnit
The first thing I did was to add some Lombok annotations to the User and Person class which makes Java bearable:
@Setter
@Getter
@Builder(toBuilder = true)
@With
public class User {
private boolean retired;
private String retireReason, username, email;
private Person person;
}
In turn these annotations give us a lot of ways to create objects without hassle.
@Setter and @Getter generates getter and setter methods for each field in the class
@Builder allows us to use the builder pattern when creating objects:
private Person person() {
return Person.builder()
.dead(false)
.gender("Male")
.personName("John Doe")
.voided(false)
.build();
}@With allows us to use the .withX syntax to alter objects after its creation:
val input = user().withRetired(true);Combined this allows us to build new objects without too much issues, as well as changing required values without calling separate getters and setters. This approach generally works fine, but I am less happy with how tests which alter the Person object looks:
val input = user().withPerson(person().withDead(null));While this is not extremely horrible, but given hindsight I should probably have made this a little neater.
Spock
Spock has Groovy powers on its side, which means we can make it look even better.
What I decided to do is to use named mapped arguments. This way we only need to set up a map of default values, and then we can overload those values with whatever the calling function needs. For example if we have this user function:
static User user(Map args = [:]) {
def map = [retired: false, username: "bob", email: null, person: person(args)] << args
return User.builder()
.retired(map["retired"] as boolean)
.username(map["username"] as String)
.person(map["person"] as Person)
.email(map["email"] as String)
.build()
}And let’s say that we want to get a user, but we want to change the retired field, we only need to do this:
def user = user(retired: true)Another neat feature of this can be viewed in this line of code:
def map = [retired: false, username: "bob", email: null, person: person(args)] << argsBasically we are sending in the whole arguments map into the person method. As an example this allows us to change the person gender we can simply do this:
user([gender: “Female”])For production code I would not do it this way, as that would be highly confusing. Tests on the other hand have different requirements. In this scenario we are forced to deal a lot with object creation, but at the same time we only need to worry about two objects - one that is nested within the other. Making that process easier is a great benefit, but it can quickly spiral and be confusing if we add more objects to this chain.
This is definitely one of those instances where one should stop and think whether or not something is worth doing from a maintainability point of view. In this type of test I can see value in it, but I can also see how it can be abused.
If this pattern is not appropriate then Groovy has other creation patterns, or we can always fall back on the methods generated by Lombok.
One thing to note is that we do use our Lombok builder for this example, but in this specific scenario we don’t get much from it. We could have just as easily used constructor parameters or getters and setters without it affecting readability much.
Naming conventions
I think naming things properly is important. By naming things correctly we get a glimpse into the intention behind the test and if the test aligns with the author's vision.
For the JUnit tests I stuck with my go-to naming style: [UnitOfWork_StateUnderTest_ExpectedBehavior]
JUnit itself doesn’t really care much how you name your tests as long as they can be compiled into “legal” Java methods.
I did however struggle a little bit with which naming convention I would use for the Spock tests. In Spock you name your tests like a string, which means that spaces and all are okay:
def "must reject missing retiredReason from a retired user"() {This way of naming tests seems to be a bit too loosy goosy for my taste, yet this test so clearly defines what the feature we’re verifying. If retiredReason is missing and the user is retired, then the field should be rejected. In fact, it is so plain that a non-programmer could understand it. There is a valid argument that this way of naming something doesn’t actually show what’s under test, and that it muddles the state and expected behavior into normal human language. Maybe the more appropriate name would be something along these lines:
def "validate | user is retired without retiredReason | retiredReason rejected"() {For this scenario I decided to go with the loosy goosy approach. In a real application I might decide to go with something like the latter example, but that also comes down to what the development team prefers.
To me it feels like Spock wants us to build tests in the same spirit as Cucumber, which also contains similar labels as Spock.
Naming is hard, and we can write bad names in either frameworks. Spock gives developers a little more freedom without having to add more annotations. Whether that is a good thing or bad thing I’m unsure of, but I’m sure both JUnit and Spock are equally open for abuse when it comes to naming.
Test structure
JUnit
JUnit puts no restrictions on what a test should or should not contain.
The minimum viable unit test looks like this:
@Test
public void thisIsATest() { }By default this tests passes, but it is a pretty meaningless one, so let’s expand it into the most common pattern:
@Test
public void thisIsATest() {
//Arrange
//Act
//Assert
}Here we can pretend that we have some code in the different “sections” of the test. Arrange is the first one where we set up all the things our test needs, if anything. The act section is where we execute the code we are testing, and the assert is where we verify the result.
These sections are also commonly named “given”, “when”, “then” which is exactly the language Spock uses.
Personally I am not overly strict on these sections. They are great defaults and do make things readable, but at a certain point something is so easy to understand that we don’t really benefit from it being so much more readable. Verbosity is not always a virtue. Let’s look at an example from one of our tests:
@Test
public void validateUsername_usernameIsEmailAndEmailIsInvalid_usernameRejected() {
//Arrange
validator.setEmailAsUsername(true);
when(emailValidator.isValid("this is not an email")).thenReturn(false);
val input = user().withUsername("this is not an email");
//Act
val result = validator.validate(input);
//Assert
errorsAssertThat(result).hasCode("username", "error.username.email");
}Seems clean right? Yes, it definitely is. Let’s look at how I have actually written it:
@Test
public void validateUsername_emailIsInvalid_emailRejected() {
when(emailValidator.isValid("this is not an email")).thenReturn(false);
val input = user().withEmail("this is not an email");
errorsAssertThat(validator.validate(input)).hasCode("email", "error.email.invalid");
}Compactness is not a goal a developer should have, but rather we should aim for a balance between verbosity and simplicity. In hindsight maybe this could be a better middle ground:
@Test
public void validateUsername_emailIsInvalid_emailRejected() {
when(emailValidator.isValid("this is not an email")).thenReturn(false);
val input = user().withEmail("this is not an email");
val result = validator.validate(input);
errorsAssertThat(result).hasCode("email", "error.email.invalid");
}The point of this section is not to come to an agreement of what is the proper look of a unit test. Rather that JUnit doesn’t enforce any particular style. It is up to the developer to have her own set of guidelines of what is appropriate and what is not.
Spock
Spock has very strong opinions on what a test should look like and which steps should be included. While JUnit sees an empty method as a valid test Spock doesn’t even attempt to execute it. Spock instead uses labels which is the only place where the developer is allowed to write code:
def "must reject missing retiredReason from a retired user"() {
when:
def errors = validator.validate(user(retired: true))
then:
errors.errorCount == 1
errors.objectName == "john.mikael.gundersen.healthcare.User"
errors.getFieldError("retireReason").codes.contains("error.null")
}In the example above we see two of the labels used:
when: This is where you execute the test code
then: This is where we verify the result. Note that every line within a then block is executed as an assert, which is why in the example above we simply need make boolean checks
We also have expect and where:
def "verify valid usernames"(String username) {
expect:
!validator.validate(user([username: username])).hasErrors()
where:
username << ["", "John", "John-doe", "john_Doe", "John.Doe"]
}expect: Basically functions as a combined “when” and “then” block. Excellent for those one line tests
where: Here we usually put our data for parameterized tests. In this case we are changing the username variable for each test run
There are a few other labels such as “given” and “cleanup”, and there are rules on which order they can appear in. The full documentation can be found here.
Spock is very opinionated when it comes to how it thinks developers should write tests and I like that. While there are no silver bullet solutions for getting developers to write good and clean tests I do think that an opinionated framework with sensible restrictions can go a long way.
Some might look at the JUnit and Spock test example and say that there’s no meaningful difference between either. One can write equally good tests in JUnit and structure it the same way, which is entirely true. The main difference is that Spock forces that structure, while JUnit expects the developer to do the right thing without any hand holding.
Annotations
JUnit
For better or for worse, Java really uses a lot of annotations. Generally I am not a huge fan of annotations. They seem to take things related to the method and place it above the method declaration. This doesn’t mean that we cannot do great things using annotations, and JUnit puts them to good use, but they rarely look nice.
The biggest offender in this case is the @Test tag which has to come before every single test, or else JUnit see it as a test and therefore not execute it.
The other thing that bothers me is the @ValueSource annotation. The functionality of it is great, but it looks ugly:
@ParameterizedTest
@ValueSource(strings = {
")SpecialSymbol",
"anotherSpecialSymbol#",
"username with spaces",
"ThisIsASuperLongUsernameWhoWouldEvenHaveSuchAUsername",
"-usernameStartingWithDash"
})
public void validateUsername_usernameIsNotEmailAndUsernameIsInvalid_usernameRejected(String username) {
val input = user().withUsername(username);
errorsAssertThat(validator.validate(input))
.hasCode("username", "error.username.pattern");
}We will talk more about this annotation in the parameterized tests section of this post where we will also discuss different approaches. Let’s for now just say that the solution above isn’t very pretty.
Spock
Spock removes pretty much every annotation there is. It is clever enough to differentiate between feature methods and non-feature methods. While the Spock framework does contain a few annotations I got away with using none of them in this example.
I personally really like this as I find annotations to be an eyesore, especially with the @ValueSource annotation. In Spock however the tests are usually completely defined within the brackets of the method. The only methods that get called are the ones we explicitly call in the code.
Assertions
AssertJ
While JUnit has a capable assertion API I must admit that I find it a bit clunky. It is not bad, but it doesn’t look very aesthetically pleasing. Therefore I decided to use AssertJ, a library which does fluent assertion really well.
At this point you might be thinking “But aren’t you giving JUnit an unfair advantage by allowing for third party tools such as AssertJ?”, and I hear you but that is not the right attitude. Comparing JUnit to Spock without considering the larger ecosystem is foolish. When we do professional work we should write clean and maintainable code, therefore we should utilize tools which help us in this regard, even third party ones. In this scenario I do think that AssertJ helps me to write cleaner JUnit tests, therefore I should use it.
The main difference between the JUnit assertion API and AssertJ is that AssertJ allows us to chain assertions in a more readable way. Here’s an example taken from the AssertJ documentation:
@Test
void a_few_simple_assertions() {
assertThat("The Lord of the Rings").isNotNull()
.startsWith("The")
.contains("Lord")
.endsWith("Rings");
}One thing which we must acknowledge when using AssertJ is that we are actually doing multiple asserts within one test, yet most people recommend that we aim to have a single assert. In the example above we are actually doing 4 assertions.
This is the main issue I have with AssertJ - it becomes too easy to “hide” assertions in tests which would usually warrant more tests.
We see this happen once in our tests as well:
@Test
public void validate_isRetiredWithoutRetireReason_retireReasonRejected() {
val input = user().withRetired(true);
errorsAssertThat(validator.validate(input))
.hasErrorCount(1)
.withObjectName("john.mikael.gundersen.healthcare.User")
.hasCode("retireReason", "error.null");
}In reality I am not happy with how the test turned out. For one the name doesn’t tell the whole truth. We say that we expect that the field “retireReason” is rejected, yet we also verify that no other errors have been thrown and we verify that the object name.
Another thing to consider is that we are actually performing 3 asserts in this test. Look at this test:
@Test
public void validate_isRetiredWithoutRetireReason_retireReasonRejected() {
val input = user().withRetired(true);
val result = validator.validate(input);
assertEquals(1, result.getErrorCount());
assertSame("john.mikael.gundersen.healthcare.User", result.getObjectName());
assertThat(result.getFieldError("retireReason").getCodes()).containsOnly("error.null");
}Functionally, there’s absolutely no difference between these two examples. They do exactly the same thing. In programming there’s an advice that says we should only have one assertions per test, and this should have made the tests above into 3 tests:
One which verifies that there’s exactly one error
One which verifies that the object name is correct
One which verifies that the correct field is getting rejected
One can even argue that checking that there’s exactly one error isn’t a very valuable test case, so in reality we would probably end up with two tests. The lesson here is that even if the word “assert” shows up only once in a test doesn’t mean that we are only doing one assert.
JUnit’s base assertion API does make it obvious when we’re making more than one assertion, and that is great. AssertJ has a nasty habit of allowing developers to hide assets, but at the same time tests written with AssertJ ends up looking very pretty.
Spock
The main reason I didn’t change the test in the previous example is that I wanted to showcase how Spock deals with multiple assertions:
def "must reject missing retiredReason from a retired user"() {
when:
def errors = validator.validate(user(retired: true))
then:
errors.errorCount == 1
errors.objectName == "john.mikael.gundersen.healthcare.User"
errors.getFieldError("retireReason").codes.contains("error.null")
}Everything in the “then” block is treated as an “assertTrue”, and by having more than one line here does mean that we are running multiple asserts. Spock makes things pretty clean by directly dealing with booleans and not having any form of “assert” keyword whatsoever. While Spock doesn’t use the word “assert” it still manages to make it obvious when we’re doing multiple assertions.
Note that the way Spock does assertions isn’t as smooth or pretty as AssertJ. It might be somewhat less elegant, but at the same time Spock gives us exactly what we see. We have to explicitly state what is true and what isn’t, while AssertJ will hide much of that complexity.
If we really want fluent assert syntax in Spock we can always resort to using Hamcrest, which will give us much of that syntactical flair which AssertJ provides.
In either way there’s no right or wrong approach in this regard. We will have clear and obvious tests if you use the base language or framework assertions. If we want something which reads a little cleaner we can always use an assertion API. We’re free to do whatever we want in either framework.
Parameterized tests
JUnit
Let’s revisit this example:
@ParameterizedTest
@ValueSource(strings = {
")SpecialSymbol",
"anotherSpecialSymbol#",
"username with spaces",
"ThisIsASuperLongUsernameWhoWouldEvenHaveSuchAUsername",
"-usernameStartingWithDash"
})
public void validateUsername_usernameIsNotEmailAndUsernameIsInvalid_usernameRejected(String username) {
val input = user().withUsername(username);
errorsAssertThat(validator.validate(input))
.hasCode("username", "error.username.pattern");
}Doesn’t it look strange when a list of strings within an annotation is bigger than the method which it belongs to? Maybe that is just me, but I do think it looks odd, especially considering that so much of the test is defined outside the method itself. Whether or not this is an actual problem is up for debate, and I am by all means not claiming it to be one, but to me this is not very aesthetically pleasing.
We could have made it prettier by making a custom annotation, using an @MethodSource or even put the code in a file and read it from there. However we are basically moving the ugliness around at that point.If this was a huge dataset, or a dataset which was used by multiple tests, the value of moving it elsewhere would definitely be there. This dataset, however, is only 5 strings long and is used only by this test.
Though it might not be pretty it gets the job done.
Spock
Let’s look at the the previous example, but written as a Spock test:
def "username is not email and is invalid"(String username) {
when:
def errors = validator.validate(user([username: username]))
then:
errors.getFieldError("username").codes.contains("error.username.pattern")
where:
username << [")SpecialSymbol",
"anotherSpecialSymbol#",
"username with spaces",
"ThisIsASuperLongUsernameWhoWouldEvenHaveSuchAUsername",
"-usernameStartingWithDash"]
}Here we see Spock’s strict styling come into play. In Spock we use the “where” tag to indicate that we want to change a test parameter. We can do so by using a data table or reading from files, but here we chose to use the equivalent to @ValueSource which is called a data pipe.
While we cannot say that the string array is pretty at least it is confined within the method itself. The bigger bonus in my opinion is that the test reads much better:
def "username is not email and is invalid"(String username) {
when: "we do this..."
then: "the error code should be..."
where: "the username is any of ..."
}There isn’t any superior way of doing parameterized tests, but I am really liking the elegance which Spock and Groovy provides in this regard. I would not be surprised if Groovy had some neat tricks to make it even prettier.
Writing tests with parameters is very much a trivial task in Spock. I especially like the look of using data tables when dealing with multiple parameters, and IntelliJ does a great job of automatically formatting those tables.
Size
When I first looked at Spock I feared that the strict and enforced way we have to write Spock tests would increase the overall size of the specification/test class. Luckily for everyone, I was very wrong. Let’s compare:
@Test
public void validatePerson_personMissingGender_genderRejected() {
val input = user().withPerson(person().withGender(null));
errorsAssertThat(validator.validate(input))
.hasCode("person.gender", "error.null");
}Here we see the JUnit test taking up 6 lines. In JUnit I think it is only fair to count the annotations seeing as they are very much required.
Here’s the same test in Spock:
def "person missing gender field"() {
when:
def errors = validator.validate(user([gender: null]))
then:
errors.getFieldError("person.gender").codes.contains("error.null")
}Exactly 6 lines as well!
This is a common pattern when I’ve compared JUnit and Spock tests - they are roughly the same size. What this does not factor in is the extra code that I for example wrote in the ErrorAssertThat class to make each test a bit shorter and concise for the JUnit solution.
Generally I don’t view method sizes as an important factor in this comparison. The goal is maintainability and readability, not small sizes.
This Spock example uses the labels “when” and “then” which some might look at as bloat or something being too verbose. Personally I think it is really impressive that Spock still manages to rival JUnit in test length while enforcing its style.
Mocking
JUnit doesn’t have built-in mocking, but that doesn’t mean we shouldn’t talk about it. In something like JUnit we would resort to using a mocking framework like Mockito. To truly consider whether or not Spock is capable we must also look at its mocking capabilities in relation to something like Mockito.
Consider the following JUnit test:
@Test
public void validateUsername_emailIsInvalid_emailRejected() {
when(emailValidator.isValid("this is not an email")).thenReturn(false);
val input = user().withEmail("this is not an email");
errorsAssertThat(validator.validate(input))
.hasCode("email", "error.email.invalid");
}Here we have the when(thisIscalled).thenReturn(thisValue) syntax which is used in Mockito. This is very readable and straight forward. Here is the same test in Spock:
def "user has invalid email"() {
when:
def email = "this is not an email"
emailValidator.isValid(email) >> false
def errors = validator.validate(user([email: email]))
then:
errors.getFieldError("email").codes.contains("error.email.invalid")
}We see that the Spock mock is a bit shorter, but its meaning isn’t as obvious as Mockito’s syntax. Personally I don’t have a favorite. When we work with a framework we should know the special symbols and language used by that framework, so if people are working with Spock I would expect them to understand what “>>” means in this context.
This is obviously not taking anything away from Mockito which reads very well with its “when(this).thenReturn(that)” syntax.
Another thing we can look at is the creation of mock objects. This is how it looks in Mockito:
private EmailValidator emailValidator = mock(EmailValidator.class);We could have annotated the class and have Mockito automatically initialization of any mocks, like this:
@RunWith(MockitoJUnitRunner.class)
public class UserValidatorImplTest {
@Mock
private EmailValidator emailValidator
...To take it a step further we can use the @InjectMocks annotation to automatically create our UserValidatorImpl object.
For bigger tests with many dependencies it can clean up the test class a little bit by having automated mocks and injections. To my knowledge Spock doesn’t have automatic injections, but that is usually not a huge problem. It is a “nice to have”, but not a requirement.
The end result is that Spock’s mocking framework is very capable and I have absolutely full faith that it can pull off whatever I need it to do. It may lack some of the minor features which Mockito has, but as a whole package I must say that Spock does very well.
Dependencies
JUnit
With the JUnit solution we have so far pulled in quite a few dependencies to get the tests in the way we want them:
JUnit, because we can’t really write JUnit tests without it
AssertJ, for pretty asserts
JUnit params, so that we can make parameterized tests
Mockito, for mocking
Lombok, for sensible object creations
That we have quite a few dependencies to make our JUnit tests isn’t necessarily a blow against JUnit. Rather it shows a healthy ecosystem of libraries and frameworks working together to make my tests better. Most of these dependencies are entirely optional, but they are really good at what they do. AssertJ is really good at assertions. Mockito is really good at mocking.
One might look at the list of dependencies and say that JUnit isn’t good enough if one needs to pull in these dependencies to even write nice tests. My counterargument to that is that it also frees us up to pick and choose which specialized framework we want to use. If we don’t the asserts already built into JUnit we can use Truth, but we can also use AssertJ. If neither of those are to our liking we can always use Hamcrest. The same goes for mocking frameworks.
Spock
Spock required very few dependencies:
Spock, for writing Spock tests
Gmavenplus, so that we can compile our groovy code
There are a couple of optional things which I have specified in my project, but that’s not strictly needed:
The Groovy language
JUnit
Spock comes with these dependencies already, but if we want to specify the version of these dependencies we must include them in the project..
A valid argument is that Spock brings more invasive dependencies with it, like Groovy. After all, a whole programming language is probably going to contain more stuff than an assertion API. While this is entirely true, in a world of build tools such as Gradle or Maven I’m not super worried about it.
What did surprise me was how little I had to do to make my Spock tests pretty, or at least on par with my JUnit tests. I did say that one of the strengths of JUnit is that it is pretty easy to include new systems within it and I still stand by that, but then Spock’s strength must be that I never felt the need to include anything else.
Conclusions
There’s a lot of things I didn’t cover in this post. I didn’t talk about how we can verify calls to our mocks, nor did I talk about how we can extend our tests to interface with stuff like databases or APIs and serve as integration tests. I have done these things with JUnit/Mockito, and by reading the Spock documentation I’m confident that Spock can handle these things as well.
Personally I think Spock is an absolutely fantastic test framework and I really do love using it. I am not saying that Spock is superior to JUnit, but I am saying Spock falls very much in-line with my values and ways of thinking. Given the chance I will probably write my tests in Spock from now on.
“But John” you say, “You said that this wasn’t about picking a winner!”. Yes dear reader, you’re completely right, which is why I haven’t declared a winner. There simply isn’t a “better” framework. There’s a lot of factors playing into which test framework is appropriate for which project. Sometimes developers want to stay with a single language. Sometimes adding a new language can cause problems. Migrating tests might be more difficult if you have to re-write them in Spock. Maybe the team uses some technology which interfaces really well with a specific test framework. As I said previously, it is not about picking a winner, but picking a preference - and right now I am preferring Spock.
That I really like Spock doesn’t make it functionally superior to JUnit, as JUnit can be just as effective, if not more if one matches JUnit with other frameworks and dependencies. The choice between these two test frameworks doesn’t usually come down to the functionality, as both of them are excellent. Rather it comes down to preference from the developers and suitability for a given project.
The Spock tests
package john.mikael.gundersen.healthcare
import spock.lang.Specification
class UserValidatorImplSpec extends Specification {
EmailValidator emailValidator = Mock()
UserValidatorImpl validator
def setup() {
validator = new UserValidatorImpl(false, emailValidator)
}
def "must reject missing retiredReason from a retired user"() {
when:
def errors = validator.validate(user(retired: true))
then:
errors.errorCount == 1
errors.objectName == "john.mikael.gundersen.healthcare.User"
errors.getFieldError("retireReason").codes.contains("error.null")
}
def "user has multiple errors"() {
when:
def errors = validator.validate(
user([gender: null, retired: true, username: "username with spaces"])
)
then:
errors.errorCount == 3
}
def "a user without person object must be rejected"() {
when:
def errors = validator.validate(user([person: null]))
then:
errors.getFieldError("person").codes.contains("error.null")
}
def "a person without the gender field must be rejected"() {
when:
def errors = validator.validate(user([gender: null]))
then:
errors.getFieldError("person.gender").codes.contains("error.null")
}
def "person without the dead field must be rejected"() {
when:
def errors = validator.validate(user([dead: null]))
then:
errors.getFieldError("person.dead").codes.contains("error.null")
}
def "person without the voided field must be rejected"() {
when:
def errors = validator.validate(user([voided: null]))
then:
errors.getFieldError("person.voided").codes.contains("error.null")
}
def "person without a name must be rejected"() {
when:
def errors = validator.validate(user([personName: null]))
then:
errors.getFieldError("person").codes.contains("Person.names.length")
}
def "empty string for username must be rejected"() {
when:
def errors = validator.validate(user([personName: ""]))
then:
errors.getFieldError("person").codes.contains("Person.names.length")
}
def "must reject illegal usernames"(String username) {
when:
def errors = validator.validate(user([username: username]))
then:
errors.getFieldError("username").codes.contains("error.username.pattern")
where:
username << [")SpecialSymbol",
"anotherSpecialSymbol#",
"username with spaces",
"ThisIsASuperLongUsernameWhoWouldEvenHaveSuchAUsername",
"-usernameStartingWithDash"]
}
def "must accept legal usernames"(String username) {
expect:
!validator.validate(user([username: username])).hasErrors()
where:
username << ["", "John", "John-doe", "john_Doe", "John.Doe"]
}
def "must accept valid usernames"() {
when:
validator.emailAsUsername = true
def email = "john@test.com"
emailValidator.isValid(email) >> true
then:
!validator.validate(user([username: email])).hasErrors()
}
def "must reject username when email is username and the email is invalid"() {
when:
validator.emailAsUsername = true
def email = "this is not an email"
emailValidator.isValid(email) >> false
def errors = validator.validate(user([username: email]))
then:
errors.getFieldError("username").codes.contains("error.username.email")
}
def "must reject email when not null and invalid"() {
when:
def email = "this is not an email"
emailValidator.isValid(email) >> false
def errors = validator.validate(user([email: email]))
then:
errors.getFieldError("email").codes.contains("error.email.invalid")
}
static User user(Map args = [:]) {
def map = [retired: false, username: "bob", email: null, person: person(args)] << args
User.builder()
.retired(map["retired"] as boolean)
.username(map["username"] as String)
.person(map["person"] as Person)
.email(map["email"] as String)
.build()
}
static def person(Map args = [:]) {
def map = [dead: false, gender: "Male", personName: "John Doe", voided: false] << args
Person.builder()
.dead(map["dead"] as Boolean)
.gender(map["gender"] as String)
.personName(map["personName"] as String)
.voided(map["voided"] as Boolean)
.build()
}
}The JUnit tests
package john.mikael.gundersen.healthcare;
import lombok.val;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import static john.mikael.gundersen.healthcare.asserts.ErrorsAssert.errorsAssertThat;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
public class UserValidatorImplTest {
private EmailValidator emailValidator = mock(EmailValidator.class);
private UserValidatorImpl validator;
@BeforeEach
public void init() {
validator = new UserValidatorImpl(false, emailValidator);
}
@Test
public void validate_isRetiredWithoutRetireReason_retireReasonRejected() {
val input = user().withRetired(true);
errorsAssertThat(validator.validate(input))
.hasErrorCount(1)
.withObjectName("john.mikael.gundersen.healthcare.User")
.hasCode("retireReason", "error.null");
}
@Test
public void validate_hasMultipleErrors_multipleErrorsReturned() {
val input = user()
.withRetired(true)
.withPerson(person().withGender(null))
.withUsername("username with spaces");
errorsAssertThat(validator.validate(input)).hasErrorCount(3);
}
@Test
public void validatePerson_personIsNull_userRejected() {
val input = user().withPerson(null);
errorsAssertThat(validator.validate(input))
.hasCode("person", "error.null");
}
@Test
public void validatePerson_personMissingGender_genderRejected() {
val input = user().withPerson(person().withGender(null));
errorsAssertThat(validator.validate(input))
.hasCode("person.gender", "error.null");
}
@Test
public void validatePerson_personNeitherDeadOrAlive_genderRejected() {
val input = user().withPerson(person().withDead(null));
errorsAssertThat(validator.validate(input))
.hasCode("person.dead", "error.null");
}
@Test
public void validatePerson_personVoidedStatusMissing_voidedRejected() {
val input = user();
input.getPerson().setVoided(null);
errorsAssertThat(validator.validate(input))
.hasCode("person.voided", "error.null");
}
@Test
public void validatePerson_personNameIsNull_nameRejected() {
val input = user().withPerson(person().withPersonName(null));
errorsAssertThat(validator.validate(input))
.hasCode("person", "Person.names.length");
}
@Test
public void validatePerson_personNameIsEmptyString_nameRejected() {
val input = user();
input.getPerson().setPersonName("");
errorsAssertThat(validator.validate(input))
.hasCode("person", "Person.names.length");
}
@ParameterizedTest
@ValueSource(strings = {
")SpecialSymbol",
"anotherSpecialSymbol#",
"username with spaces",
"ThisIsASuperLongUsernameWhoWouldEvenHaveSuchAUsername",
"-usernameStartingWithDash"
})
public void validateUsername_usernameIsNotEmailAndUsernameIsInvalid_usernameRejected(String username) {
val input = user().withUsername(username);
errorsAssertThat(validator.validate(input))
.hasCode("username", "error.username.pattern");
}
@ParameterizedTest
@ValueSource(strings = {"", "John", "John-doe", "john_Doe", "John.Doe"})
public void validateUsername_usernameIsNotEmailAndUsernameIsValid_noErrors(String username) {
val input = user().withUsername(username);
assertThat(validator.validate(input).hasErrors()).isFalse();
}
@Test
public void validateUsername_usernameIsEmailAndEmailIsValid_noErrors() {
validator.setEmailAsUsername(true);
when(emailValidator.isValid(any())).thenReturn(true);
val input = user().withEmail("john@test.com");
assertThat(validator.validate(input).hasErrors()).isFalse();
}
@Test
public void validateUsername_usernameIsEmailAndEmailIsInvalid_usernameRejected() {
validator.setEmailAsUsername(true);
when(emailValidator.isValid("this is not an email")).thenReturn(false);
val input = user().withUsername("this is not an emIail");
errorsAssertThat(validator.validate(input))
.hasCode("username", "error.username.email");
}
@Test
public void validateUsername_emailIsInvalid_emailRejected() {
when(emailValidator.isValid("this is not an email")).thenReturn(false);
val input = user().withEmail("this is not an email");
errorsAssertThat(validator.validate(input))
.hasCode("email", "error.email.invalid");
}
private User user() {
return User.builder()
.retired(false)
.username("Bob")
.person(person())
.build();
}
private Person person() {
return Person.builder()
.dead(false)
.gender("Male")
.personName("John Doe")
.voided(false)
.build();
}
}