How can we automatically check if our game code works properly?
This article is also available on Medium.
This tutorial is available either in video format or in text format – see below 🙂
Get the scripts and assets for this tutorial on my Github 🙂
In this article, we’re going to see what unit testing is and how we can use the Unity Test Framework package to run our C# unit tests.
Disclaimer: before we dive into this video, however, a big shout out to Andrew, a very kind viewer who helped me come up with the topic of today’s episode. If you too want to suggest your own ideas for the next tutorial, feel free to leave a comment! 🙂
And with that said – let’s jump in!
About unit testing
Alright: first things first, let’s recall some definitions and discuss what unit testing is about.
There are those who go as far as saying that:
A code that cannot be tested is flawed.
Despite perhaps being a bit extreme, this quote does highlight something crucial in programming: testing, although hard and not always fun, is absolutely essential in programming.
But yes: validating that the functions you wrote indeed do what you intended is not a trivial task because, more often than not, they’ll all collaborate to a grand behaviour and it will be pretty hard to isolate their individual contributions.
Unit testing tries to mitigate this issue by focusing on writing tests at the most granular level: the point is to split your codebase into logical tiny units and test each on its own to be sure that at least this part of your project is sound.
Before we talk of actual code, let’s take a look at a basic example, such as this math formula with matrices:
Lots of programming languages have ready-made packages for linear algebra that allow you to do sums, multiplications and unary operations like taking the opposite. But what if you didn’t have this toolbox at your disposal? What if you were tasked with implementing the “add”, “multiply” and “negate” functions for matrices yourself?
Once you’ve written some code to properly compose the inputs together, you’ll probably want to check that your methods indeed work. Otherwise, you run the risk of building the rest of your algebra on pretty shaky ground…
And so, at that point, you may enter the world of unit testing. The idea here would be to code, parallel to your actual logic implementation (i.e. the “add”, “multiply” and “negate” functions) a series of matching test functions that verify the outputs of each of your methods.
To do these checks, you would take a few representative (and hopefully diverse) example cases and assert that, in each case, you actually get the expected result.
Why unit testing is interesting
The nice thing is that, because we are at a very granular level, it isn’t too complex – checks are quick and straight-forward to write.
Because they are often pretty nice to read, too, unit testing can be a kind of a live documentation for your code.
And of course, by checking that these low-levels operations work, you insure that you have solid foundations for the rest of your project.
But, maybe most importantly, unit testing is key to avoiding regressions in your code. In layman’s terms, code regression is when you wrote this unbelievably awesome function yesterday and it just worked perfectly; but then, after your coffee this morning, you added a little line of code that has nothing to do with it and, nonetheless, you basically broke everything. So it’s when something that once worked has now stopped working properly because of a refactor, a new feature, a system upgrade, a change in the process environment… or other crazy factors you didn’t think of.
While being a real thorn in any developer’s side, regressions are pretty common because, as you engage on more ambitious projects, you will obviously deal with more intricate and involved code. Programming a big network of services and making sure they all run smoothly is no piece of cake, but maintaining such a network and making it robust to changes, new features and large refactors is even harder!
And, to me, that’s where unit testing really shines 🙂
Unit testing and game developers
But now, here’s a question for you: how often do you hear of unit testing in the gamedev community? That’s pretty rare, right? It might be just my experience, but it feels like these goals of maintenance and robustness are more a concern for devops and software engineers than game developers.
On the other hand, unit tests do seem pretty interesting for games as well, given their benefits in terms of solidity, regression testing and iterative development!
So, today, let’s talk a bit about how we can at least start doing some basic unit testing with Unity, relying on their Test Framework.
Unity’s official Test Framework
Installing & getting the basics
This package is very easy to add to your project, thanks to Unity’s package manager. Simply open the package manager window, search for the Test Framework module and click “Install” – or simply check that it already is! 🙂
Once it’s added to your project, you’ll see that you have a new window available called “Test Runner”:
The first thing you need to understand when doing unit testing is the concept of assemblies, because unit testing with the Test Framework relies heavily on defining assemblies for your test suites. Assemblies are basically a way of grouping together several scripts and telling the compiler that they are a coherent whole. It helps organise the code, create dependencies and reduce your compile time – because you’ll be able to limit the recompile to only a selected portion of your assemblies.
The Test Framework makes it easy to automate the setup of your project for unit testing: if you click the “Create TestAssembly Folder” button in the Test Runner window, you’ll directly get a brand new folder and associated assembly definition asset for your tests. Note that the package allows you to run tests both in edit mode and play mode, but that each mode requires its own assembly definition asset.
After you’ve created your assembly definition, you can click on it to inspect it and, in particular check out its dependencies. Test assemblies should have dependencies to two or three things:
Nunit framework (that’s the go-to C# unit testing lib, it’s a unit testing framework for all .Net languages)
3. and the
UnityEditor.TestRunner if your assembly is for edit mode testing
Below, you’ll also have the list of platforms this assembly will be compiled for. Test assemblies for play mode can use the various standalone platforms but, again, if you’re aiming for edit mode, be careful to only check the Editor platform in here 🙂
Now that you’ve prepared your assembly, you’re ready to actually add some tests inside it!
To do this, you can go back to the Test Runner window and click the “Create Test Script” button. This should add a new C# script in the same folder as your assembly definition asset that you can rename if you want.
As soon as it’s compiled, you’ll see the test listed in the Test Runner window. This panel shows us how unit testing is usually organised as a hierarchy of nested elements: the test suite (in red), optionally the test class (in orange) and the test cases (in yellow).
Those test cases are where the actual testing code is written, they’re the core of the unit testing process. The higher levels are for organising your tests, setting up some specific context or defining prerequisite steps.
Diving into the C# scripts
Ok, now, let’s actually dive into this test script and see how to write our tests!
As you can see when you open the new C# script (here: my
MyTestScript class), the list of test cases we saw in the Test Runner window earlier comes from the different methods that are in our test class. Each of these functions has one of the
UnityTest method attributes – so these attributes are how your mark methods for the test framework to be looked up and run as part of the test suite.
The package example file also shows us how to run simple functions or coroutine functions with enumerators. Note however that the package doesn’t yet support async processes.
The fundamental keyword among unit testers is: “assert”. This word basically means: “check that this condition is true, else raise an error and stop there”. It’s the goal of any test case: you need to validate one or more asserts. If every assert in your method passes, then the test case passes; else it fails. Similarly, a test suite only passes if all test cases inside pass.
For now, this code will automatically pass, because we’re not really testing anything – we don’t have any asserts in our test cases methods!
If we go back to the Test Runner window and click the “Run all”, or “Run selected” buttons, you see that we instantly get green checks next to every test case and test suite because all asserts have passed:
But now, what if we change the code inside one of the test cases to get an error? Something like: “check that 1 equals 2”, for example?
To do this, we simply need to use the Assert class that has a bunch of functions do very common comparisons like “equal”, “not equal”, “less”, “more” and so on:
Then, if I re-run this specific test, I’ll get a red icon because the case has failed – and so the suite did too!
Of course, if we want an actual real-life scenario, we could easily take our previous example of matrix operations and re-code it in C#, to check that our code is indeed ok via unit testing…
I want to point out, however, that the granularity of test cases isn’t an absolute and quantifiable thing. Some people will write only one assert per test case, others will group the similar asserts; my opinion is that, in the end, unit testing can use a bit of creativity and personal habits – as long as it does test your code properly!
Mock data & stubs
Something crucial with unit testing, however, is that you should always be very careful of what you’re testing, and in particular how much you’re testing indirectly. For example, let’s say you want to know whether your XML parsing method works properly, and so you write a test case for this XML parsing process. Then it might sounds obvious, but you better make sure that the input string you pass your test case is indeed valid XML… and so you should avoid generating it with one of your own functions, because that function could potentially have some errors, too!
Instead, you’ll usually want to rely on mock data, stub functions and other placeholders to “simulate” prior steps while insuring the output from these previous stages is valid. This way, you’re sure that your test case is testing the method C, and not the methods A or B that would normally run previously and give you the input for C 🙂
Roughly put, mock data and stubs are very instrumental in breaking possible dependencies and error transfers.
Automation & CI/CD
Another neat consequence of having unit tests is that those can be run automatically and you can thus validate your code programmatically whenever you make a change.
In traditional software engineering, this naturally leads to the idea of CI/CD, or continuous integration and continuous development. This is the process of automating the build, validation and delivery of your code so that you don’t have to manually repeat these steps each time you publish a new version.
They are very common features of online Git providers like Github, Bitbucket or Gitlab. But honestly, in that domain, game tools such as Unity aren’t always the best. Fully automating the deployment of your Unity game from start to finish isn’t easy to do – plus building for multiple platforms is usually hard, you actually need to have the target hardware to build for it!
In any case, unit testing is a necessary step in any CI/CD process because it insures your new publish doesn’t cause regressions, bugs, unwanted side-effects on other features and so on.
Finally, before I end this video, I want to quickly discuss a code philosophy commonly associated with unit testing: test-driven development, or TDD. In a nutshell, the idea of this design process is to start by writing the tests (i.e. the intended behaviour) and only then write the actual logic implementation.
This can avoid a pitfall lots of devs discover too late with unit testing: writing tests to make them pass… Even if you don’t realise it, if you’ve spent weeks polishing your implementation, you’ll often be quite lenient with testing because you just want everything to pass and validate that you’re a good programmer 😉
TDD reduces the risk of this happening because, when you write the test, you haven’t yet developed any emotional attachment to the feature!
So – to add a new feature using test-driven development, you’ll follow these steps:
1. design the feature, or in other words decide what the code should do
2. then write the test with the proper asserts to check for your expected behaviour
3. run the test and validate that it fails: at this point, since you haven’t written any code yet, if it passes it means that either your feature is improperly defined or your test is not written right
4. code the features
5. re-run the test… and hope that it passes, this time!
Of course, you might need to re-iterate the last two steps until the test actually passes… but all in all, this philosophy can make development faster because it benefits from the perks of unit testing: less regression, autonomous and readable granular pieces of logic, iterating on your features, etc.
Just like unit testing, although TDD is a classic of software development, from what I’ve seen, game teams rarely use it. And sure, it can be a bit slow at first because you have to setup some structure and processes. But it can also bring a lot of robustness and quality to your code, and it can bring a lot of peace of mind to your devs, so it might be worth looking into for your next project!
And that’s it for today, folks! In this tutorial, we saw what unit testing is, why it’s important and how we can setup a basic testing process in Unity, with the official Unity Test Framework.
I hope you enjoyed this episode – don’t hesitate to share your ideas for future topics you’d like me to make Unity tutorials on!
Also, when writing this post, they were a couple more things I wanted to discuss but I thought were too far-fetched to be in the tutorial itself. So here are some further references and math extras for the curious 🙂
1. If you’re interested in this idea of “re-defining the basics of maths”, you might want to take a look at the notion of algebraic structures. Those are core math objects that basically start with fuzzy and unconstrained sets of operators and operations, and then as you add more rules and particularities build up into the sets of elements we know and love like the integers, the reals, the complex numbers or even geometrical transformations 🙂
2. The canonical testing hierarchy, with test suites and test cases, is interesting from the angle of dependency, too. In theory, you should be able to run two test cases in parallel without them conflicting with each other. This means that, quite often, when using unit testing, your logic should be as self-contained as possible – and so you’ll quickly lean towards functional programming (and its pure functions).
3. My example of a basic matrix C# class relies heavily on operator overloading (that you can see in detail in the Github repository 🚀), as well as explicitly throwing some exceptions – I might make some additional tutorials on these topics to dive into this a bit more!