Discover how to get detailed reports on your Unity unit tests during dev or deploys!
This post was originally published on Codemagic’s blog.
A few weeks ago, I wrote an article about unit testing Unity projects – I discussed why testing is so valuable for game devs and I showed how to re-create a basic Pong game with some unit tests. And we also took a look at how to automate the testing and deployment of our Unity project into a CI/CD pipeline with Codemagic 🙂
If you want to take a look at the code of the sample project, it’s available on Github! 🚀
Unit tests are a nice way of insuring you deliver and deploy valid code and you’re not shipping errors… but what about times when you actually do have some errors? How can you know what needs fixing? Having an overall “passed” or “failed” state isn’t enough for in-depth debugging! You need to know which exact test has failed, so that you know where to look.
That’s why when you work with unit tests, you usually put in place what is called test reporting to continuously get the detailed list of test suites and test cases. This allows you to quickly check where the errors come from and why your pipeline failed. And this is what we’re going to discuss in this article.
So, today, let’s see:
- how to do test reporting during development, directly in Unity,
- and how to show and download your test results from your Codemagic workflow for checks during the deployment phase!
Unity test reporting in Unity editor
In the previous article, I talked about the Unity Test Framework official package and I shared a few code snippets which showcased how to use it for our Pong game. But the nice thing with this tool is that it also comes with built-in reporting inside the Unity editor 🙂
Remember the Test Runner window that appeared after we installed the Test Framework package?
Once you’ve created your test suite with an Assembly Reference asset and populated it with some test cases (or even if you just create a default test suite with the tool’s auto-generator), your Unity editor will recompile the assembly and instantly present you with the list of test suites and cases in this window:
Here, you see the usual hierarchy of unit tests:
- in red: the test suite, the root level at which all of your tests live,
- in orange (optional): the test class, an intermediary level to create logical units in your codebase and better organise the code,
- in yellow: the test cases, the actual “testing functions” that contain your asserts.
Note: this hierarchical structure is useful, because, most of the time, you’ll get “grouped” statistics at the medium level and you’ll therefore have an easy-to-catch bird’s eye-view of your various unit “blocks”. In the case of our Pong game, we’d quickly get an overview of what works and what doesn’t for the Paddle-related functions, the Ball-related functions and the Score-related functions:
At that point, you can use the “Run all” or “Run selected” buttons at the top to directly have the tool go through all your test cases… and give you a live report of the results!
The Unity Test Framework will show you the usual test reporting info: the total number of tests, how many passed or failed, which ones, etc. You can even select your tests to see at which line they crashed and which assert failed!
The tool also tells you how long each test took to run and total duration of the suite and the run.
Unity test reporting in CI/CD
Checking that your tests work inside the Unity editor is sweet, but ultimately you’ll probably be building and running your project in quite a different environment: your Unity exports can be for another OS, a mobile platform, WebGL, and so on.
So even though everything was OK in the dev environment, you’d better make sure that you properly monitor the production code, too, and that you have enough info on the deployed version.
Just like last time, let’s take advantage of Codemagic to deploy our Unity app from our Git repository, and let’s learn how to add test reporting easily to our Unity apps on Codemagic…
Preparing the test reporter: converting NUnit to JUnit
If you take a look at the Unity Test Framework’s docs, you’ll see that when you run the tool from the command line, you can pass an additional option to the executable:
-testResults. This parameter allows you to specify the path for the test report, as an XML file.
For example, for my basic Pong game, Unity will get me an XML file that looks like this:
The problem is that this XML file is not structured properly for common test report prettifiers, such as the one Codemagic uses! By default, the package outputs the results in the NUnit format, whereas most CI/CD tools (like Jenkins or Codemagic for example) prefer another XML format, the JUnit XML reporting file format:
As you can see, this format uses different tags and attributes than the one the Test Framework creates. This means we can’t rely on the built-in feature to output our test results into a valid XML format.
After searching for a while on the net, I haven’t actually found any XML formatter from the Unity Test Framework
ITestAdaptor result C# object to the JUnit-XML format… so I decided I would code one myself 😉
The idea was simply to take the JUnit specifications I linked above and code a little C# function to produce the proper XML document from the tree-structure of my Unity Test Framework results. I just had to make sure to skip the intermediary global Unity assembly level because the JUnit XML format doesn’t allow for nested test suites, but other than that, it was mainly about matching the attributes to the right field in the
ITestAdaptor object and its sub-children:
Note: keep in mind that this code works well for my simple project, but it is fairly simple and limited – so you might need to adapt and/or extend it if you work with more advanced unit tests.
So, now I have a very basic way of creating an XML document from my test results! I’ll integrate this new function in my
Runner script and call it when my test run is finished if the user passed a specific argument to the executable,
-testsOutput, to specify the path of the XML report:
My project is now ready for test reporting inside a Codemagic workflow – so let’s jump to the configuration of our app via its
codemagic.yaml file 😉
Configuring Codemagic for our Unity app
I won’t detail all the steps on how to create a Codemagic app for a Unity project in this post: be sure to check out the previous article (the “Setting up Codemagic CI/CD” section) for more info on that 🙂
Important note: at the time of writing this, deploying Unity apps with Codemagic requires a special Codemagic account and a Pro Unity License – don’t hesitate to contact Codemagic to get started!
If you’ve followed the steps from the first tutorial, or if you already have a Unity app set up with Codemagic, then you’ll have a configuration file for your Codemagic workflow that looks something like this:
codemagic.yaml file should be placed at the root of your Git repository, and it will tell the Codemagic workflow runner which automation steps to take to validate, build and deploy your project. It basically sets up the right working environment for the workflow (in particular it imports or defines the necessary environment variables), then it runs the unit tests (and stops immediately if they fail), then it builds the project for the given target platform, and finally it deploys the generated artifact.
Now, let’s add just a little bit of configuration to this pipeline so that our
Runner script creates a JUnit-compatible XML report (thanks to our brand new
-testsOutput command-line argument) and that the Codemagic UI then displays it in a user-friendly way.
All we need to do is define a new environment variable with the path to our test report file, and update the “Run Unit Tests” script block in the middle.
- first, we’ll define
environmentsection and set it to “tests.xml” – that’s where the test results will be written to, using our handmade JUnit-XML reporter.
- then, we’ll add the
-testsOutputoption and use this environment variable to actually produce the report with our
- finally, we’ll use the
test_reportworkflow parameter and pass it this
TEST_REPORT_PATHreference so that our results can be pretty-printed it in our Codemagic build page.
And that’s it! Let’s commit these changes and run a new build on Codemagic using this new codebase: after the workflow is finished, we can look at the “Run Unit Tests” step and switch to the “Results” tab to see our test report:
Now Codemagic is reporting on unit test for our Unity app correctly: the test report is nice and clean, and it gives us a detailed description of the unit tests with some global stats (on the left) as well as each test case name and status.
As you can see, I didn’t bother re-creating nodes for the different suites and just grouped everything under the same block, so I don’t have a separation between “Paddle” and “Ball” tests for example – but we could totally rework the
Reporter class to handle this and further improve the display 🙂
Bonus: adding the Unity unit test report as an artifact
To wrap this up, let’s make another small improvement to our Codemagic workflow and make our XML test report an artifact of the pipeline. This can be useful if you want to have a local copy of the data, or if you need to share it with your teammates without actually redirecting them to the Codemagic app dashboard…
To make the XML report available as an artifact, we just need to add it to the
artifacts block of our
codemagic.yaml configuration file:
Let’s push this in a commit and re-run a new build, then download the artifacts that are packaged as a separate zip archive at the end of the workflow:
If you uncompress this archive and take a look inside, you’ll see it contains our “tests.xml” file with all the test report info! 🙂
To be honest, I wish the Unit Test Framework had some built-in feature for exporting JUnit-XML-compatible test reports, because it’s a common format for CI/CD. But all in all, making this handmade JUnit-XML reporter was really fun: I found it very interesting to go back to the basics and build it gradually based on the JUnit specs.
As I said, we could obviously improve it to handle more complex test suites and to make better reports of errors… but that’s a topic for another time! 😉
Don’t forget: you can get the sample “unit tested” Pong project with the test reporting and the
codemagic.yaml file on GitHub over here 🚀