Unit testing automation with Unity/C# and Codemagic

Learn to automate the validation and the deployment of your Unity apps!

This post was originally published on Codemagic’s blog.

The more you work on larger Unity projects, the quicker you see that there are areas of expertise that are fundamental to building a solid, long-term project, which you may have not considered while working with smaller Unity projects. In particular, unit testing is a must, although it’s complex and sometimes a bit boring. With unit testing, by splitting your codebase in tiny logical units and checking that each work at a granular level, you insure solid foundations for your app and you avoid code regressions.

But setting up unit tests and having them run automatically can be quite tricky in Unity. So today, let’s discuss how to create and automate Unit tests in Unity using a CI/CD pipeline built with Codemagic.

So, in this article, you’ll learn:

  • Why automated unit tests are necessary for effective agile development.
  • What are the problems with setting up unit test automation in Unity.
  • How to create a basic Pong game that we’ll be using as a sample project.
  • How to set up unit tests for our Pong game.
  • How to set up Codemagic to automate unit testing and building for Unity projects.

Let’s go!

Devops and automation

“DevOps”, which is a concatenation of “development” and “operations” (“ops”), is a fuzzy word that can describe various things: a team (the DevOps Team), a set of practices, a philosophy… All in all, it’s about taking the best out of the worlds of development and IT to produce better code and deliver releases easily and continuously.

DevOps is a common philosophy for web developers, for example. When you work on a big online platform, you’ll need to regularly ship new versions and upgrade the software for your customers. Since it’s an online service, you basically need to replace the source code somewhere on a computer (we all know that any cloud is just someone else’s computer) with the latest updates to publish your new release.

But you naturally want to avoid publishing a large bug, or blocking your infrastructure because of a bad line of code. And given how often you’ll want to release new things (be it a huge refactor or a very small bug fix), it would be best to have all the testing for bugs and publishing to be done automatically, right?

That’s the whole point of one of the core concerns of DevOps: continuous integration and continuous delivery, or CI/CD. These are key processes in traditional software engineering that aim at automating the delivery of your product alongside the version control tools… while making sure that this delivery won’t cause any issues, result in downtime and require quick fixes.

That’s why unit testing is a fundamental concept for DevOps and automation: it is one of the key blocks in this chain of automation because it insures that the code you release is valid and won’t crash the product upon release. So your CI/CD process usually integrates unit testing somewhere as a required step. Basically, if the unit tests fail, then the delivery process is interrupted and the bugged version doesn’t get released.

Unity and automation

The problem with Unity’s barebones CI/CD tool

However, I personally think that Unity is not really optimised for automating delivery. The DevOps philosophy is not that readily available when working with this engine.

To be fair, there are now tools built in the Unity engine for this, namely Unity Cloud Build, that try to fill this gap and answer people’s needs for automating Unity projects. But it has its limitations. In particular:

  • Unity Cloud Build doesn’t allow for publishing. Quite often, your CI/CD pipeline ends with the actual delivery of your update to a target device, or an online store (like Google Play or the App Store). Unity Cloud Build can’t do that, and you still have to take care of this final step manually!
  • The tool is also pretty slow: even if there are some caching options, Unity Cloud Build is clearly not optimised for speed at the moment.
  • Also, Unity Cloud Build is not that great with Git (partly because Unity relies more on Perforce that is usually better for sharing digital assets… but is also less used by common developers!)
  • Finally, Unity Cloud Build works for pure Unity projects, so if your project combines Unity with, say, React Native or other framework, you won’t be able to use Unity Cloud Build for it.

All of this makes the Unity Cloud Build a nice but still imperfect choice for Unity CI/CD. It will help you with automation only to a certain degree, and it might enforce some pretty “edge-casy” habits…

How Codemagic can help

On the other hand, if you don’t want to deal with these various difficulties, a possible alternative to Unity Cloud Build is Codemagic!

This online tool allows you to quickly setup CI/CD by connecting your Git repositories and taking just a few configuration steps. It’s also a nice way of getting access to specific hardware: with Codemagic, you can easily build your game for Mac, iOS, Windows, Android, etc. The service also scales well in terms of price and makes it simple to publish your project on the App Store or Google Play.

So, today, let’s see how to use Codemagic to setup some automation for a sample Pong-like Unity game.

A basic example: re-creating Pong

Note: the code for this example project is available on Github! 🚀

Now that we’ve discussed the issues and possible solutions for CI/CD with Unity, and more specifically the importance of unit testing in this automation process, let’s take a basic example and create a sample Unity project that we then “automate” using to Codemagic. We are going to re-create the famous Pong game – with unit tests!

Coding a basic Pong game

For this project, the point is not to invent a brand new game and discuss game design. Rather, I’ll take a very well-known game, Pong, and re-code it partially; then, I’ll use unit testing to check that it works properly.

The main idea for this game is to have:

  • a PaddleManager C# class to control each paddle (with <W> and <S> for the left paddle, or the up/down arrow keys for the right paddle); their movement will be constrained to the camera view.
  • a GameHandler and a GameManager that create the paddles, keep track of the score and respawn the ball with a random velocity. I decided to separate my core functions to a GameHandler class (which is not a MonoBehaviour but a vanilla C# class) to make it easy to run them in my unit tests!
  • a ScoreTrigger that is applied to 2D trigger colliders on the left and right borders of the screen. This checks if the ball gets out of bounds – if it does, then the opponent gains 1 point and the ball is respawned.

The game will run forever once started, so I won’t prepare any additional menu or pause feature and simply focus on these core mechanics of the Pong game. For the various bounces and collisions, I’ll rely on Unity’s 2D physics and prepare a scene with the following elements:

  1. two paddles (2D rectangles with a BoxCollider2D component and the PaddleManager script);
  2. the ball (a 2D circle with a CircleCollider2D and a Rigidbody2D component);
  3. two physical walls at the top and the bottom of the screen (to constrain the ball to the camera view and have it “bounce” off the borders of the screen);
  4. two triggers on the left and the right of the screen (invisible 2D elements each have a BoxCollider2D component with the isTrigger flag enabled and the ScoreTrigger script attached).

Installing Unity’s Test Framework package

When you want to do unit testing in Unity, you can take advantage of user-friendly tools that are provided by the official Unity Test Framework package. This is a module by the Unity team that gives you some neat debug interface and code shorthands to write C# unit tests easily. It is very easy to add to your project – open the Unity package manager window, search for the Test Framework module and click “Install” (or even just check that it already is installed).

Once it’s added to your project, you’ll see that you have a new option available under Window called Test Runner:

Creating test assemblies and test scripts in Unity

In this window, you can easily create new assemblies to group your tests together and get the right dependencies. Once you’ve create your assembly definition asset, you can also add a new C# test script with some example test cases by clicking on the “Create Test Script” button.

In my case, I will write all my tests in three test classes: BallTests, PaddleTests and ScoreTests. Each class will have various “granular” test cases to check the different features.

To register a C# methods for the test session, I just need to give it the Test method attribute (for a normal function) or the UnityTest method attribute (for a coroutine). Then, the method should contain one or more assert (using the Assert class) to actually run some tests.

The granularity (i.e. the number of asserts) in a single test case is up to you, but in general, each test case should have as limited a scope as possible. For example, in my BallTests class, I will have just one test case that insures the ball is properly respawned when I use my GameHandler.InitializeBall() function:

This test asserts that two things are true after the ball is reset: one, that the ball is in the middle of the screen and two, that it has a non-null velocity.

The ScoreTests class differentiates between scoring for the left and the right player – each situation is taken care of in its own test case:

But it’s the PaddleTests class that is really interesting!

The first test cases are just about game initialisation – I check that the paddles are properly instantiated, placed and configured. But then, the following methods are coroutines (that use the UnityTest attribute) that simulate a small portion of time during which I move a paddle.

Since the two paddles work exactly the same and only the input is different, I don’t discriminate between left and right and assume that I can test only one. I run 4 tests on the left paddle:

  • ShouldMovePaddleUp() and ShouldMovePaddleDown() simply assert that the MoveUp() and MoveDown() functions do change the position of the paddle;
  • then, ShouldKeepPaddleBelowTop() and ShouldKeepPaddleAboveBottom() assert that this movement is constrained to the camera bounds and doesn’t overshoot the top and bottom edges.

Apart from the asserts, these methods all follow the same logic: first, I increase the timeScale so that the test runs faster (basically, it’s like running your video game in quick mode); then, I simulate the situation I want; and finally, I reset the timeScale to its normal value of 1.

Here is for example the first test case, ShouldMovePaddleUp():

If you want to check out the other functions, you can take a look at the Github repo 🚀

Running Unity via the console?

First of all, to run checks programatically and within an automated workflow, we obviously can’t use Unity’s user interface. Instead, we need to rely on command-line execution of Unity to run our tests and integrate our unit testing phase to our CI/CD pipeline.

To better control the unit tests we run and the exit code, we’re going to write our own test running function and call it from the command-line.

So let’s create a new Runner C# class next to our tests scripts with the following code:

This uses the Unity Test Framework to run the tests programmatically and run a specific callback routine when all tests are finished.

Now, we can run it in the terminal by calling the Unity executable from the command-line. The path is a bit different on Windows and Mac computers, here is an example for an OS X system:

/Applications/Unity/Hub/Editor/2020.3.20f1/Unity.app -batchmode -projectPath /path/to/your/project -nographics -executeMethod Runner.RunUnitTests -logFile unit_test_logs.log

The first part of the command line is the path to Unity executable; then -batchmode and -nographics are here to run the editor in console mode without any user interface. The -executeMethod option will directly run our function, and the -logFile option will specify the path to the execution logs output file.

Don’t forget to adapt the -projectPath option to your own Unity project path!

Setting up the CI/CD

The final step is to integrate this unit testing step inside a Codemagic workflow, so that it is part of an automated CI/CD process.

Important note: At the moment of publishing this article, Unity’s build workflows require a special Codemagic account, and you’ll need to contact Codemagic here to get one. Otherwise, this CI/CD tutorial will not work, because you won’t have access to Mac Pro or Windows instances with Unity installed!
It also requires a Pro Unity License – you’ll need to activate your license during the workflow for building and publishing to work properly.

Codemagic relies on apps: every project is an app that pulls the code from a remote repository (GitHub, GitLab, Bitbucket or custom). Once you’ve created a Codemagic account, you can go to your dashboard to create and configure a new app:

First, you’ll need to link the Git repository you want to pull the project code from:

Then, you’ll be able to configure the app a bit more, in particular by adding some environment variables. This is a secure way of adding sensitive data such as credentials without committing them to the Git repository!

As explained in the Codemagic docs for Unity app building, you’ll need to define three environment variables: UNITY_SERIAL, UNITY_USERNAME and UNITY_PASSWORD – make sure to add all of them to the unity group:

The next step is to add a C# build script to your project, as Editor/Build.cs, to build the project programmatically instead of manually clicking Build in the Build Settings panel:

Here, I’ve only defined the build for Macs, but the Codemagic docs show you some other platform build configurations.

And then, the really nice thing with Codemagic is that you can actually do everything in the workflow definition file! This configuration is defined in the codemagic.yaml file that you put at the root of your project (next to the Assets/ main folder)! Because our workflow is pretty simple here, the idea is simply to chain a list of shell commands to execute the different steps of the CI/CD process, like this:

As you can see, in this file, we specify the execution steps, and also the type of instance to run the workflow on, the environment variables to load, the artifacts to produce, etc.

The environment part lets us “inject” all the env vars we defined previously in our “unity” group, as well as some custom vars on the spot (like the UNITY_BIN). The scripts block defines the various steps of our workflow:

  • first, we activate our Pro Unity License so that we have access to all the required tools;
  • then, we run the unit tests: if everything passes (i.e. if we have a 0 exit code), we continue on to the next step; else, the workflow is interrupted immediately, and nothing is published;
  • then, we actually build the new version into an app and store it as an artifact of the build;
  • and finally, we deactivate the license and do some clean-ups.

Before the clean-up, we can also add a step to publish to the stores, as explained here.

In the end, we’ll have produced a single artifact (see the artifacts block): the executable itself, our runnable Pong game.

Once you’ve committed and pushed all of this to your repository, simply go back to your Codemagic dashboard and start a build of your app, picking this workflow from the list:

You’ll see the different CI/CD steps on the right: first, the remote machine will initialise, then the code will be pulled from your repository, and then the steps from your codemagic.yaml configuration file will be executed.

If your workflow fails with a 1 exit code for some reason (for example, if the unit tests fail), the process will get interrupted:

Else, the whole thing will run until the end and deploy your app automatically:

You can go to your Builds to check the various states of your workflow builds – you’ll see the repository and commit each build uses as well the time it took to complete, and you’ll get a link to download the artifacts created by the workflow:

Conclusion

And tadaa: thanks to Codemagic, we now have automated our Unity CI/CD process with unit testing to validate that our code is OK and doesn’t cause any regressions or bugs! Now, you can go even further with test reporting or workflow scheduling

I hope you enjoyed this tutorial – and of course, don’t hesitate to share your ideas for other DevOps topics you’d like me to make Unity tutorials on! You can find me on Codemagic’s Slack, on Medium or on Twitter.

Don’t forget you can get the sample project for this tutorial together with the codemagic.yaml file on Github, over here 🚀

Leave a Reply

Your email address will not be published.