Why you should use caching for your Codemagic Unity projects

How to you build smarter…

This article was originally published on Codemagic’s blog.

When you work on complex Unity projects and you start adding more and more resources, you’ll quickly notice that the build time in your Codemagic workflow grows along as well. And it may be pretty annoying to realise that a good chunk of the build time is usually devoted to restoring the env and, in particular, the libraries and packages… even if those are exactly identical each time!

Generally speaking, on a local computer, we avoid this issue because after the project has built a first time, it has cached some assets and data in your Unity project folder to re-use the following times and speed up this process. But what about on a remote Codemagic machine? Can we use cache?

The answer is, luckily: yes.

And it’s worth it.

So let’s have a look at how we can leverage this nice Codemagic tool for our Unity projects 🙂

In this article, we are going to:

  • Quickly explore the fundamentals of caching for a Unity project
  • Learn why removing unused packages is important
  • Discover how to set up a Codemagic workflow with caching
  • Study a useful trick for caching an entire Unity version on a Codemagic machine

Note: if you already know about the Unity cache system, or if you just want to see how to use it with Codemagic, you can skip to the third section.

Pre-requisites

For this article, we will assume that you are already familiar with how to create a sample Unity project, version it as a Git repo and link it to Codemagic by creating a new associated app. If you want to have a reminder of all these steps, feel free to have a look at this other article.

If you want to read ahead and look through the final sample project, you can find it on Gitlab, over here 🚀

You will also need a Unity Plus or Pro license that can be activated via the command-line in the Codemagic workflow.

A quick tour of Unity’s caching

A Unity project is made of a lot of files, some being your own assets, and others being auto-generated logs, temporary data and references for look-up tricks. The fact that Unity can reconstruct these non-user resources is why, when you version a Unity project, you typically ignore all folders except for the Assets, the Packages and the ProjectSettings.

Among those auto-generated files, the ones that are crucial to caching are the ones inside of Library: basically, this is where Unity pulls and stores all the external packages your app requires, and that are listed in the manifest.json file inside of Packages. This setup is a very common one: the Package Manager tool keeps this list of necessary dependencies up-to-date behind the scenes, and when you share your project you just share this list, but not the actual contents of the required packages. Then, any other Unity executable on another machine can re-read this list and re-fetch the dependencies.

Those files are the ones worth caching because they take quite some time to restore (since the computer has to download the entire list), and they usually don’t change that often once your project is setup and ready for build. Chances are that, when comes the time to export your game, you know what packages you need, and you won’t update this list at least before the next big feature comes in.

To see how the auto-generated folders, and in particular the Library folder, impact the build time, we can run a series of tests where we either keep or remove those directories before building, and compare the results. On a very simple project like the sample one 🚀, and running each test 10 times to alleviate randomness, we can get something like:

Build time (without caching): 70.8 sec
Build time (with caching): 23.5 sec

We clearly see that keeping those folders is a great way of reducing the build time!

Removing unused packages

Another important thing to keep in mind is that, by default, lots of Unity templates pre-install a lot of dependencies… even if your project doesn’t actually need them! So, when you’re down to building and shipping your game, remember to check if your Package Manager currently lists libraries you don’t use.

For example, still with the same sample project, removing all the (unused) 2D-related packages severely decreases the build time:

Build time (without caching, with 2D): 70.8 sec
Build time (without caching, no 2D): 40.8 sec

You might think that this is only interesting if you don’t use cache, but if you try the same comparison while keeping the cache, you’ll notice that there is a difference too:

Build time (with caching, with 2D): 23.5 sec
Build time (with caching, no 2D): 16.0 sec

That’s because Unity still has to read and “register” the packages when it starts the build, so whether the package is actually used or not it will still impact the initialisation time a bit.

Setting up a Codemagic workflow with Unity build cache

Codemagic has a built-in cache system that is very easy to use: you just need to use the cache and cache_paths keywords to tell the tool which directories or files you want to keep in memory from one run to the other:

This cache system works per worfklow, meaning that you can “save” different file paths for the different worfklows defined in your codemagic.yaml configuration file.

In our case, as discussed before, we want to cache the Library folder that will be pulled from the Git repo during the initialisation steps; so we need to add the following to our Codemagic workflow setup:

where CM_BUILD_DIR is a Codemagic built-in environment variable available in any Codemagic workflow run that points to the path where the code was cloned.

Note that the cache path should always be written in this Unix-style, even on a Windows computer. For example, in my sample project meant to run on a Windows Codemagic instance, I have the following:

Once you’ve run the workflow with cache once, you’ll see that in your Codemagic app settings, in the Caching tab, you can click the Refresh list button on the right to see a new entry corresponding to your workflow ID and the time you last ran the workflow and updated the cache.

Here, you can also clear the cache (either just one line or all the cache for all the workflows) to force a full rebuild next time, and get all your new updated environment.

To see the cache in action, you can compare the first build (for which the cache wasn’t computed yet) and the second one (with the cache already prepared).

Without cache (1st build)

With cache (2nd build)

Installing and caching a Unity version

Installing a specific Unity version on a Codemagic machine

Sometimes, your workflow requires a specific Unity version. This can be because you’ve prepared your project in an older editor, or just one that is not retro-compatible with the one installed by default on a Codemagic machine (see the docs for a Mac or a Windows machine).

Now, to install a given Unity executable on the build machine, you have to know it’s exact version and version changeset.

If you need a Unity editor that is neither the one pre-installed on the Codemagic machine nor the one you use locally, it can be a bit of a long process to get this info. You have to go to the Unity download archive page, find the version you’re looking for, open its changelog and scroll to the bottom of the page to get its changeset.

However, as explain in the Codemagic docs, if you want to install the Unity version you’re using locally, then all this info is actually readily available in your project’s files. More precisely, it’s in your ProjectSettings/ProjectVersion.txt config file, that should look something like this:

m_EditorVersion: 2020.3.29f1
m_EditorVersionWithRevision: 2020.3.29f1 (2ff179115da0)

Once you’ve found this version and changeset, you just need to define them as environment variables in your Codemagic app, like UNITY_VERSION and UNITY_VERSION_CHANGESET.

At that point, you’ll have the data you need to fetch and install this specific Unity editor using the Hub CLI.

Before you do anything with this CLI, however, you need to make sure that you already have an active Unity license on the computer. Which means that you need to activate said license before you install the new executable with the CLI. That’s why, for the license activation and return steps, you still need to use the pre-installed Unity version.

To differentiate between this pre-installed executable and the newly downloaded one, you can once again use environment variables in your workflow with the path to both executables (in addition to the usual license-related Unity env vars stored in my unity env group):

  • on Mac:
  • on Windows:

After you’ve activated your license, all that’s left to do is call a few commands to actually get and install the Unity editor with a specific version:

  • on Mac:
  • on Windows:

And there you are! From that point, you’ll be able to use the UNITY_VERSION_BIN environment variable to call the new executable (i.e. the newly installed Unity version) and do all your build steps 🙂

Caching a Unity version

This is all nice and swell, but if you take a look at a workflow with this Unity installation step, you’ll quickly notice that it is quite time-consuming. Since the machine has to go online, get the executable, download it and then install it, it can easily take several minutes, and actually take up most of your “build time”.

Once again, caching comes to the rescue here!

A “Unity version” is, in truth, just a folder on the computer that is in a specific location. This means that we can use a cache path like we did before, except this time we target a directory inside the Unity Hub installation folders.

This is fairly straight-forward – on a Mac for example, you’d just add this line in your caching configuration:

On a Windows, the folder would be:

C:\Program Files\Unity\Hub\Editor\$UNITY_VERSION

This will tell the Codemagic machine to only install the new Unity editor at this location if it not there already, and thus avoid re-downloading the exact same content every time.

If you run this new workflow (at least twice to get the cache), you’ll notice a couple of things. For example, on my own trials:

Without cache

With cache

  • the difference is pretty clear: we do avoid repeating the install step, which took about 11 minutes the first time
  • on the other hand, the Cleaning up step was slightly longer than usual: that’s because we need to package and save the full Unity editor, which requires about 4 minutes
  • all in all, this caching reduces the total build time by 10 minutes!

We see that caching a Unity version is very valuable and can significantly reduce our Codemagic workflow duration. However, it does imply a larger cache which can cause some performance issues. If you go back to your cache page after running this kind of pipeline, you’ll see that Codemagic displays a warning and highlights your heavy cache.

Just as a reference, here is the final full workflow for the Mac cached version (feel free to have a look at the Gitlab 🚀 for the Windows sample):

Conclusion

A Unity project is a combination of many assets – those text files, images, 3D models and metadata are what make your game. But some of them are not handmade: they are auto-generated by Unity if not readily available.

This logic is nice but it also requires some computing power and some time. If the files to regenerate are the same each time, then it can be interesting to use caching to keep them in-between your build runs. Luckily, Codemagic has an easy-to-use caching system that makes it easy to store a folder of dependencies or even an entire Unity version 🙂

I hope you enjoyed this tutorial — and of course, don’t hesitate to share your ideas for other DevOps topics you’d like me to make Unity tutorials on! You can find me on Codemagic’s Slack, on Medium, or on Twitter.

You can checkout the sample project project on Gitlab over here 🚀

Leave a Reply

Your email address will not be published.