Making a RTS game #31: Refactoring our save/load system with binary serialisation 2/2 (Unity/C#)

Let’s keep working on our RTS and refactor our serialisation process to better protect our data!

This article is also available on Medium.

Last time, we talked about using binary serialisation for load/save process. We saw the advantages of binary data over JSON in terms of security, and we saw how to apply this type of serialisation to basic data. We also discussed how reflection is a powerful tool that we can use to automate data extraction and make our methods more agnostic of the objects being handled.

Today, it’s time to actually apply all of this to our RTS and… there are yet a bit more surprises in store! 😉

2 important notes:

  1. throughout this tutorial, we will be touching and modifying our game parameters assets. Depending on when you run your game, this data might be invalid for a while, until we fix it later on in the tutorial. So if you have some settings you’d like to keep, make sure to back them up before hand, or wait for the end of the tutorial before running your game!
  2. to test the load/save process in the editor, you might need to deactivate the preprocessor check we added a while ago to prevent serialisation from the Unity editor:

Using binary serialisation for our game parameters

Now that we’re finished with our tests, we can clean up our GameManager Awake() method and remove all the code that is related to our TestScriptableObject object (plus the public slot). So we’ll go back to our plain old Awake():

To begin with, let’s “reroute” our game parameters class hierarchy to use the BinarySerializableScriptableObject instead of the JSONSerializableScriptableObject:

Given what we prepared last time, we could naively think that this is enough to make our game parameters save and reload using binary serialisation rather than JSON, right? Well…

The ugly truth

Most of the stuff we want to be saved will work without a hiccup. But! There are a few things that will cause issues, because some of our data cannot yet be serialised with our current setup.

For example, if you look at the saved data for the game player parameters, you might notice that, for now, only the myPlayerId is saved:

ˇˇˇˇ‚System.Collections.Generic.Dictionary`2[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.Object, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]VersionComparerHashSize
KeyValuePairsíSystem.Collections.Generic.GenericEqualityComparer`1[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]ÊSystem.Collections.Generic.KeyValuePair`2[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.Object, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]][]		íSystem.Collections.Generic.GenericEqualityComparer`1[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]‰System.Collections.Generic.KeyValuePair`2[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.Object, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]¸ˇˇˇ‰System.Collections.Generic.KeyValuePair`2[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.Object, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]keyvalue
myPlayerId

We see that, at the very end, there is no players key – and that’s actually to be expected! Remember we said that, with our system, we have to explicitly list the serialisable types or define a custom serialising/deserialising logic for the other types.

Fair enough: if we look back at the GamePlayersParameters.cs script, and more precisely at the PlayerData struct, we see that we marked it as [Serializable] in previous episodes. That looks promising right?

The problem is that, once again, what Unity calls “serialisable” and what we call serialisable in the context of our binary load/save process are different things. To show that clearly, you can just try and add the PlayerData type to the list of serialisable types:

Now run your game… and you should get an error, saying:

Basically, the array of PlayerData instances would indeed be serialisable for our BinaryFormatter if all its fields were too… but here, we get a nasty Color variable that the binary serialisation can’t handle natively.

In the first part, we defined a custom serialiser/deserialiser for Colors – so why is it not at play here? Because, when working on our players variable, we are one level deeper in the recursion…

What’s going on?

Ok so – what’s happening to our data exactly? Something important to understand is that, when we serialise data, the serialisation is done recursively.

For basic types like strings or ints, we have just one level. Then, if we make a class with fields of these basic types, we’ll have two levels: the binary serialiser will first serialise the inner data, then the wrapper around it. And so on with more complex types…

The thing is that you have to have serialisable objects at every level. In the previous tutorial, we saw that our custom class InputBinding serialises directly; that’s because it only contains basic types. On the other hand, PlayerData doesn’t serialiser because the binary serialiser doesn’t know what to do with the Color field.

So far, what we’ve coded up in the BinarySerializableData class only affects the second level: it will work if we are serialising a Scriptable Object (high level) that contains Color fields (low level), but it won’t be used at deeper levels.

When we are serialising our game players parameters asset, we have a 3-levels serialisation: level 3 for the Scriptable Object, level 2 for the array of PlayerData, level 1 for the fields inside the PlayerData struct. That’s why the binary serialiser crashes on the Color field: our custom serialiser isn’t used here!

Defining a serialiser for the lower levels

The ISerializable interface

To be able to serialise and deserialise data at lower levels, we have to create another generic class and define our custom serialisation logic using the C# ISerializable interface – this will allow the BinaryFormatter to write and read instances of this class.

So, today, we are going to create a BinarySerializable class that uses some methods from the BinarySerializableData class and implements the ISerializable interface:

Here, we implement the C# ISerializable interface by creating a protected constructor that creates an instance from some SerializationInfo (that’s the load function) and the GetObjectData() that writes our instance to a SerializationInfo dictionary (that’s the save function).

Note: the [SecurityPermission] attribute is a safeguard for binary serialisation – if you want to learn more, check out the Microsoft docs 😉

Using some reflection once again!

Of course, we won’t be instantiating this class directly – it’s just like our BinarySerializableScriptableObject class from last time: what we want instead is to have other classes inherit from it to make them valid for binary serialisation.

So once again, this class will rely on reflection to get a structure-agnostic logic: we’ll make sure that our functions automatically list the fields in our class and be as generic as possible in writing and reading serialised data.

What we’ll do is re-use the functions we coded previously in our BinarySerializableData, the Serialize() and Deserialize() methods, and use those in our BinarySerializable class to save or load our data.

A very basic version of our custom serialiser could therefore be:

This sounds good enough, right? We are writing our data to the stream in the GetObjectData() (save) function, and retrieving it back into our instance in the constructor (load) function.

Let’s try this out: we’ll transform our PlayerData into a class and have it inherit our brand new BinarySerializable class (note that we have to call the base constructor explicitly because it is not the default one):

Now, what if we run our game again and try to re-save our game parameters?

Well – all should work fine for serialising, so the save will work properly, and you can even check that you do have a binary file on your computer for the game players parameters (ignoring the beginning of the file):

keyvalueplayers	¯ˇˇˇ¸ˇˇˇ	
myPlayerId
PlayerData	
	

PlayerDatanamecolorEnemy	

Mina	
PΩB>Ä?Ä?Ä?™‚«=Ä?

It looks like the data is here and that we’ve got some colours stored for next time!

But when we try and reload, here’s the error message that we get:

You see that Unity tells us some data in our saved file does not implement the IConvertible interface and therefore can’t be deserialised. What does that mean?

Handling arrays

In truth, there are two essential things to note with our current code:

  • obviously, we’ll need to make sure that our Serialize() function only returns basic types so that the value we fill with it is binary serialisable. It’s the case for now since we’re only returning ints, floats, bools, strings of arrays of those types (for example, for the Color, we implemented a custom serialiser that represents it as an array of floats).
  • but we are not taking care of those arrays! Binary serialisation cannot store arrays directly, we’ll have to “expand” them into single values one after the other in our file, and also store the size to be able to rebuild it during deserialisation.

Basically, instead of storing this:

We’ll need to deconstruct it manually and store this:

To know if the data we are storing is an array or not, we’ll need to refactor a bit our BinarySerializableData class and in particular add a new method: the GetSerializedType() function. This method will return the type of object that our Serialize() method returns for a given field, by testing it on a default instance of this field:

(Note that I’ve also extracted the core serialisation logic to its own function so I can call it from the new GetSerializedType() function more easily)

To be clear, this method gives back the type of value that’s expected after the serialisation and that will be written into the SerializationInfo dictionary. For example, it will be “int” for ints, “array of booleans” for an array of booleans and “array of floats” for a Color given our custom serialiser that represents Colors as arrays of 4 floats.

Now, we need to use this new info in our BinarySerializable class to check whether the data we wan’t to read/write is an array or not.

I’ll spare you the details of implementation – it’s mostly about diving in the C# docs and finding the proper functions for doing reflection with arrays and creating variables of the right type at the right time… all in all, we get the following code:

So, for arrays:

  • during the save, we iterate through the elements and store them contiguously, and we also remember the total size of the array
  • during the load, we first get the size we stored and prepare a container of this dimension, then populate back the elements one by one

For “basic” elements, i.e. single instances of basic types that are not a container, we simply write or read them from the SerializationInfo dictionary.

Sum up

We now have a really cool binary serialisation process that handles various types of data and that can be quite easily extended, just by defining a custom serialisation/deserialisation process for new types or by adding them to the list of serialisable types!

This time, if you try and save your game parameters, then modify some and reload the game, you’ll see that the data is properly restored from the binary files and re-injected into the Scriptable Objects 🙂

Choosing the variables to serialise

At that point, we are able to serialise about any type we want, which is pretty great! But, actually, we might be serialising a bit too much… 😉

With this code, Unity will go over every field in our BinarySerializableScriptableObject instance (and recursively in those fields) and will try to put all of them inside of our binary file. This means that, for example, any integer field will get serialised because it’s a basic type and the serialiser knows how to handle them.

It’s good that our logic can do that, but it doesn’t mean it should.

Cherry-picking to get safer data dumps!

Once again, remember that one of the benefits of doing serialisation by hand is that we get to choose how we perform the data write/read. Meaning, among other things, that we can decide to filter out some fields if we want to.

In the first part, we saw that some of the data that is saved in our game parameters should be editable during games (these are mainly the variables we expose in our in-game settings, plus some additional session-related info like our current player ID) but that others are internals of the game and should stay as is. For now, our process doesn’t distinguish between the two.

What would be nice is if, in addition to picking which fields we expose in the in-game settings, we could also choose which fields were serialised or not. This way, for all the fields that we exclude from the serialisation (i.e. that are not saved to the binary file), we’ll get the default value stored in our Scriptable Object at the moment of the build.

This is actually pretty straight-forward: just like we previously defined a list of _fieldsToShowInGame in our GameParameters class, we can make a list of _fieldsToSerialize in our BinarySerializableScriptableObject:

Note that we also pass this list to our BinarySerializableData constructor so that it knows which fields to keep 🙂

Then, in our serialisation process, we will only take the fields that are marked as “serialised”:

We’re now ready to choose which fields to serialise and make sure that no internal configuration is ever visible by the players, even in binary form!

Improving our custom Unity inspector for game parameters assets

To make it easier to edit, though, we should update our custom game parameters editor. We can refactor a bit our buttons to have two different togglers (one for the in-game display, and one of the serialisation):

But it might be a bit hard to remember that the first button is for in-game display and the second one for serialisation; so instead, we can use icons:

As you can see, with the IMGUI system for the editor, adding an icon is as easy as calling the EditorGUIUtility.IconContent() function with some pre-defined assets from Unity!

For a full list of available icons, you can take a look at this link 🙂

Note: I’ve also added a basic button-derived GUI style to my button to remove the padding and get a “full-size” icon.

Try toggling on only a few fields, and leaving some serialisable fields toggled off: if you save your game parameter asset again, you’ll see that even if they can be serialised, they won’t be because our custom binary serialiser will ignore those…

Here is an example of my global parameters with just the enableDayAndNightCycle and enableFOV fields set as serialised, as shown in the picture above (ignoring the beginning of the file):

keyvalueenableDayAndNightCycle˙ˇˇˇ¸ˇˇˇ	enableFOV

The other basic types in my object (baseGoldProduction, goldBonusRange…) don’t get serialised and we get a stripped-down data dump that only contains the data we need to store and possibly reload. The rest is fixed at build time and embedded inside our “baked” Scriptable Objects 🙂

Important note: of course, you should now go through your various game parameters assets and check that all the fields you want to serialise are properly “toggled on” for serialisation, because by default the list will be empty so none will be saved to the binary file!

Conclusion

Note: once you’re done testing, you should probably re-enable the preprocessor check in the GameManager to avoid saving your dev data and overriding your Scriptable Objects…

With these two episodes, we’ve seen how to use binary serialisation to better protect the data saved on disk and reduce the size of the files. We’ve used reflection to dynamically extract serialisable data from our Scriptable Objects and we’ve updated our class hierarchy a little to directly benefit from this new feature.

In this tutorial, we’ve talked about cheating and we’ve made sure that players can’t corrupt their data files to try and modify the inner workings of the game. But, still, sometimes, it’s nice to be able to just have endless amount of gold and have a bit of fun, right? 😉

So, next time, we’ll see how to add a little debug console and have some cheat codes!

Leave a Reply

Your email address will not be published. Required fields are marked *