Making a RTS game #30: Refactoring our save/load system with binary serialisation 1/2 (Unity/C#)

Let’s keep working on our RTS and refactor our serialisation process to better protect our data!

This article is also available on Medium.

In a previous tutorial, we talked about saving and loading data from files on the disk. We prepared basic serialisers and deserialisers using the JSON format that rely on the JSONUtility built-in Unity package.

Today and in the next episode, we are going to refactor this system to use binary files rather than JSON to benefit from the advantages of binary serialisation!

Important note: doing binary serialisation has lots of advantages but it is also heavier for developers. To me, it was a good opportunity to talk about this technique and data-agnostic reflection-based code in C#. But, of course, to protect our data, we could also stick with JSON data and encrypt/decrypt it in various ways… 😉

Why use binary serialisation?

“Ok, what’s wrong with our current JSON serialisation technique?”

A while ago, we saw how to use JSON serialisation to save our game parameters assets in edit mode or during runtime. We created a JSONSerializableScriptableObject class for our game parameters class to inherit from.

However, the strength of JSON data is also its weakness: it is very easy to read and modify! So even though we’ve somewhat “hidden” our files using the Application.persistentDataPath built-in variable, a player who finds these files can still access them to change the parameters by hand.

For some of those, it’s ok: for example, the music volume or the key bindings. Basically, the variables that are exposed in our in-game settings panel could technically be updated in the JSON file, too, even if it’s not as handy for the users.

But what about the internals of the game?

If you run the build game (or if you re-enable the save system for the Unity editor in the GameManager), you’ll see that the Global Parameters JSON-data naturally contains all the variables we defined, including, for example, our new evolution curves for the level up system. This means that a sneaky player could go ahead and modify this to boost up its upgrades.

Sure, this is pretty tech-savvy cheating, but still: it would be better if we protected our data a bit more.

The benefits of binary data

Binary data is, well, much, much harder to read for humans! With the exception of strings that can be easily re-decoded by a text editor, the other data types will turn into strange characters and so binary files usually look like that:

??????@gff@???@?@gf?@gf??A	??,A
@A
  43SA
??yAgf?A?A???A43?A?̬Agf?A?A???A43?A???Agf?A?A???A?BgfB43
                                                       BB??B ??B!gfB"43#B#(B$??,B%??1B&gf6B'43;B(@B)??DB*??IB+gfNB,43SB-XB.??\B/??aB0gffB143kB2pB3??tB4??yB5gf~B6???B7?B8gf?B9?̈B:43?B;???B<?B=gf?B>?̔B?43?B@???BA?BBgf?BC?̠BD43?BE???BF?BGgf?BH?̬BI43?BJ???BK?BLgf?BM?̸BN43?BO???BP?BQgf?BR???BS43?BT???BU?BVgf?BW???BX43?BY???BZ?B[gf?B\???B]43?B^???B_?B`gf?Ba???Bb43?Bc???B%

Sweet, right? This is clearly not as readable as our previous JSON data!

In addition to being harder to decipher, binary files are also shorter than their JSON equivalents. Here, for example, I’ve made a sequence of about 300 values with ints, floats and bools, and all of this resulted in just those few lines. In comparison, the equivalent JSON version (with some additional keys, though) would be:

{"values":[{"i":0,"f":0.0,"b":true},{"i":1,"f":1.2000000476837159,"b":false},{"i":2,"f":2.4000000953674318,"b":false},{"i":3,"f":3.6000001430511476,"b":false},{"i":4,"f":4.800000190734863,"b":false},{"i":5,"f":6.0,"b":false},{"i":6,"f":7.200000286102295,"b":false},{"i":7,"f":8.40000057220459,"b":true},{"i":8,"f":9.600000381469727,"b":false},{"i":9,"f":10.800000190734864,"b":false},{"i":10,"f":12.0,"b":false},{"i":11,"f":13.200000762939454,"b":false},{"i":12,"f":14.40000057220459,"b":false},{"i":13,"f":15.600000381469727,"b":false},{"i":14,"f":16.80000114440918,"b":true},{"i":15,"f":18.0,"b":false},{"i":16,"f":19.200000762939454,"b":false},{"i":17,"f":20.400001525878908,"b":false},{"i":18,"f":21.600000381469728,"b":false},{"i":19,"f":22.80000114440918,"b":false},{"i":20,"f":24.0,"b":false},{"i":21,"f":25.200000762939454,"b":true},{"i":22,"f":26.400001525878908,"b":false},{"i":23,"f":27.600000381469728,"b":false},{"i":24,"f":28.80000114440918,"b":false},{"i":25,"f":30.000001907348634,"b":false},{"i":26,"f":31.200000762939454,"b":false},{"i":27,"f":32.400001525878909,"b":false},{"i":28,"f":33.60000228881836,"b":true},{"i":29,"f":34.80000305175781,"b":false},{"i":30,"f":36.0,"b":false},{"i":31,"f":37.20000076293945,"b":false},{"i":32,"f":38.400001525878909,"b":false},{"i":33,"f":39.60000228881836,"b":false},{"i":34,"f":40.80000305175781,"b":false},{"i":35,"f":42.0,"b":true},{"i":36,"f":43.20000076293945,"b":false},{"i":37,"f":44.400001525878909,"b":false},{"i":38,"f":45.60000228881836,"b":false},{"i":39,"f":46.80000305175781,"b":false},{"i":40,"f":48.0,"b":false},{"i":41,"f":49.20000076293945,"b":false},{"i":42,"f":50.400001525878909,"b":true},{"i":43,"f":51.60000228881836,"b":false},{"i":44,"f":52.80000305175781,"b":false},{"i":45,"f":54.000003814697269,"b":false},{"i":46,"f":55.20000076293945,"b":false},{"i":47,"f":56.400001525878909,"b":false},{"i":48,"f":57.60000228881836,"b":false},{"i":49,"f":58.80000305175781,"b":true},{"i":50,"f":60.000003814697269,"b":false},{"i":51,"f":61.20000076293945,"b":false},{"i":52,"f":62.400001525878909,"b":false},{"i":53,"f":63.60000228881836,"b":false},{"i":54,"f":64.80000305175781,"b":false},{"i":55,"f":66.0,"b":false},{"i":56,"f":67.20000457763672,"b":true},{"i":57,"f":68.4000015258789,"b":false},{"i":58,"f":69.60000610351563,"b":false},{"i":59,"f":70.80000305175781,"b":false},{"i":60,"f":72.0,"b":false},{"i":61,"f":73.20000457763672,"b":false},{"i":62,"f":74.4000015258789,"b":false},{"i":63,"f":75.60000610351563,"b":true},{"i":64,"f":76.80000305175781,"b":false},{"i":65,"f":78.0,"b":false},{"i":66,"f":79.20000457763672,"b":false},{"i":67,"f":80.4000015258789,"b":false},{"i":68,"f":81.60000610351563,"b":false},{"i":69,"f":82.80000305175781,"b":false},{"i":70,"f":84.0,"b":true},{"i":71,"f":85.20000457763672,"b":false},{"i":72,"f":86.4000015258789,"b":false},{"i":73,"f":87.60000610351563,"b":false},{"i":74,"f":88.80000305175781,"b":false},{"i":75,"f":90.0,"b":false},{"i":76,"f":91.20000457763672,"b":false},{"i":77,"f":92.4000015258789,"b":true},{"i":78,"f":93.60000610351563,"b":false},{"i":79,"f":94.80000305175781,"b":false},{"i":80,"f":96.0,"b":false},{"i":81,"f":97.20000457763672,"b":false},{"i":82,"f":98.4000015258789,"b":false},{"i":83,"f":99.60000610351563,"b":false},{"i":84,"f":100.80000305175781,"b":true},{"i":85,"f":102.00000762939453,"b":false},{"i":86,"f":103.20000457763672,"b":false},{"i":87,"f":104.4000015258789,"b":false},{"i":88,"f":105.60000610351563,"b":false},{"i":89,"f":106.80000305175781,"b":false},{"i":90,"f":108.00000762939453,"b":false},{"i":91,"f":109.20000457763672,"b":true},{"i":92,"f":110.4000015258789,"b":false},{"i":93,"f":111.60000610351563,"b":false},{"i":94,"f":112.80000305175781,"b":false},{"i":95,"f":114.00000762939453,"b":false},{"i":96,"f":115.20000457763672,"b":false},{"i":97,"f":116.4000015258789,"b":false},{"i":98,"f":117.60000610351563,"b":true},{"i":99,"f":118.80000305175781,"b":false}]}

Now, to be honest, our files will have a bit more readable text than the example I gave above, because we’ll be using dictionaries and other C# wrappers that will add some text “overhead” to our file. This will also make the files larger than if they had just he bare data, but they will still be way smaller than JSON, and anyway size is not a big issue for us since we don’t have very heavy data. Using those wrappers will make it easier for us to code 🙂

So, today, we’re going to see how to make binary files using the BinaryFormatter class from C# and how to use C# reflection to auto-generate the serialisable data for this formatter from our Scriptable Objects.

Note: be careful, though: this technique is not completely cheater-proof! As explained in the Microsoft docs, the BinaryFormatter has some security issues and is not fully reliable for really sensitive data. But, in our case, it’s enough: the point is just to discourage people from editing this file by hand, and I’d wager a file like this is discouraging enough 😉

A basic example

Before we actually apply this to our game, let’s first get used to doing binary serialisation with a simpler example.

Quick note on serialisation

If you want to serialise data with the BinaryFormatter, it has to be… serialisable. I know, it’s obvious, but it actually means a few things in Unity/C#. I talked about it in a recent article about custom Unity editors.

First, if you want to serialise just a field, there are a few rules:

  • it has to be of a serialisable type: int, float, bool or string; or a container of those basic types
  • it can’t be staticconst or readonly

Then, if you want to serialise a class, it has to contain serialisable fields and be marked serialisable with the [Serialisable] attribute.

Preparing some test data class

So let’s say we have some basic data in a serialisable class like this one:

This class contains various serialisable fields: an int, a float and a boolean. All of these are serialisable, so the class is too and we’ll be able to write or read it with binary serialisation.

Saving the data

Now, let’s use the BinaryFormatter to save an instance of this class.

The first part is pretty similar to what we did in our JSON-based save system: we get the file path, check if the location exists and then create a file ready to input our data in. To access the file, we need an IO stream. Then, we simply create a new BinaryFormatter and use it to serialise our data to our stream:

Note: binary files are pure data – so we can give them the file extension we want. I chose .data because it’s pretty descriptive but you could put whatever you want (.txt, .dat or even custom ones like .mine or .mygame! Just make sure to use the same one for reloading later on 🙂

I can then create an instance of my basic TestData serialisable class and use my brand new SaveToFile() function to save it to a binary file (I’m doing it in the GameManager for now, it’s just for testing purposes):

If you navigate to your save folder (the application’s persistent data path, which can differ depending on your OS), you should see a new file called test.data. If you open it with a text editor, you should see the following content:

ˇˇˇˇFAssembly-CSharp, Version=0.0.0.0, Culture=neutral, PublicKeyToken=nullTestDataanIntaFloataBool33¡

You’ll notice that there is some additional text at the beginning of the file because we wrapped the data in our TestData class, but at the end we do get our three variables with some (somewhat hidden) values.

Loading back the data

Then, of course, we’ll want to make sure we can reload our data. We’ll still use the BinaryFormatter but this time we’ll create a stream on a file in “open” mode:

Just like we did before with the JSON serialisation, we’ll try and load the values from the disk or stick with the default current ones as a fallback.

And then we can update our GameManager to read the file instead of writing it:

If you run your game, you’ll get back the data that we saved previously! So: we’ve successfully saved and reloaded data using binary serialisation 🙂

Applying binary serialisation to our RTS

Ok, now that have an idea of how to do data serialisation with the binary format, time to actually use it in our game!

A naive go at it?

All in all, the functions that we’ll code up will be very similar. But we’ll need to integrate them in our current class hierarchy.

Remember that, at the moment, our GameParameters class (the parent class for all other game parameter classes) inherits from the JSONSerializableScriptableObject. What we’d want is to replace this by a brand new class, the BinarySerializableScriptableObject:

  • this class should also inherit from the ScriptableObject class to keep our overall hierarchy logic
  • but instead of calling the JSONUtility, it should call the BinaryFormatter like we so in the previous section

That sounds simple enough, right? Let’s try it out!

To begin with, let’s try to simply transform our TestData class to inherit from the ScriptableObject Unity built-in class. We’ll rename it TestScriptableObject to clearly show this dependency and change the fields a bit, just to clearly differentiate it from our previous data. Also, I’m not explicitly passing in the name of the file anymore, instead I’m using the name of the asset itself.

Since it’s now a Scriptable Object and we’ve added the [CreateAssetMenu] attribute, we can instantiate this class in our project by creating a new asset, here “Test SO”. I’ll fill it with some basic values:

Then, once again, we’ll use our GameManager to test and debug things. Let’s create a temporary public slot for our TestScriptableObject instance (don’t forget to drag it in the Inspector!) and save it directly in our Awake() function:

Sounds neat! Let’s run this…

Hum. That’s a nasty error. So, what’s going on?

Scriptable Objects can’t be serialised to binary!

Well, there’s just one problem: our scriptable objects are not actually binary serialisable! 😉

Scriptable Objects are meant to be used by Unity and managed by its own saving/serialisation system. This means that we can’t directly put a Scriptable Object through our BinaryFormatter to save it to an open file stream.

Thus even if we put the [Serializable] attribute, the instance cannot be crunched into our binary stream.

The solution? Auto-extracting the serialisable data!

It looks like we’re stuck. We are clearly not going to rewrite all of our nice Scriptable Object-based logic just for this binary formatting thing. What we want is to bypass this limitation by turning this Scriptable Object into a blob of equivalent data that is serialisable.

In other words, what we need to do is “extract” the serialisable data from our Scriptable Objects when we save them and then restore this chunk of info when we reload them.

If we think about it, Scriptable Objects are just a basic C# class on steroids. So, what if we were to strip all of this enhanced behaviour from it and just isolate the variables that the Scriptable Object contains? In our example, we’d get an instance of another C# class, one that is serialisable, and that has 4 fields: myIntField, aFloat, bFloat and myBoolVariable.

Yes – what we want is to dynamically remap the contents of our Scriptable Objects into a more “basic” class instance so we can feed this “dumbed down” version to our BinaryFormatter 😉

Moreover, this will allow us to make our generic binary serialising class, the BinarySerializableScriptableObject, and make it work for any child class that inherits from it – because rather than listing the fields by hand, we’ll get them automatically so we’ll be able to handle any Scriptable Object data structure.

And for this, we need a really nice and powerful C# tool: reflection!

Time to reflect!

I’ve mentioned and even used reflection a bit previously in this project, in particular when we worked on our in-game settings panel and we automated the listing and display of our parameters.

Still, as a quick reminder: C# reflection allows you to dynamically interact with your object types, for example to create an object with a dynamically computed type, or to get the type of variable that is not knowable beforehand by the programmer… or to get the list of fields on an instance and their current values without having to pass in the name of the variables explicitly!

Note: reflection can be a bit disorienting at first but it is an amazing tool that lets you dive deeply into your data and do pretty crazy stuff. When you mix it with custom editors, you can quickly get really cool features for your game dev team, especially tools to help programmers and designers collaborate safely and with ease – for more info, check out this nice video by Matt Gambell from Game Dev Guide 😉

In our case, what we want to do is to be able to get all of the fields in our Scriptable Object and put them in a container that we can then pass to our BinaryFormatter. Here, I’ll rely on two things:

  • first, I’ll create a helper class, the BinarySerializableData class, that has the [Serializable] attribute and is instantiated from a Scriptable Object by extracting its fields and their current values
  • those fields and values will be stored in a dictionary called properties that will map strings (the field name) to objects (that’s the C# lazy type that can “host” any type of value using boxing)

This gives me the following structure:

Now the question is: how do you go through your fields dynamically using reflection? We did it a while ago when we worked on our sound system, actually: all we need to do is import the System.Reflection package and then use the GetFields() method to get the fields in our instance:

Ok, at that point we’re ready to make our generic BinarySerializableScriptableObject class and use this new BinarySerializableData in its save and load functions. Those methods will basically be the same as in our TestScriptableObject class, except that it will use the data wrapper 😉

You can already go ahead, create this new class and make the TestScriptableObject inherit from it:

Saving our data

Instead of trying to pass in the instance directly, we’ll extract its serialisable data with our new class and give this “light” version to the BinaryFormatter during the save – more precisely the properties sub-element:

Of course, remember to remove the SaveToFile() method from our TestScriptableObject class so that it uses this one, that it inherits from its parent class! 🙂

Now, if we try to re-run our game, the GameManager will not error anymore and we’ll get a new file in our save folder, called Test SO.data (because it takes the name of our test asset).

You can try and open it with a text editor, and you’ll something like this:

ˇˇˇˇ‚System.Collections.Generic.Dictionary`2[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.Object, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]VersionComparerHashSize
KeyValuePairsíSystem.Collections.Generic.GenericEqualityComparer`1[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]ÊSystem.Collections.Generic.KeyValuePair`2[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.Object, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]][]		íSystem.Collections.Generic.GenericEqualityComparer`1[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]‰System.Collections.Generic.KeyValuePair`2[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.Object, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]¸ˇˇˇ‰System.Collections.Generic.KeyValuePair`2[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.Object, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]keyvalue
myIntField
˙ˇˇˇ¸ˇˇˇaFloatöôô?¯ˇˇˇ¸ˇˇˇ	bFloatöôôæˆˇˇˇ¸ˇˇˇmyBoolVariable

As you can see, there’s a lot of things related to our wrapping, our dictionary and so on at the beginning, but then the final lines remind us of our TestScriptableObject class structure!

Reloading the data

Since we stored the properties sub-element, to get the data back, we need to tell the BinaryFormatter to expect a dictionary of strings and objects key-value pairs. Then, using reflection again, we can assign back the data stored in the file to the matching field on our instance:

To test it works, let’s first “clean up” our Test SO asset by resetting all its values to default ones, and then change our GameManager for loading instead of saving:

If we run this, we’ll see that we get the right debugs and that the Scriptable Object has been updated directly with the values stored on disk! Note that as usual with Scriptable Objects, these changes are permanent so they’ll stay even if I stop the game 😉

Handling non-serialisable types

We now have a piece of logic that seems to work properly for basic data… but our game parameter classes are a bit more complex than that! Over the last episodes of this series, we’ve added new variables like animation curves or even custom InputBindings.

Not all of these will work out of the box!

The good variables…

On the one hand, variables of a basic type will be serialisable seamlessly. For example, the music volume (which is an int) or the toggle for the day/night cycle (which is a bool) won’t be an issue. Also, some custom types like our InputBinding are totally serialisable by the BinaryFormatter: the class is marked [Serializable] and it only contains strings which are a basic (binary serialisable) type.

We can actually try this out by adding a binding variable to our TestScriptableObject class:

And setting its values in the Inspector:

If we re-save this object and look at the file, we get data that looks like this (skipping the wrappers at the beginning):

myIntField
˙ˇˇˇ¸ˇˇˇaFloatöôô?¯ˇˇˇ¸ˇˇˇ	bFloatöôôæˆˇˇˇ¸ˇˇˇmyBoolVariable
FAssembly-CSharp, Version=0.0.0.0, Culture=neutral, PublicKeyToken=nullÙˇˇˇ¸ˇˇˇbinding	InputBindingdisplayNamekey
inputEvent
A test inputtEvent

If we reset the Test SO asset and reload it in our GameManager, we get the right debugs and the Scriptable Object is re-filled with the saved values:

If you look at the file we just saved, you see that strings are still pretty readable in binary format because characters are atomic elements that text editors can decode directly: we see the key names as well the matching string values “in plain text”.

(But this is not an issue since we now have a lot of control on which fields we write to our binary file and which ones we ignore! 🙂 )

… the bad ones?

On the other hand, Unity built-in types like Color or Vector3 are not directly serialisable. And, of course, complex objects like an AnimationCurve cannot be written in a BinaryFormatter like that.

To handle this, we’ll need to write some custom logic to solve the issue in these corner cases. At that point, we have two possibilities:

  • we either ignore the non-serialisable types completely and assume that the variables that we want to save, i.e. the variables that the user can modify, are all of basic types
  • or we write custom functions for serialising and deserialising these specific types

Well, the best is to do both 😉

Let’s create two new functions in our BinarySerializableData class called Serialize() and Deserialize(). These methods will both return a boolean flag that indicates whether the conversion was successful or not, and it will output the converted value in an out value if possible.

For serialisation, we’ll pass in the Scriptable Object instance to extract the field value from it; for deserialisation, we’ll pass in the data extracted from the binary value that we wish to reintegrate in the Scriptable Object:

The first easy thing we can do is define a list of serialisable types and, for those, pass the value through without any further modification:

Don’t forget to also update the BinarySerializableScriptableObject class: in its load function, we should use this new deserialiser to get our data back:

Ok so – with that code, we are just ignoring the non-serialisable types… but everything is in place to handle them as well! 🙂

For example, if we want to be able to serialise and deserialise Colors as well, we can add a custom serialiser/deserialiser:

Let’s test this by adding a Color field to our TestScriptableObject class:

If we set a colour and re-save the Test SO asset, we’ll get the additional data at the end of the binary file:

myIntField
˙ˇˇˇ¸ˇˇˇaFloatöôô?¯ˇˇˇ¸ˇˇˇ	bFloatöôôæˆˇˇˇ¸ˇˇˇmyBoolVariable
FAssembly-CSharp, Version=0.0.0.0, Culture=neutral, PublicKeyToken=nullÙˇˇˇ¸ˇˇˇbinding	ˇˇˇ¸ˇˇˇmyColor	InputBindingdisplayNamekey
inputEvent
A test inputtEvent„≠l?CHfl=CHfl=†ü?

And we can then reload it thanks to our custom deserialiser:

Conclusion

Now that we know how to use binary serialisation and we’ve dived into reflection, we’ll need to actually apply this to the objects that we are currently storing with JSON, namely our game parameters assets.

But this article is already quite long, and we still have a bit more work ahead for everything to work properly, so we’ll wrap this up next time! 😉

Leave a Reply

Your email address will not be published. Required fields are marked *