Ramblings about rigging, programming and games.

Effortless Serialisation with Cattrs

February 23, 2023

Cattrs is an amazing library that I feel isn't known well enough.
I am making this post as a quick introduction hoping more people will pick up on it.

Serialising Dataclasses

One of its biggest strength is how it's capable of serialising dataclasses (and attrs classes) simply by leveraging the type annotations of the fields of the classes.

Info

In all the examples I will be using attrs but everything would work equally as well with dataclasses

Definitely not me trying to get you to use attrs over dataclasses, I swear.

Say we have a simple Vector3 class like so:

from attr import define

@define
class Vector3:
    x: float
    y: float
    z: float

we can serialise it with cattrs.unstructure which gives us this:

import cattrs

my_vector = Vector3(1, 2, 3)
serialised_vector = cattrs.unstructure(my_vector)

print(serialised_vector)
# >>> {'x': 1.0, 'y': 2.0, 'z': 3.0}

and to deserialise it with cattrs.structure:

deserialised_vector = cattrs.unstructure(serialised_vector, Vector3)

print(deserialised_vector)
# >>> Vector3(x=1.0, y=2.0, z=3.0)

Question

That's fine and all but how is that different from asdict and Vector3(**serialised_vector)?

Glad you asked!
In this simple case, the results would be exactly the same but let's compare with a more complex example.

let's first improve our Vector3 class to make our lives easier

@define
class Vector3:
    x: float
    y: float
    z: float

    @classmethod
    def zeros(cls) -> Self:
        return cls(0, 0, 0)

    @classmethod
    def ones(cls) -> Self:
        return cls(1, 1, 1)

This lets us easily create vectors with commonly used default values.

Now let's define a Transform class that uses our Vector3 class:

@define
class Transform:
    translate: Vector3 = field(default=Vector3.zeros)
    rotate: Vector3 = field(default=Vector3.zeros)
    scale: Vector3 = field(default=Vector3.ones)

Here's how asdict fares against cattrs.unstructure

transform = Transform()

print(asdict(transform))
# >>> {'translate': {'x': 0, 'y': 0, 'z': 0}, 'rotate': {'x': 0, 'y': 0, 'z': 0}, 'scale': {'x': 1, 'y': 1, 'z': 1}}

print(cattrs.unstructure(transform))
# >>> {'translate': {'x': 0, 'y': 0, 'z': 0}, 'rotate': {'x': 0, 'y': 0, 'z': 0}, 'scale': {'x': 1, 'y': 1, 'z': 1}}

Alright, exactly the same.

What about Transform(**serialised_transform) vs cattrs.structure?

serialised_transform = cattrs.unstructure(transform)

print(Transform(**serialised_transform))
# >>> Transform(translate={'x': 0, 'y': 0, 'z': 0}, rotate={'x': 0, 'y': 0, 'z': 0}, scale={'x': 1, 'y': 1, 'z': 1})

print(cattrs.structure(serialised_transform, Transform))
# >>> Transform(translate=Vector3(x=0.0, y=0.0, z=0.0), rotate=Vector3(x=0.0, y=0.0, z=0.0), scale=Vector3(x=1.0, y=1.0, z=1.0))

Oh! Now there's a difference!

Transform(**serialised_transform) didn't deserialise things recursively and our translate, rotate, scale fields are plain old dictionnaries when they should be Vector3 instances.

However, cattrs.structure did what we were expecting and we can use our Transform instance as intended.

Note

You'll notice that the individual Vector values are actual float and not int.
This makes sense since this is how we've annotated the fields but it's nice to see.

This alone makes cattrs a worthwile addition to your toolbet, but it doesn't stop there.

Serialising Anything

So, we've seen this worked well with nicely type annotated classes, but this isn't always the case and we don't always have the possibility to add annotations to all the types we use (in legacy code, 3rd party libraries, etc).

I'm guessing most of my readers are Maya Users so let's use that as an example and modify our Transform class to use MVector instead

from maya.api.OpenMaya import MVector


@define
class Transform:
    translate: MVector = field(default=MVector.kZeroVector)
    rotate: MVector = field(default=MVector.kZeroVector)
    scale: MVector = field(default=MVector.kOneVector)

Let's see how that serialises:

transform = Transform()

print(cattrs.unstructure(transform))
# >>> {'translate': maya.api.OpenMaya.MVector(0, 0, 0), 'rotate': maya.api.OpenMaya.MVector(0, 0, 0), 'scale': maya.api.OpenMaya.MVector(1, 1, 1)}

Eh. cattrs did its best, but at this point it has no way of knowing what a serialised MVector looks like.

Fortunately, we can tell it how to do exactly that:

cattrs.register_unstructure_hook(MVector, lambda v: {"x": v.x,"y": v.y,  "z": v.z})

print(cattrs.unstructure(transform))
# >>> {'translate': {'x': 0.0, 'y': 0.0, 'z': 0.0}, 'rotate': {'x': 0.0, 'y': 0.0, 'z': 0.0}, 'scale': {'x': 1.0, 'y': 1.0, 'z': 1.0}}

Now that's better!

Can it deserialise it though?

serialised_transform = cattrs.unstructure(transform)

print(cattrs.structure(serialised_transform, Transform))
# Error: While structuring Transform (3 sub-exceptions)
# Traceback (most recent call last):
#   File "<maya console>", line 19, in <module>
#   File "C:\Users\loic\projects\website\.venv\Lib\site-packages\cattrs\converters.py", line 309, in structure
#     return self._structure_func.dispatch(cl)(obj, cl)
#   File "<cattrs generated structure __main__.Transform>", line 22, in structure_Transform
#     if errors: raise __c_cve('While structuring ' + 'Transform', errors, __cl)
# cattrs.errors.ClassValidationError: While structuring Transform (3 sub-exceptions) #

Uh oh!
Just like with serialisation cattrs doesn't know what to do with our MVector, but again, we can tell it what to do:

cattrs.register_structure_hook(MVector, lambda d, t: MVector(*d.values()))

serialised_transform = cattrs.unstructure(transform)

print(cattrs.structure(serialised_transform, Transform))
# >>> Transform(translate=maya.api.OpenMaya.MVector(0, 0, 0), rotate=maya.api.OpenMaya.MVector(0, 0, 0), scale=maya.api.OpenMaya.MVector(1, 1, 1))

Success!

Note

MVector doesn't take keyword arguments so we have to unpack the values directly.
for most classes you would be able to write this instead

cattrs.register_structure_hook(C, lambda d, t: C(**d))

Working with JSON

Now as nice as this is, you'll want to write this to a file at some point but unfortunately, this won't always work

Let's go over a failing example:

from typing import Set

@define
class MyClass:
    my_set: Set[float]

And let's try dumping that to a json string

import json

my_instance = MyClass({1, 2, 3, 4})

serialised_instance = cattrs.unstructure(my_instance)

print(serialised_instance)
# >>> {'my_set': {1, 2, 3, 4}}

json_str = json.dumps(serialised_instance)
# Error: Object of type set is not JSON serialisable
# Traceback (most recent call last):
#   File "<maya console>", line 12, in <module>
#   File "C:\Program Files\Autodesk\Maya2023\Python\lib\json\__init__.py", line 231, in dumps
#     return _default_encoder.encode(obj)
#   File "C:\Program Files\Autodesk\Maya2023\Python\lib\json\encoder.py", line 199, in encode
#     chunks = self.iterencode(o, _one_shot=True)
#   File "C:\Program Files\Autodesk\Maya2023\Python\lib\json\encoder.py", line 257, in iterencode
#     return _iterencode(o, 0)
#   File "C:\Program Files\Autodesk\Maya2023\Python\lib\json\encoder.py", line 179, in default
#     raise TypeError(f'Object of type {o.__class__.__name__} '
# TypeError: Object of type set is not JSON serialisable #

We can see that cattrs.unstructure worked as intended, but JSON has no concept of what a set is and simply doesn't know what to do with it.

There are a couple options to solve this, I can think of 3 right now:

  1. Make a custom JSON encoder. Eh.
  2. Register structure/unstructure hooks for set.
  3. Use the preconfigured Json Converter provided by cattrs.

Let's explore that 3rd point: Up until now when we've registered hooks, we've registered them in cattrs' "Global Converter" Cattrs provides us with a bunch of preconfigured Converters with some pre-defined hooks, including one for json. On top of that, these converters implement the loads and dumps functions, making them easy to use instead of the actual

from cattrs.preconf.json import make_converter
json = make_converter()

my_instance = MyClass({1, 2, 3, 4})

serialised_instance = cattrs.unstructure(my_instance)

json_str = json.dumps(serialised_instance, indent=2)
# >>> '{"my_set": [1, 2, 3, 4]}'

And we can load it back like that:

new_instance = json.loads(json_str, MyClass)

Note

cattrs provides a few other converters, you can find more information about them here

Conclusion

Hopefully that gave you a good taste of what cattrs is capable of and how easy it is to work with. I encourage you to go through the docs to find out more and learn about some of the features I haven't covered.

Back to top