JSONify All Things

Extending the nlohmann/json Library

Nicolas Arnaud-Cormos

14 April 2022

The Problem

I particularly like how you can easily define serialization/deserialization for your own type:

struct SimpleStruct {
    int id;
    std::string text;
};
NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE(SimpleStruct, id, text);

But what about for a more complex data structure, using a std::optional or std::variant, like this:

struct ComplexStruct {
    std::variant<std::string, int> id;
    std::optional<std::string> text;
    std::optional<std::variant<std::vector<int>, bool>> values;
};

Unfortunately, it's not supported out-of-the-box by nlohmann/json; see some of the issues here:

I'll tell you what I want, what I really, really want: being able to write:

NLOHMANN_JSONIFY_ALL_THINGS(ComplexStruct, id, text, values)

Variants

nlohmann/json gives you everything you need to write down your own serialization code; see the documentation for arbitrary type conversions. In our case, this means writing an adl_serializer for std::variant.

namespace nlohmann {
template <typename... Ts>
struct adl_serializer<std::variant<Ts...>> {
    static void to_json(nlohmann::json &j, const std::variant<Ts...> &data) { /*TODO*/ }
    static void from_json(const nlohmann::json &j, std::variant<Ts...> &data) { /*TODO*/ }
};
}

We just need to fill the TODO. The issue I linked earlier gives one solution. Unfortunately, it requires storing the index of the type close to the value in the JSON file. This is only possible if you control the whole chain. If you need to integrate in an existing protocol, it may not be possible.

The solution below will focus on the case where you have only one value for the variant and no indication of the type.

Serialization

To serialize the to_json method, we just want to automatically set into j the type that is in the variant. Fortunetaly, std::visit comes to the rescue and it ends up being a one liner:

static void to_json(nlohmann::json &j, const std::variant &data) {
    // Will call j = v automatically for the right type
    std::visit([&j](const auto &v) { j = v; }, data);
}

Deserialization

To deserialize is a bit more complex as we don't know the exact type. So, we need to try them all to find the right one. If you can use C++17, this can be done quickly with fold expressions:

// Try to set the value of type T into the variant data if it fails, do nothing
template <typename T, typename... Ts>
void variant_from_json(const nlohmann::json &j, std::variant<Ts...> &data) {
    try {
        data = j.get<T>();
    } catch (...) {
    }
}
static void from_json(const nlohmann::json &j, std::variant<Ts...> &data) {
    // Call variant_from_json for all types, only one will succeed
    (variant_from_json<Ts>(j, data), ...);
}

The fold expression (line 11) allows us to handle variadic templates without the need of writing recursive code. The ... means that it's going to repeat what is on the left of the comma for all types.

Be careful; this solution has some issues:

The types need to be "exclusive," meaning you shouldn't be able to convert one from another. For example, std::variant<int, long long> won't work as expected.
During deserialization, if your type variant has n types, you will raise (and catch) n-1 exceptions.

Optionals

Optionals are more complex, because we can't have a to_json/from_json at the level of the property, as the property may not exist at all. So we need to go one level up. There are actually some detailed explanations on the issue I linked before, thanks to all the people who've shared their solutions.

First Step

The first thing is to write down code to serialize/deserialize an optional. This code will be called later on in the parent json value to_json/from_json:

template <class T>
void optional_to_json(nlohmann::json &j, const char *name, const std::optional<T> &value) {
    if (value)
        j[name] = *value;
}
template <class T>
void optional_from_json(const nlohmann::json &j, const char *name, std::optional<T> &value) {
    const auto it = j.find(name);
    if (it != j.end())
        value = it->get<T>();
    else
        value = std::nullopt;
}

But we still need to write down, explicitly, the code to serialize/deserialize the structure using the optionals. At this point, we have only done half of the work.

Second Step

To go further, we need to look at how the macro is implemented. Looking at the code, this is what is done for all properties you pass in the macro NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE:

#define NLOHMANN_JSON_TO(v1) nlohmann_json_j[#v1] = nlohmann_json_t.v1;
#define NLOHMANN_JSON_FROM(v1) nlohmann_json_j.at(#v1).get_to(nlohmann_json_t.v1);

In the case of a std::optional, we want to call our own optional_[from|to]_json rather than the default. Again, with C++17 to the rescue, using if constexpr we can write:

template <typename>
constexpr bool is_optional = false;
template <typename T>
constexpr bool is_optional<std::optional<T>> = true;

template <typename T>
void extended_to_json(const char *key, nlohmann::json &j, const T &value) {
    if constexpr (is_optional<T>)
        optional_to_json(j, key, value);
    else
        j[key] = value;
}
template <typename T>
void extended_from_json(const char *key, const nlohmann::json &j, T &value) {
    if constexpr (is_optional<T>)
        optional_from_json(j, key, value);
    else
        j.at(key).get_to(value);
}

We then use our extended version to create our own macro, still copying what his done in the nlohmann/json library:

#define EXTEND_JSON_TO(v1) extended_to_json(#v1, nlohmann_json_j, nlohmann_json_t.v1);
#define EXTEND_JSON_FROM(v1) extended_from_json(#v1, nlohmann_json_j, nlohmann_json_t.v1);

#define NLOHMANN_JSONIFY_ALL_THINGS(Type, ...)                                          \
  inline void to_json(nlohmann::json &nlohmann_json_j, const Type &nlohmann_json_t) {   \
      NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(EXTEND_JSON_TO, __VA_ARGS__))            \
  }                                                                                     \
  inline void from_json(const nlohmann::json &nlohmann_json_j, Type &nlohmann_json_t) { \
      NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(EXTEND_JSON_FROM, __VA_ARGS__))          \
  }

Conclusion

That's it! With this code, we are now able to write down:

struct ComplexStruct {
    std::variant<std::string, int> id;
    std::optional<std::string> text;
    std::optional<std::variant<std::vector<int>, bool>> values;
};
NLOHMANN_JSONIFY_ALL_THINGS(ComplexStruct, id, text, values)

and it just works out of the box !

Now why is it not by default in the library? In my opinion, there are multiple reasons for that:

the current implementation for std::variant has a hard pre-requisite that all types must be exclusive;
the current implementation for std::variant is not the most performant one, as it requires multiple exceptions;
it would be better to store the index of the type's variant in the json, if possible;
the current implementation for std::variant expects that an empty optional does not exist, while for some it may be on a null value in JSON.

Overall, the problem is complex and I don't think one solution will fit them all, meaning the chances for inclusion in the library are probably very low.

A big thanks to Niels Lohmann for creating this amazing piece of the library! I love it and am using it when I can.

Annex: full code all together

A big thank to Andrew for finding an issue in the code in the blog (now fixed).
You'll find the whole code all together below, if you want to just copy/paste it.

#include <nlohmann/json.hpp>

#include <optional>
#include <variant>

namespace nlohmann {

///////////////////////////////////////////////////////////////////////////////
// std::variant
///////////////////////////////////////////////////////////////////////////////
// Try to set the value of type T into the variant data if it fails, do nothing
template <typename T, typename... Ts>
void variant_from_json(const nlohmann::json &j, std::variant<Ts...> &data) {
    try {
        data = j.get<T>();
    } catch (...) {
    }
}

template <typename... Ts>
struct adl_serializer<std::variant<Ts...>>
{
    static void to_json(nlohmann::json &j, const std::variant<Ts...> &data) {
        // Will call j = v automatically for the right type
        std::visit([&j](const auto &v) { j = v; }, data);
    }

    static void from_json(const nlohmann::json &j, std::variant<Ts...> &data) {
        // Call variant_from_json for all types, only one will succeed
        (variant_from_json<Ts>(j, data), ...);
    }
};
///////////////////////////////////////////////////////////////////////////////
// std::optional
///////////////////////////////////////////////////////////////////////////////
template <class T>
void optional_to_json(nlohmann::json &j, const char *name, const std::optional<T> &value) {
    if (value)
        j[name] = *value;
}
template <class T>
void optional_from_json(const nlohmann::json &j, const char *name, std::optional<T> &value) {
    const auto it = j.find(name);
    if (it != j.end())
        value = it->get<T>();
    else
        value = std::nullopt;
}

///////////////////////////////////////////////////////////////////////////////
// all together
///////////////////////////////////////////////////////////////////////////////
template <typename>
constexpr bool is_optional = false;
template <typename T>
constexpr bool is_optional<std::optional<T>> = true;

template <typename T>
void extended_to_json(const char *key, nlohmann::json &j, const T &value) {
    if constexpr (is_optional<T>)
        nlohmann::optional_to_json(j, key, value);
    else
        j[key] = value;
}
template <typename T>
void extended_from_json(const char *key, const nlohmann::json &j, T &value) {
    if constexpr (is_optional<T>)
        nlohmann::optional_from_json(j, key, value);
    else
        j.at(key).get_to(value);
}

}

#define EXTEND_JSON_TO(v1) extended_to_json(#v1, nlohmann_json_j, nlohmann_json_t.v1);
#define EXTEND_JSON_FROM(v1) extended_from_json(#v1, nlohmann_json_j, nlohmann_json_t.v1);

#define NLOHMANN_JSONIFY_ALL_THINGS(Type, ...)                                          \
  inline void to_json(nlohmann::json &nlohmann_json_j, const Type &nlohmann_json_t) {   \
      NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(EXTEND_JSON_TO, __VA_ARGS__))            \
  }                                                                                     \
  inline void from_json(const nlohmann::json &nlohmann_json_j, Type &nlohmann_json_t) { \
      NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(EXTEND_JSON_FROM, __VA_ARGS__))          \
  }

Tags:

c++performance

About KDAB

The KDAB Group is a globally recognized provider for software consulting, development and training, specializing in embedded devices and complex cross-platform desktop applications. In addition to being leading experts in Qt, C++ and 3D technologies for over two decades, KDAB provides deep expertise across the stack, including Linux, Rust and modern UI frameworks. With 100+ employees from 20 countries and offices in Sweden, Germany, USA, France and UK, we serve clients around the world.

7 Comments

Hi,

Here is the full code for variant, copied from my project:

namespace nlohmann {
///////////////////////////////////////////////////////////////////////////////
// std::variant
///////////////////////////////////////////////////////////////////////////////
// Try to set the value of type T into the variant data
// if it fails, do nothing
template <typename T, typename... Ts>
void variant_from_json(const nlohmann::json &j, std::variant<Ts...> &data)
{
    try {
        data = j.get<T>();
    } catch (...) {
    }
}

template <typename... Ts>
struct adl_serializer<std::variant<Ts...>>
{
    static void to_json(nlohmann::json &j, const std::variant<Ts...> &data)
    {
        // Will call j = v automatically for the right type
        std::visit(
            [&j](const auto &v) {
                j = v;
            },
            data);
    }

    static void from_json(const nlohmann::json &j, std::variant<Ts...> &data)
    {
        // Call variant_from_json for all types, only one will succeed
        (variant_from_json<Ts>(j, data), ...);
    }
};
}

This should work out of the box if you have a C++17 compiler.

Hi Nicolas, Thanks for the super useful post. I had the same error “a template argument list is not allowed in a declaration of a primary template” as described above when using Visual C++ 2019 and Intel Compilers 19.2 and 2022 ( all set to C++17)

The code needed one change to add 'nlohmann::' struct nlohmann::adl_serializer

@Andrew Thank you a lot, I was wondering what went wrong, as everything was working for me. That's what happens when you copy/paste only part of a file.

I've updated both the blog and my comment, as well as added an annex with everything together.

Hi. Sorry my example is still not working. I use the latest json version 3.10.5. I use visual Studio 2022.

But I still get the error Error C2672: 'nlohmann::basic_json>::get': no matching overloaded function found (346)

This is my code

namespace mynamespace
{
    using valuetype = std::variant;

    inline void from_json(const json& j, valuetype& value)
    {
       value = j.at("value").get();
    }
}

Superb! I really hope this gets to the library.

Unfortunately I don't think it can go into the library, as the current implementations depends mostly on my use case and have some hard pre-requisites (like all types need to be exclusive for std::variant). But it's nice that the library allows us to implement that ourselves.

Just a quick heads-up if anyone is having problems compiling the code by trying to use the NLOHMANN_JSONIFY_ALL_THINGS macro in the same way as NLOHMANN_DEFINE_TYPE_INTRUSIVE. The former is outside the struct/class definition, wheres the latter is inside:

struct ComplexStruct {
    std::variant id;
    std::optional text;
    std::optional<std::variant<std::vector, bool>> values; 
};
// Outside the struct!
NLOHMANN_JSONIFY_ALL_THINGS(ComplexStruct, id, text, values)

struct person {
    std::string name;
    std::string address;
    int age;

    friend ostream& operator<<(ostream& os, const person& p);

    public:
       // Inside the struct!
        NLOHMANN_DEFINE_TYPE_INTRUSIVE(person, name, address, age)
};

I banged my head against the wall for a little while because I copied and pasted a struct. To use NLOHMANN_JSONIFY_ALL_THINGS inside the class, replace inline with friend:

#define NLOHMANN_JSONIFY_ALL_THINGS(Type, ...)                                          \
  friend void to_json(nlohmann::json &nlohmann_json_j, const Type &nlohmann_json_t) {   \
      NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(EXTEND_JSON_TO, __VA_ARGS__))            \
  }                                                                                     \
  friend void from_json(const nlohmann::json& nlohmann_json_j, Type& nlohmann_json_t) { \
      NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(EXTEND_JSON_FROM, __VA_ARGS__))          \
  }

Nicolas Arnaud-Cormos

Senior Software Engineer & Teamlead

Senior software engineer and teamlead at KDAB, Nicolas has actively developed with Qt since 2001 and is a founding member of Qtfr, the French Qt community site. He has worked on multiple Qt widgets or QML projects, with a particular emphasis on API design and software architecture. He has held Qt training classes for companies such as Michelin, Ford and ST-Ericsson. Nicolas holds an MSc in Computer Science.

JSONify All Things

Extending the nlohmann/json Library

The Problem

Variants

Serialization

Deserialization

Optionals

First Step

Second Step

Conclusion

Annex: full code all together

7 Comments

Related Content

Sign up for the KDAB Newsletter

JSONify All Things

Extending the nlohmann/json Library

The Problem

Variants

Serialization

Deserialization

Optionals

First Step

Second Step

Conclusion

Annex: full code all together

7 Comments

Related Content

Setting C++ Defines with CMake

10 Tips to Make Your QML Code Faster and More Maintainable

Mastering Cross-platform Desktop Apps

Sign up for the KDAB Newsletter