JSONify All Things Extending the nlohmann/json Library

14.04.2022 Nicolas Arnaud-Cormos 7 comments

The nlohmann/json library is everything a developer can expect from a modern library — easy to integrate and JSON objects are treated as first class citizens with a very intuitive API.

However, it has one problem that is widely mentioned across the internet, which I’ll tell you about below. Various solutions to the problem have been developed and shared, but none seem to be easy-to-use.

In this blog post, we will see how one can serialize and deserialize almost anything by extending the library a bit.

The Problem

I particularly like how you can easily define serialization/deserialization for your own type:

struct SimpleStruct {
    int id;
    std::string text;
};
NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE(SimpleStruct, id, text);

But what about for a more complex data structure, using a std::optional or std::variant, like this:

struct ComplexStruct {
    std::variant<std::string, int> id;
    std::optional<std::string> text;
    std::optional<std::variant<std::vector<int>, bool>> values;
};

Unfortunately, it’s not supported out-of-the-box by nlohmann/json; see some of the issues here:

• https://github.com/nlohmann/json/issues/1261
• https://github.com/nlohmann/json/issues/1749

I’ll tell you what I want, what I really, really want: being able to write:

NLOHMANN_JSONIFY_ALL_THINGS(ComplexStruct, id, text, values)

Variants

nlohmann/json gives you everything you need to write down your own serialization code; see the documentation for arbitrary type conversions. In our case, this means writing an adl_serializer for std::variant.

namespace nlohmann {
template <typename... Ts>
struct adl_serializer<std::variant<Ts...>> {
    static void to_json(nlohmann::json &j, const std::variant<Ts...> &data) { /*TODO*/ }
    static void from_json(const nlohmann::json &j, std::variant<Ts...> &data) { /*TODO*/ }
};
}

We just need to fill the TODO. The issue I linked earlier gives one solution. Unfortunately, it requires storing the index of the type close to the value in the JSON file. This is only possible if you control the whole chain. If you need to integrate in an existing protocol, it may not be possible.

The solution below will focus on the case where you have only one value for the variant and no indication of the type.

Serialization

To serialize the to_json method, we just want to automatically set into j the type that is in the variant. Fortunetaly, std::visit comes to the rescue and it ends up being a one liner:

static void to_json(nlohmann::json &j, const std::variant &data) {
    // Will call j = v automatically for the right type
    std::visit([&j](const auto &v) { j = v; }, data);
}

Deserialization

To deserialize is a bit more complex as we don’t know the exact type. So, we need to try them all to find the right one. If you can use C++17, this can be done quickly with fold expressions:

// Try to set the value of type T into the variant data if it fails, do nothing
template <typename T, typename... Ts>
void variant_from_json(const nlohmann::json &j, std::variant<Ts...> &data) {
    try { 
        data = j.get<T>();
    } catch (...) {
    }
}   
static void from_json(const nlohmann::json &j, std::variant<Ts...> &data) {
    // Call variant_from_json for all types, only one will succeed
    (variant_from_json<Ts>(j, data), ...);
}

The fold expression (line 11) allows us to handle variadic templates without the need of writing recursive code. The ... means that it’s going to repeat what is on the left of the comma for all types.

Be careful; this solution has some issues:

• The types need to be “exclusive,” meaning you shouldn’t be able to convert one from another. For example, std::variant<int, long long> won’t work as expected.
• During deserialization, if your type variant has n types, you will raise (and catch) n-1 exceptions.

Optionals

Optionals are more complex, because we can’t have a to_json/from_json at the level of the property, as the property may not exist at all. So we need to go one level up. There are actually some detailed explanations on the issue I linked before, thanks to all the people who’ve shared their solutions.

First Step

The first thing is to write down code to serialize/deserialize an optional. This code will be called later on in the parent json value to_json/from_json:

template <class T>
void optional_to_json(nlohmann::json &j, const char *name, const std::optional<T> &value) {
    if (value)
        j[name] = *value;
}
template <class T>
void optional_from_json(const nlohmann::json &j, const char *name, std::optional<T> &value) {
    const auto it = j.find(name);
    if (it != j.end())
        value = it->get<T>();
    else
        value = std::nullopt;
}

But we still need to write down, explicitly, the code to serialize/deserialize the structure using the optionals. At this point, we have only done half of the work.

Second Step

To go further, we need to look at how the macro is implemented. Looking at the code, this is what is done for all properties you pass in the macro NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE:

#define NLOHMANN_JSON_TO(v1) nlohmann_json_j[#v1] = nlohmann_json_t.v1;
#define NLOHMANN_JSON_FROM(v1) nlohmann_json_j.at(#v1).get_to(nlohmann_json_t.v1);

In the case of a std::optional, we want to call our own optional_[from|to]_json rather than the default. Again, with C++17 to the rescue, using if constexpr we can write:

template <typename>
constexpr bool is_optional = false;
template <typename T>
constexpr bool is_optional<std::optional<T>> = true;
 
template <typename T>
void extended_to_json(const char *key, nlohmann::json &j, const T &value) {
    if constexpr (is_optional<T>)
        optional_to_json(j, key, value);
    else
        j[key] = value;
}
template <typename T>
void extended_from_json(const char *key, const nlohmann::json &j, T &value) {
    if constexpr (is_optional<T>)
        optional_from_json(j, key, value);
    else
        j.at(key).get_to(value);
}

We then use our extended version to create our own macro, still copying what his done in the nlohmann/json library:

#define EXTEND_JSON_TO(v1) extended_to_json(#v1, nlohmann_json_j, nlohmann_json_t.v1);
#define EXTEND_JSON_FROM(v1) extended_from_json(#v1, nlohmann_json_j, nlohmann_json_t.v1);
 
#define NLOHMANN_JSONIFY_ALL_THINGS(Type, ...)                                          \
  inline void to_json(nlohmann::json &nlohmann_json_j, const Type &nlohmann_json_t) {   \ 
      NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(EXTEND_JSON_TO, __VA_ARGS__))            \
  }                                                                                     \
  inline void from_json(const nlohmann::json &nlohmann_json_j, Type &nlohmann_json_t) { \
      NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(EXTEND_JSON_FROM, __VA_ARGS__))          \
  }

Conclusion

That’s it! With this code, we are now able to write down:

struct ComplexStruct {
    std::variant<std::string, int> id;
    std::optional<std::string> text;
    std::optional<std::variant<std::vector<int>, bool>> values; 
};
NLOHMANN_JSONIFY_ALL_THINGS(ComplexStruct, id, text, values)

and it just works out of the box !

Now why is it not by default in the library? In my opinion, there are multiple reasons for that:

• the current implementation for std::variant has a hard pre-requisite that all types must be exclusive;
• the current implementation for std::variant is not the most performant one, as it requires multiple exceptions;
• it would be better to store the index of the type’s variant in the json, if possible;
• the current implementation for std::variant expects that an empty optional does not exist, while for some it may be on a null value in JSON.

Overall, the problem is complex and I don’t think one solution will fit them all, meaning the chances for inclusion in the library are probably very low.

A big thanks to Niels Lohmann for creating this amazing piece of the library! I love it and am using it when I can.

Annex: full code all together

A big thank to Andrew for finding an issue in the code in the blog (now fixed). You’ll find the whole code all together below, if you want to just copy/paste it.

#include <nlohmann/json.hpp>

#include <optional>
#include <variant>

namespace nlohmann {

///////////////////////////////////////////////////////////////////////////////
// std::variant
///////////////////////////////////////////////////////////////////////////////
// Try to set the value of type T into the variant data if it fails, do nothing
template <typename T, typename... Ts>
void variant_from_json(const nlohmann::json &j, std::variant<Ts...> &data) {
    try {
        data = j.get<T>();
    } catch (...) {
    }
}

template <typename... Ts>
struct adl_serializer<std::variant<Ts...>>
{
    static void to_json(nlohmann::json &j, const std::variant<Ts...> &data) {
        // Will call j = v automatically for the right type
        std::visit([&j](const auto &v) { j = v; }, data);
    }

    static void from_json(const nlohmann::json &j, std::variant<Ts...> &data) {
        // Call variant_from_json for all types, only one will succeed
        (variant_from_json<Ts>(j, data), ...);
    }
};
///////////////////////////////////////////////////////////////////////////////
// std::optional
///////////////////////////////////////////////////////////////////////////////
template <class T>
void optional_to_json(nlohmann::json &j, const char *name, const std::optional<T> &value) {
    if (value)
        j[name] = *value;
}
template <class T>
void optional_from_json(const nlohmann::json &j, const char *name, std::optional<T> &value) {
    const auto it = j.find(name);
    if (it != j.end())
        value = it->get<T>();
    else
        value = std::nullopt;
}

///////////////////////////////////////////////////////////////////////////////
// all together
///////////////////////////////////////////////////////////////////////////////
template <typename>
constexpr bool is_optional = false;
template <typename T>
constexpr bool is_optional<std::optional<T>> = true;

template <typename T>
void extended_to_json(const char *key, nlohmann::json &j, const T &value) {
    if constexpr (is_optional<T>)
        nlohmann::optional_to_json(j, key, value);
    else
        j[key] = value;
}
template <typename T>
void extended_from_json(const char *key, const nlohmann::json &j, T &value) {
    if constexpr (is_optional<T>)
        nlohmann::optional_from_json(j, key, value);
    else
        j.at(key).get_to(value);
}

}

#define EXTEND_JSON_TO(v1) extended_to_json(#v1, nlohmann_json_j, nlohmann_json_t.v1);
#define EXTEND_JSON_FROM(v1) extended_from_json(#v1, nlohmann_json_j, nlohmann_json_t.v1);

#define NLOHMANN_JSONIFY_ALL_THINGS(Type, ...)                                          \
  inline void to_json(nlohmann::json &nlohmann_json_j, const Type &nlohmann_json_t) {   \ 
      NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(EXTEND_JSON_TO, __VA_ARGS__))            \
  }                                                                                     \
  inline void from_json(const nlohmann::json &nlohmann_json_j, Type &nlohmann_json_t) { \
      NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(EXTEND_JSON_FROM, __VA_ARGS__))          \
  }