Skip to content

JSONify All Things Extending the nlohmann/json Library

The nlohmann/json library is everything a developer can expect from a modern library — easy to integrate and JSON objects are treated as first class citizens with a very intuitive API.

However, it has one problem that is widely mentioned across the internet, which I’ll tell you about below. Various solutions to the problem have been developed and shared, but none seem to be easy-to-use.

In this blog post, we will see how one can serialize and deserialize almost anything by extending the library a bit.

The Problem

I particularly like how you can easily define serialization/deserialization for your own type:

struct SimpleStruct {
    int id;
    std::string text;
};
NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE(SimpleStruct, id, text);

But what about for a more complex data structure, using a std::optional or std::variant, like this:

struct ComplexStruct {
    std::variant<std::string, int> id;
    std::optional<std::string> text;
    std::optional<std::variant<std::vector<int>, bool>> values;
};

Unfortunately, it’s not supported out-of-the-box by nlohmann/json; see some of the issues here:

I’ll tell you what I want, what I really, really want: being able to write:

NLOHMANN_JSONIFY_ALL_THINGS(ComplexStruct, id, text, values)

Variants

nlohmann/json gives you everything you need to write down your own serialization code; see the documentation for arbitrary type conversions. In our case, this means writing an adl_serializer for std::variant.

namespace nlohmann {
template <typename... Ts>
struct adl_serializer<std::variant<Ts...>> {
    static void to_json(nlohmann::json &j, const std::variant<Ts...> &data) { /*TODO*/ }
    static void from_json(const nlohmann::json &j, std::variant<Ts...> &data) { /*TODO*/ }
};
}

We just need to fill the TODO. The issue I linked earlier gives one solution. Unfortunately, it requires storing the index of the type close to the value in the JSON file. This is only possible if you control the whole chain. If you need to integrate in an existing protocol, it may not be possible.

The solution below will focus on the case where you have only one value for the variant and no indication of the type.

Serialization

To serialize the to_json method, we just want to automatically set into j the type that is in the variant. Fortunetaly, std::visit comes to the rescue and it ends up being a one liner:

static void to_json(nlohmann::json &j, const std::variant &data) {
    // Will call j = v automatically for the right type
    std::visit([&j](const auto &v) { j = v; }, data);
}

Deserialization

To deserialize is a bit more complex as we don’t know the exact type. So, we need to try them all to find the right one. If you can use C++17, this can be done quickly with fold expressions:

// Try to set the value of type T into the variant data if it fails, do nothing
template <typename T, typename... Ts>
void variant_from_json(const nlohmann::json &j, std::variant<Ts...> &data) {
    try { 
        data = j.get<T>();
    } catch (...) {
    }
}   
static void from_json(const nlohmann::json &j, std::variant<Ts...> &data) {
    // Call variant_from_json for all types, only one will succeed
    (variant_from_json<Ts>(j, data), ...);
}

The fold expression (line 11) allows us to handle variadic templates without the need of writing recursive code. The ... means that it’s going to repeat what is on the left of the comma for all types.

Be careful; this solution has some issues:

  • The types need to be “exclusive,” meaning you shouldn’t be able to convert one from another. For example, std::variant<int, long long> won’t work as expected.
  • During deserialization, if your type variant has n types, you will raise (and catch) n-1 exceptions.

Optionals

Optionals are more complex, because we can’t have a to_json/from_json at the level of the property, as the property may not exist at all. So we need to go one level up. There are actually some detailed explanations on the issue I linked before, thanks to all the people who’ve shared their solutions.

First Step

The first thing is to write down code to serialize/deserialize an optional. This code will be called later on in the parent json value to_json/from_json:

template <class T>
void optional_to_json(nlohmann::json &j, const char *name, const std::optional<T> &value) {
    if (value)
        j[name] = *value;
}
template <class T>
void optional_from_json(const nlohmann::json &j, const char *name, std::optional<T> &value) {
    const auto it = j.find(name);
    if (it != j.end())
        value = it->get<T>();
    else
        value = std::nullopt;
}

But we still need to write down, explicitly, the code to serialize/deserialize the structure using the optionals. At this point, we have only done half of the work.

Second Step

To go further, we need to look at how the macro is implemented. Looking at the code, this is what is done for all properties you pass in the macro NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE:

#define NLOHMANN_JSON_TO(v1) nlohmann_json_j[#v1] = nlohmann_json_t.v1;
#define NLOHMANN_JSON_FROM(v1) nlohmann_json_j.at(#v1).get_to(nlohmann_json_t.v1);

In the case of a std::optional, we want to call our own optional_[from|to]_json rather than the default. Again, with C++17 to the rescue, using if constexpr we can write:

template <typename>
constexpr bool is_optional = false;
template <typename T>
constexpr bool is_optional<std::optional<T>> = true;
 
template <typename T>
void extended_to_json(const char *key, nlohmann::json &j, const T &value) {
    if constexpr (is_optional<T>)
        optional_to_json(j, key, value);
    else
        j[key] = value;
}
template <typename T>
void extended_from_json(const char *key, const nlohmann::json &j, T &value) {
    if constexpr (is_optional<T>)
        optional_from_json(j, key, value);
    else
        j.at(key).get_to(value);
}

We then use our extended version to create our own macro, still copying what his done in the nlohmann/json library:

#define EXTEND_JSON_TO(v1) extended_to_json(#v1, nlohmann_json_j, nlohmann_json_t.v1);
#define EXTEND_JSON_FROM(v1) extended_from_json(#v1, nlohmann_json_j, nlohmann_json_t.v1);
 
#define NLOHMANN_JSONIFY_ALL_THINGS(Type, ...)                                          \
  inline void to_json(nlohmann::json &nlohmann_json_j, const Type &nlohmann_json_t) {   \ 
      NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(EXTEND_JSON_TO, __VA_ARGS__))            \
  }                                                                                     \
  inline void from_json(const nlohmann::json &nlohmann_json_j, Type &nlohmann_json_t) { \
      NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(EXTEND_JSON_FROM, __VA_ARGS__))          \
  }

Conclusion

That’s it! With this code, we are now able to write down:

struct ComplexStruct {
    std::variant<std::string, int> id;
    std::optional<std::string> text;
    std::optional<std::variant<std::vector<int>, bool>> values; 
};
NLOHMANN_JSONIFY_ALL_THINGS(ComplexStruct, id, text, values)

and it just works out of the box !

Now why is it not by default in the library? In my opinion, there are multiple reasons for that:

  • the current implementation for std::variant has a hard pre-requisite that all types must be exclusive;
  • the current implementation for std::variant is not the most performant one, as it requires multiple exceptions;
  • it would be better to store the index of the type’s variant in the json, if possible;
  • the current implementation for std::variant expects that an empty optional does not exist, while for some it may be on a null value in JSON.

Overall, the problem is complex and I don’t think one solution will fit them all, meaning the chances for inclusion in the library are probably very low.

A big thanks to Niels Lohmann for creating this amazing piece of the library! I love it and am using it when I can.

About KDAB

If you like this article and want to read similar material, consider subscribing via our RSS feed.

Subscribe to KDAB TV for similar informative short video content.

KDAB provides market leading software consulting and development services and training in Qt, C++ and 3D/OpenGL. Contact us.

Annex: full code all together

A big thank to Andrew for finding an issue in the code in the blog (now fixed). You’ll find the whole code all together below, if you want to just copy/paste it.

#include <nlohmann/json.hpp>

#include <optional>
#include <variant>

namespace nlohmann {

///////////////////////////////////////////////////////////////////////////////
// std::variant
///////////////////////////////////////////////////////////////////////////////
// Try to set the value of type T into the variant data if it fails, do nothing
template <typename T, typename... Ts>
void variant_from_json(const nlohmann::json &j, std::variant<Ts...> &data) {
    try {
        data = j.get<T>();
    } catch (...) {
    }
}

template <typename... Ts>
struct adl_serializer<std::variant<Ts...>>
{
    static void to_json(nlohmann::json &j, const std::variant<Ts...> &data) {
        // Will call j = v automatically for the right type
        std::visit([&j](const auto &v) { j = v; }, data);
    }

    static void from_json(const nlohmann::json &j, std::variant<Ts...> &data) {
        // Call variant_from_json for all types, only one will succeed
        (variant_from_json<Ts>(j, data), ...);
    }
};
///////////////////////////////////////////////////////////////////////////////
// std::optional
///////////////////////////////////////////////////////////////////////////////
template <class T>
void optional_to_json(nlohmann::json &j, const char *name, const std::optional<T> &value) {
    if (value)
        j[name] = *value;
}
template <class T>
void optional_from_json(const nlohmann::json &j, const char *name, std::optional<T> &value) {
    const auto it = j.find(name);
    if (it != j.end())
        value = it->get<T>();
    else
        value = std::nullopt;
}

///////////////////////////////////////////////////////////////////////////////
// all together
///////////////////////////////////////////////////////////////////////////////
template <typename>
constexpr bool is_optional = false;
template <typename T>
constexpr bool is_optional<std::optional<T>> = true;

template <typename T>
void extended_to_json(const char *key, nlohmann::json &j, const T &value) {
    if constexpr (is_optional<T>)
        nlohmann::optional_to_json(j, key, value);
    else
        j[key] = value;
}
template <typename T>
void extended_from_json(const char *key, const nlohmann::json &j, T &value) {
    if constexpr (is_optional<T>)
        nlohmann::optional_from_json(j, key, value);
    else
        j.at(key).get_to(value);
}

}

#define EXTEND_JSON_TO(v1) extended_to_json(#v1, nlohmann_json_j, nlohmann_json_t.v1);
#define EXTEND_JSON_FROM(v1) extended_from_json(#v1, nlohmann_json_j, nlohmann_json_t.v1);

#define NLOHMANN_JSONIFY_ALL_THINGS(Type, ...)                                          \
  inline void to_json(nlohmann::json &nlohmann_json_j, const Type &nlohmann_json_t) {   \ 
      NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(EXTEND_JSON_TO, __VA_ARGS__))            \
  }                                                                                     \
  inline void from_json(const nlohmann::json &nlohmann_json_j, Type &nlohmann_json_t) { \
      NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(EXTEND_JSON_FROM, __VA_ARGS__))          \
  }

Categories: C++ / How to / KDAB Blogs / KDAB on Qt / Technical

Tags: / / / / / /

7 thoughts on “JSONify All Things”

  1. Hi,

    Here is the full code for variant, copied from my project:

    namespace nlohmann {
    ///////////////////////////////////////////////////////////////////////////////
    // std::variant
    ///////////////////////////////////////////////////////////////////////////////
    // Try to set the value of type T into the variant data
    // if it fails, do nothing
    template <typename T, typename... Ts>
    void variant_from_json(const nlohmann::json &j, std::variant<Ts...> &data)
    {
        try {
            data = j.get<T>();
        } catch (...) {
        }
    }
    
    template <typename... Ts>
    struct adl_serializer<std::variant<Ts...>>
    {
        static void to_json(nlohmann::json &j, const std::variant<Ts...> &data)
        {
            // Will call j = v automatically for the right type
            std::visit(
                [&j](const auto &v) {
                    j = v;
                },
                data);
        }
    
        static void from_json(const nlohmann::json &j, std::variant<Ts...> &data)
        {
            // Call variant_from_json for all types, only one will succeed
            (variant_from_json<Ts>(j, data), ...);
        }
    };
    }
    

    This should work out of the box if you have a C++17 compiler.

    1. Hi Nicolas,
      Thanks for the super useful post.
      I had the same error “a template argument list is not allowed in a declaration of a primary template” as described above when using Visual C++ 2019 and Intel Compilers 19.2 and 2022 ( all set to C++17)

      The code needed one change to add ‘nlohmann::’
      struct nlohmann::adl_serializer

      1. @Andrew
        Thank you a lot, I was wondering what went wrong, as everything was working for me. That’s what happens when you copy/paste only part of a file.

        I’ve updated both the blog and my comment, as well as added an annex with everything together.

  2. Hi.
    Sorry my example is still not working.
    I use the latest json version 3.10.5.
    I use visual Studio 2022.

    But I still get the error
    Error C2672: ‘nlohmann::basic_json<std::map,std::vector,std::string,bool,int64_t,uint64_t,double,std::allocator,nlohmann::adl_serializer,std::vector<uint8_t,std::allocator>>::get’: no matching overloaded function found (346)

    This is my code

    namespace mynamespace
    {
        using valuetype = std::variant;
    
        inline void from_json(const json& j, valuetype& value)
        {
           value = j.at("value").get();
        }
    }
    
    1. Unfortunately I don’t think it can go into the library, as the current implementations depends mostly on my use case and have some hard pre-requisites (like all types need to be exclusive for `std::variant`).
      But it’s nice that the library allows us to implement that ourselves.

  3. Just a quick heads-up if anyone is having problems compiling the code by trying to use the `NLOHMANN_JSONIFY_ALL_THINGS` macro in the same way as `NLOHMANN_DEFINE_TYPE_INTRUSIVE`. The former is _outside_ the `struct`/`class` definition, wheres the latter is inside:

    “`
    struct ComplexStruct {
    std::variant id;
    std::optional text;
    std::optional<std::variant<std::vector, bool>> values;
    };
    // Outside the struct!
    NLOHMANN_JSONIFY_ALL_THINGS(ComplexStruct, id, text, values)

    struct person {
    std::string name;
    std::string address;
    int age;

    friend ostream& operator<<(ostream& os, const person& p);

    public:
    // Inside the struct!
    NLOHMANN_DEFINE_TYPE_INTRUSIVE(person, name, address, age)
    };
    “`

    I banged my head against the wall for a little while because I copied and pasted a `struct`. To use `NLOHMANN_JSONIFY_ALL_THINGS` inside the class, replace `inline` with `friend`:

    “`
    #define NLOHMANN_JSONIFY_ALL_THINGS(Type, …) \
    friend void to_json(nlohmann::json &nlohmann_json_j, const Type &nlohmann_json_t) { \
    NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(EXTEND_JSON_TO, __VA_ARGS__)) \
    } \
    friend void from_json(const nlohmann::json& nlohmann_json_j, Type& nlohmann_json_t) { \
    NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(EXTEND_JSON_FROM, __VA_ARGS__)) \
    }
    “`

Leave a Reply

Your email address will not be published. Required fields are marked *