Creating Python bindings for Qt libraries

25.08.2020 Renato Araujo No comments

Python is a handy all-purpose language.  It can make you very productive within a short time period and has powerful expressiveness for data manipulation and processing. Yet, it’s not a great fit for lots of tasks. C++ is far better at achieving anything that needs bare metal performance, deterministic timing, or low-level access. Thankfully, some great tools are available that make it relatively easy to create Python bindings that let Python functions call into C++ code.

In this blog, we’re going to explain the process of creating Python bindings for your Qt library using one of our own open source Qt libraries as an example, KDDockWidgets. You can use the same process to create Python bindings for plain C++ (non-Qt) libraries, too. Once you see how straightforward the process is, you’ll want to add Python accessibility for all of your existing libraries. Consequently, a whole new community of programmers will be able to use them. And if you want a working example to download, you can get KDDockWidgets off GitHub.

Getting started

KDDockWidgets is our open source Qt-based library for adding a sophisticated docking system to Qt applications. (It’s got a number of advanced features that are missing from QDockWidgets.) Making this module accessible from Python requires a number of steps:

1. Installing PySide. Download the PySide2 package, which gives you access to the Qt5 APIs and data structures, in other words, Qt for Python.
2. Pointing CMake to PySide2. Create configuration files that tell CMake how to find the new PySide2 libraries and binaries. (The entire process of creating Python bindings uses CMake – check out our CMake whitepaper if you need a primer.)
3. Creating Shiboken invocation. Build a CMake file that calls the Shiboken with all the proper flags and files. Shiboken is the code generation tool that actually creates our bindings.
4. Creating a type system file. Create a file that exposes the classes that need to have bindings generated.
5. Creating a header for classes. Make a simple header file that includes all the C++ classes that the Shiboken tool will need to access during the generation process.
6. Customizing binding make. Add the CMake code to execute shiboken2 on your C++ headers and source.
7. Creating a Python module. Create a file that accesses the Python binding classes, allowing them to be used as a standard Python module.
8. Compiling bindings. Auto-generate and compile the Python and C++ wrappers that transition between the two languages.
9. Testing the Python package. Try it out and make sure it works!

While this looks like a lot of work, each step is pretty simple, and the tools will do most of the work. Let’s take a more detailed look at each step.

1. Installing PySide

You can do this pretty easily with the Python package installer; however, you must replace [Qt-Version] with the version of Qt you’re using.

~$ python3 -m pip install --index-url=http://download.qt.io/snapshots/ci/pyside/[Qt-Version]/latest/ shiboken2-generator pyside2 --trusted-host download.qt.io

The “gotcha” here is that an incompatible version of PySide2 and Qt can cause crashes. For example, if you’re building on Qt 5.15, you need to use this:

~$ python3 -m pip install --index-url=http://download.qt.io/snapshots/ci/pyside/5.15/latest/ shiboken2-generator pyside2 --trusted-host download.qt.io

You have to install the Pyside2 package directly from the repository. That’s necessary since the standard pip package doesn’t contain the Shiboken application that generates bindings, while the repository version does.

(If you’re planning on binding a plain C++ library rather than a Qt-specific C++ library, you can eliminate the pyside2 in the above step.)

Note that although Shiboken was created to handle generic C++, it has been tested primarily against Qt libraries. That means that if your library uses language features normally absent from Qt code – such as complex templates or smart pointers – you could possibly run into some issues in the generated bindings. (If you do find any bugs, you may wish to report them.)

2. Pointing CMake to PySide2

The next thing you need to do is use a bit of CMake configuration magic to ensure your build process can find PySide2 and Shiboken when required. We use two CMake scripts to do this – FindPySide2.cmake and FindShiboken.cmake – that run some inline Python code to discover where these modules are in your Python installation and set a handful of CMake variables that point to them for later use.

Thankfully, we’ve done all of that work already – you can grab those two CMake files directly from our GitHub repo and use them unchanged.

3. Creating Shiboken invocation

You could add the requisite CMake variables and configuration to your build process every time you wanted to create Python bindings. However, it’s more convenient to do this with a helper CMake file that allows you to create a Python module with a single macro call. This macro will automatically fill in all the paths, flags, libraries, and variables as needed, making it much easier for any new Python bindings you might need.

We created a CMake macro called CREATE_PYTHON_BINDINGS that sets things up. You’ll need PySide2ModuleBuild.cmake for this – just grab it from our repo and add it to your project.

4. Creating a type system file

The fourth step requires a bit more work than the previous steps, where the work is mostly done for you. This time, you must describe to the Shiboken generator the API that your C++ code will make available to Python. More specifically, you need to outline which C++ classes, methods, and types will be accessible, and what they will look like to a Python caller.

This API definition is called a typesystem specification, and it’s defined in an XML file. There is a great deal of power available; you can rename classes and methods, change or add parameters, modify access levels, even add new functions.

The typesystem definition for KDDockWidgets is:

<?xml version="1.0"?>
<!-- The package name -->
<typesystem package="KDDockWidgets">
    <!-- Pre-defined typesystem that contains types used by our class.
        PySide has one typesystem for each module, so here we use only the
        Widgets typesystem because it already includes gui and core
        typesystems -->
    <load-typesystem name="typesystem_widgets.xml" generate="no"/>
 
    <!-- Our classes are declared in a namespace, so we need to define one -->
    <namespace-type name="KDDockWidgets">
        <!-- This is used in a public virtual pure function.
             We need to declare it otherwise shiboken will ignore the
             function and will fail to create a wrapper -->
        <primitive-type name="DropAreaWithCentralFrame"/>
 
        <!-- Export our public enums and flags -->
        <enum-type name="Location"/>
        <enum-type name="MainWindowOption" flags="MainWindowOptions"/>
        <enum-type name="AddingOption"/>
        <enum-type name="RestoreOption" flags="RestoreOptions"/>
        <enum-type name="DefaultSizeMode"/>
        <enum-type name="FrameOption" flags="FrameOptions"/>
 
        <!-- Export our classes
             For classes we can use two types:
                object-type: class that does not have a copy-constructor
                             and cannot be passed as value to functions;
                value-type: class that can be passed as value for functions
 
             Here we only use 'object-type' since all our classes are
             derived from QWidget.
         -->
        <object-type name="MainWindowBase" />
        <object-type name="MainWindow" />
 
        <!-- DockWidgetBase contains an internal enum, so we declare it
             inside of the object-type -->
 
        <object-type name="DockWidgetBase" >
            <enum-type name="Option" flags="Options" />
        </object-type>
 
        <object-type name="DockWidget" />
    </namespace-type>
</typesystem>

Code sample 1 – Typesystem_kddockwidgets.xml

Three main blocks are in this definition: your package (for setup and loading other typesystem files), your public types (the enum-types), and your classes (the object-types).

Because we’re providing a straight-through API, our needs are pretty simple. We just need to list the public classes and enums, and then let Shiboken determine the types and method signatures for everything. If you need to provide finer control over how your Python interface looks, you’ll want to consult the reference docs for the XML you’ll need.

5. Creating a header for classes

When you call the CREATE_PYTHON_BINDINGS macro within PySide2ModuleBuild.cmake, you’ll need to pass it a few arguments. One is the location of a header file that includes all the classes that you might need during export. The header itself is simple; you just need to include other headers that Shiboken will need to access all type definitions. In the following example, we’ve included all the headers with the class interfaces that we’re adding to our Python package.

#pragma once

// Make "signals:", "slots:" visible as access specifiers
#define QT_ANNOTATE_ACCESS_SPECIFIER(a) __attribute__((annotate(#a)))

#include <MainWindowBase.h>
#include <MainWindow.h>
#include <DockWidgetBase.h>
#include <DockWidget.h>

Code sample 2 – kddockwidgets_global.h

The special QT_ANNOTATE_ACCESS_SPECIFIER line ensures that signals and slots are visible to the generator. (Since this header file is only used by the generator, it’s not harmful to leave it in even if you don’t need it.)

6. Customizing binding make

Here, you set up CMake to invoke Shiboken and add any necessary dependencies. You’ll need to customize this file for your application. Replace all the instances of KDDockWidgets directories, source, and header files with those from your application.

# Auto-Generate files - every class will have a cpp and h file
set(PyKDDockWidgets_SRC
    # individual classes
    ${CMAKE_CURRENT_BINARY_DIR}/KDDockWidgets/kddockwidgets_dockwidgetbase_wrapper.cpp
    ${CMAKE_CURRENT_BINARY_DIR}/KDDockWidgets/kddockwidgets_dockwidgetbase_wrapper.h
    ${CMAKE_CURRENT_BINARY_DIR}/KDDockWidgets/kddockwidgets_dockwidget_wrapper.cpp
    ${CMAKE_CURRENT_BINARY_DIR}/KDDockWidgets/kddockwidgets_dockwidget_wrapper.h
    ${CMAKE_CURRENT_BINARY_DIR}/KDDockWidgets/kddockwidgets_mainwindowbase_wrapper.cpp
    ${CMAKE_CURRENT_BINARY_DIR}/KDDockWidgets/kddockwidgets_mainwindowbase_wrapper.h
    ${CMAKE_CURRENT_BINARY_DIR}/KDDockWidgets/kddockwidgets_mainwindow_wrapper.cpp
    ${CMAKE_CURRENT_BINARY_DIR}/KDDockWidgets/kddockwidgets_mainwindow_wrapper.h
    # namespace wrapper
    ${CMAKE_CURRENT_BINARY_DIR}/KDDockWidgets/kddockwidgets_wrapper.cpp
    ${CMAKE_CURRENT_BINARY_DIR}/KDDockWidgets/kddockwidgets_wrapper.h
    # global module wrapper
    ${CMAKE_CURRENT_BINARY_DIR}/KDDockWidgets/kddockwidgets_module_wrapper.cpp
    ${CMAKE_CURRENT_BINARY_DIR}/KDDockWidgets/kddockwidgets_python.h
)
                                                      
# The includes are needed to parse and build classes specified in our typesystem
set(PyKDDockWidgets_include_paths
    $<JOIN:$<TARGET_PROPERTY:KDAB::kddockwidgets,INTERFACE_INCLUDE_DIRECTORIES>,${PATH_SEP}>
)

# Set list of paths where shiboken should look for typesystem
set(PyKDDockWidgets_typesystem_paths
    # PySide path, this variable was exposed by FindPySide2.cmake
    ${PYSIDE_TYPESYSTEMS}
)

# Include flags/path that will be set in 'target_include_directories'
set(PyKDDockWidgets_target_include_directories
    ${CMAKE_SOURCE_DIR}
)

# Libraries necessary to link the target for the command 'target_link_libraries'
set(PyKDDockWidgets_target_link_libraries
    KDAB::kddockwidgets
    Qt5::Core
    Qt5::Gui
    Qt5::Widgets
)

# Dependencies - changes on these files should trigger new bindings
set(PyKDDockWidgets_DEPENDS
    ${CMAKE_CURRENT_SOURCE_DIR}/kddockwidgets_global.h
    ${CMAKE_SOURCE_DIR}/src/DockWidgetBase.h
    ${CMAKE_SOURCE_DIR}/src/DockWidget.h
    ${CMAKE_SOURCE_DIR}/src/MainWindowBase.h
    ${CMAKE_SOURCE_DIR}/src/MainWindow.h
)

CREATE_PYTHON_BINDINGS(
    "KDDockWidgets"
    "${PyKDDockWidgets_typesystem_paths}"
    "${PyKDDockWidgets_include_paths}"
    "${PyKDDockWidgets_SRC}"
    "${PyKDDockWidgets_target_include_directories}"
    "${PyKDDockWidgets_target_link_libraries}"
    ${CMAKE_CURRENT_SOURCE_DIR}/kddockwidgets_global.h
    ${CMAKE_CURRENT_SOURCE_DIR}/typesystem_kddockwidgets.xml
    "${PyKDDockWidgets_DEPENDS}"
    ${CMAKE_CURRENT_BINARY_DIR}
)

# Make moduled import from build dir works
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/__init__.py ${CMAKE_CURRENT_BINARY_DIR}/__init__.py)

# install
install(FILES ${CMAKE_CURRENT_SOURCE_DIR}/__init__.py DESTINATION ${Python3_SITELIB}/PyKDDockWidgets)

Code sample 3 – CMakeLists.txt

7. Creating a Python module

If you ran everything right now, you’d have created code that allows Python to call into C++. However, you’d still need to add one small file to the package directory to turn your code into an importable Python package.

__all__ = ['KDDockWidgets']

# Preload PySide2 libraries to avoid missing libraries while loading our module
try:
    from PySide2 import QtCore
except Exception:
    print("Failed to load PySide")
    raise

# Avoid duplicate namespace - the package name would normally look like this:
# PyKDDockWidgets.KDDockWidgets.KDDockWidgets.MainWindow
# (There is a bug PYSIDE-1325 to get this namespace duplication fixed.)
# To avoid this, we use this workaround:
from .KDDockWidgets import KDDockWidgets as _priv
KDDockWidgets = _priv

Code sample 4 – __init__.py

You’ll note we have a small workaround in this example for namespace duplication. If PYSIDE-1325 gets fixed, you won’t need the last several lines of code. In fact, you wouldn’t need any code at all since an empty __init__.py is all that Python requires to turn your code into a package.

8. Running the generator

With everything in place, you can now let the tools do all the work! Invoke CMake, and it will generate the necessary make file. Then, running make will generate and compile your binding files, adding the new Python module to your Python installation.

9. Testing the Python package

Assuming your generation and compilation was successful, you’ve now got a Python package that calls your C++ library. You can import it and start using it, just like a regular Python class.

from PyKDDockWidgets import KDDockWidgets

options = KDDockWidgets.MainWindowOption_None

dock = KDDockWidgets.DockWidget("new dock 1")
dock.resize(600, 600)
dock.show()

Code sample 5 – example.py

Summary

Using PySide and Shiboken to create Python bindings does take a bit of setup, but it’s not anywhere as difficult as doing all the Python/C++ interface work by hand! Steps 1 through 3 of the process need only be done once. After that, for each new C++ library you’ll need a new typesystem XML file, a few customizations to your CMake file, and a new header to include the binding-specific classes – all tasks that can be accomplished in a few minutes.

You’ll be Python-enabled before you know it!