Adding Python Support using Boost::Python

Choosing an Embedded Language
In my multi-year toy project, I decided it would be useful to incorporate an embedded programming language into the app to make adding features a little easier.  It basically came down to three possible languages that I thought would be appropriate: python, javascript, and lua.  Lua is known for being light-weight and fast, and is used in many games like World of Warcraft to provide scripting to the user.  Javascript itself seems to have had a bit of a resurgence in recent years in terms of stepping outside of web development.  Both languages have a variety of implementations that seemed workable including luabind for lua and V8 (produced by Google) for javascript specificially for embedding into an application.  Both provide a reasonable interface for mapping C++ classes.

Although both languages seemed appealing, I wasn't really satisfied with how the mappings were written to expose the C++ classes.  Also, I have much more experience with python, so I had to decide if I wanted to learn how to bind a scripting engine to my application AND learn a new scripting language.  I've done quite a bit of javascript, but never for a large application and wasn't sure how well things would map between it and C++, as both lua and javascript don't technically support objects the same way as python.  I finally decided just to go with python.  Although not as fast or light-weight as lua or javascript, I felt it provided a better one-to-one mapping of my classes, and figured users might be more comfortable with a more C-like language for application scripting.

Which Python Binding?
I ended up trying several different libraries to embed python into my app.  PythonQt seemed like a good candidate as I was writing my app using the Qt libraries, but I encountered a few strange bugs and the community seemed to be stagnating--unfortunate as the API seemed really intuitive.  Both SIP and SWIG are popular for binding, but both require a special syntax in external files, and I wanted to modify my qmake build as little as possible and didn't want to learn a new syntax.  After finally experimenting with boost::python, I found the library allowed me to write my mappings inside C++ without learning any new syntax or much with my build system.

Using boost::python
I had a Scene class, which naturally handles everything in my scene, which I wanted to expose to python.  boost::python has a special function called BOOST_PYTHON_MODULE, which puts a class into a particular module, which is imported into python.  Once wrapping my Scene class with the class_ function, I could then import the Scene class into python from the "scene" module.  The boost::noncopyable is an optional argument that notes not to pass the Scene object to python by value, since my scene might be rather large in memory and I didn't want multiple copies.  This is more of a compiler rule as I still have to make sure I'm not passing the scene by value, but with that I get a compiler error if I try.


BOOST_PYTHON_MODULE(scene)
{
  class_<Scene, boost::noncopyable>("Scene");
}

I have also been trying to use smart pointers for heap-allocated objects.  I started out using QSharedPointer, but boost::shared_ptr is already supported by boost, so I ended up switching my smart pointers over to boost's.
typedef boost::shared_ptr SceneP;

Sending Yourself to Python
Once that is setup, I could then pass my Scene object over to python.  I immediately hit a snag.  In particular to my Scene class, the Scene actually contained my python engine and called the python code.


      object ignored = exec(EXAMPLE_PY_FUNCTION, pyMainNamespace);
      object processFileFunc = pyMainModule.attr("Foo").attr("processFile");
      processFileFunc(this, "test.txt"); // "this" being the Scene object

Although I created the scene in a smart pointer outside the class, I didn't have access to that smart pointer inside member functions to pass to python unless I passed it into the function, which seemed unnecessary to pass a Scene member function a smart pointer to itself.  I couldn't use "this" inside the member function as boost::python wouldn't know by default how to keep that in memory since other boost pointers were already pointing to my Scene object.  I couldn't just create a shared_ptr in the function either, because I didn't want my Scene deleted when the shared pointer goes out of scope when the function returns.


I was actually getting this error because boost didn't know how to deal with the this pointer without passing it by value (which I explicitly said I didn't want copied, right?).

Error in Python: : No to_python
(by-value) converter found for C++ type: Scene



It turns out boost::python has a special way to do deal with this situation using a special class called  boost::enable_shared_from_this, which my Scene class can inherit from.

class Scene : public boost::enable_shared_from_this<Scene>

boost::enable_shared_from_this provides two functions that allow the Scene object to create shared pointers inside member functions by calling shared_from_this(), which is inherited from boost::enable_shared_from_this.


      object ignored = exec(EXAMPLE_PY_FUNCTION, pyMainNamespace);
      object processFileFunc = pyMainModule.attr("Foo").attr("processFile");
      // pass the python function a shared pointer instead of "this"
      processFileFunc(shared_from_this(), "test.txt"); // can't use boost::shared_ptr(this) either



After updating the member function, the error went away and I could then send my Scene object to python inside and outside of the member function.  Python now has access to my scene object and I can start exposing some more functions and variables inside my Scene class.

Below is a short working example of sending a shared pointer to python inside and outside of a member function (or you can look here).  Special thanks to the boost::python mailing list, which was very helpful in getting me going.



#include <iostream>




#include <boost/python.hpp>

#include <boost/python/class.hpp>

#include <boost/python/module.hpp>

#include <boost/python/def.hpp>

#include <boost/enable_shared_from_this.hpp>

using namespace boost::python;




object pyMainModule;

object pyMainNamespace;




#define EXAMPLE_PY_FUNCTION \\

  \"from scene import Scene\\n\" \\

  \"class Foo(object):\\n\" \\

  \"  @staticmethod\\n\" \\

  \"  def processFile(scene, filename):\\n\" \\

  \"    print(\'here\')\\n\"




std::string parse_python_exception();




class Scene : public boost::enable_shared_from_this<Scene>

{

public:

  void sendYourselfToPython()

  {

    try {

      object ignored = exec(EXAMPLE_PY_FUNCTION, pyMainNamespace);

      object processFileFunc = pyMainModule.attr(\"Foo\").attr(\"processFile\");

      processFileFunc(shared_from_this(), \"test.txt\");

    } catch (boost::python::error_already_set const &) {

      std::string perror = parse_python_exception();

      std::cerr << \"Error in Python: \" << perror << std::endl;

    }

  }

};

typedef boost::shared_ptr<Scene> SceneP;




BOOST_PYTHON_MODULE(scene)

{

  class_<Scene, boost::noncopyable>(\"Scene\");

}




main(int argc, char**argv)

{

  std::cout << \"starting program...\" << std::endl;




  Py_Initialize();

  pyMainModule = import(\"__main__\");

  pyMainNamespace = pyMainModule.attr(\"__dict__\");




  boost::python::register_ptr_to_python< boost::shared_ptr<Scene> >();

  PyImport_AppendInittab(\"scene\", &initscene);




  SceneP scene(new Scene());


  // sending Scene object to python inside member function

  scene->sendYourselfToPython(); 




  try {

    object ignored = exec(EXAMPLE_PY_FUNCTION, pyMainNamespace);

    object processFileFunc = pyMainModule.attr(\"Foo\").attr(\"processFile\");


    // send Scene object to python using smart pointer

    processFileFunc(scene, \"test.txt\");

  } catch (boost::python::error_already_set const &) {

    std::string perror = parse_python_exception();

    std::cerr << \"Error in Python: \" << perror << std::endl;

  }

}







// taken from http://thejosephturner.com/blog/2011/06/15/embedding-python-in-c-applications-with-boostpython-part-2/

namespace py = boost::python;

std::string parse_python_exception() {

    PyObject *type_ptr = NULL, *value_ptr = NULL, *traceback_ptr = NULL;

    PyErr_Fetch(&type_ptr, &value_ptr, &traceback_ptr);

    std::string ret(\"Unfetchable Python error\");

    if (type_ptr != NULL) {

        py::handle<> h_type(type_ptr);

        py::str type_pstr(h_type);

        py::extract<std::string> e_type_pstr(type_pstr);

        if(e_type_pstr.check())

            ret = e_type_pstr();

        else

            ret = \"Unknown exception type\";

    }




    if (value_ptr != NULL) {

        py::handle<> h_val(value_ptr);

        py::str a(h_val);

        py::extract<std::string> returned(a);

        if(returned.check())

            ret +=  \": \" + returned();

        else

            ret += std::string(\": Unparseable Python error: \");

    }




    if (traceback_ptr != NULL) {

        py::handle<> h_tb(traceback_ptr);

        py::object tb(py::import(\"traceback\"));

        py::object fmt_tb(tb.attr(\"format_tb\"));

        py::object tb_list(fmt_tb(h_tb));

        py::object tb_str(py::str(\"\\n\").join(tb_list));

        py::extract<std::string> returned(tb_str);

        if(returned.check())

            ret += \": \" + returned();

        else

            ret += std::string(\": Unparseable Python traceback\");

    }

    return ret;

}




Compiling the Code
To compile the code, I used python-config to get the includes and flags.  python-config is a simple utility that queries the path of your python headers and libs depending on which version of python is installed and designated on you system.  It's a useful utility as I usually have several versions of python on my machine at a time.  It's especially nice not having to hard code your application's build system to a particular version of python.


python-config --includes
python-config --libs

g++ test.cpp -I/usr/include/python2.7 -I/usr/include/python2.7 -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -lpthread -ldl -lutil -lm -lpython2.7 -lboost_python