Choosing an Embedded Language
In my multi-year toy project, I decided it would be useful to incorporate an embedded programming language into the app to make adding features a little easier. It basically came down to three possible languages that I thought would be appropriate: python, javascript, and lua. Lua is known for being light-weight and fast, and is used in many games like World of Warcraft to provide scripting to the user. Javascript itself seems to have had a bit of a resurgence in recent years in terms of stepping outside of web development. Both languages have a variety of implementations that seemed workable including luabind for lua and V8 (produced by Google) for javascript specificially for embedding into an application. Both provide a reasonable interface for mapping C++ classes.
Although both languages seemed appealing, I wasn't really satisfied with how the mappings were written to expose the C++ classes. Also, I have much more experience with python, so I had to decide if I wanted to learn how to bind a scripting engine to my application AND learn a new scripting language. I've done quite a bit of javascript, but never for a large application and wasn't sure how well things would map between it and C++, as both lua and javascript don't technically support objects the same way as python. I finally decided just to go with python. Although not as fast or light-weight as lua or javascript, I felt it provided a better one-to-one mapping of my classes, and figured users might be more comfortable with a more C-like language for application scripting.
Which Python Binding?
I ended up trying several different libraries to embed python into my app.
PythonQt seemed like a good candidate as I was writing my app using the
Qt libraries, but I encountered a few strange bugs and the community seemed to be stagnating--unfortunate as the API seemed really intuitive. Both
SIP and
SWIG are popular for binding, but both require a special syntax in external files, and I wanted to modify my qmake build as little as possible and didn't want to learn a new syntax. After finally experimenting with
boost::python, I found the library allowed me to write my mappings inside C++ without learning any new syntax or much with my build system.
Using boost::python
I had a Scene class, which naturally handles everything in my scene, which I wanted to expose to python.
boost::python has a special function called
BOOST_PYTHON_MODULE, which puts a class into a particular module, which is imported into python. Once wrapping my Scene class with the
class_ function, I could then import the Scene class into python from the "scene" module. The
boost::noncopyable is an optional argument that notes not to pass the Scene object to python by value, since my scene might be rather large in memory and I didn't want multiple copies. This is more of a compiler rule as I still have to make sure I'm not passing the scene by value, but with that I get a compiler error if I try.
BOOST_PYTHON_MODULE(scene)
{
class_<Scene, boost::noncopyable>("Scene");
}
I have also been trying to use smart pointers for heap-allocated objects. I started out using QSharedPointer, but boost::shared_ptr is already supported by boost, so I ended up switching my smart pointers over to boost's.
typedef boost::shared_ptr SceneP;
Sending Yourself to Python
Once that is setup, I could then pass my Scene object over to python. I immediately hit a snag. In particular to my Scene class, the Scene actually contained my python engine and called the python code.
object ignored = exec(EXAMPLE_PY_FUNCTION, pyMainNamespace);
object processFileFunc = pyMainModule.attr("Foo").attr("processFile");
processFileFunc(this, "test.txt"); // "this" being the Scene object
Although I created the scene in a smart pointer outside the class, I didn't have access to that smart pointer inside member functions to pass to python unless I passed it into the function, which seemed unnecessary to pass a Scene member function a smart pointer to itself. I couldn't use "this" inside the member function as boost::python wouldn't know by default how to keep that in memory since other boost pointers were already pointing to my Scene object. I couldn't just create a
shared_ptr in the function either, because I didn't want my Scene deleted when the shared pointer goes out of scope when the function returns.
I was actually getting this error because boost didn't know how to deal with the
this pointer without passing it by value (which I explicitly said I didn't want copied, right?).
Error in Python: : No to_python
(by-value) converter found for C++ type: Scene
It turns out boost::python has a special way to do deal with this situation using a special class called
boost::enable_shared_from_this, which my Scene class can inherit from.
class Scene : public boost::enable_shared_from_this<Scene>
boost::enable_shared_from_this provides two functions that allow the Scene object to create shared pointers inside member functions by calling
shared_from_this(), which is inherited from
boost::enable_shared_from_this.
object ignored = exec(EXAMPLE_PY_FUNCTION, pyMainNamespace);
object processFileFunc = pyMainModule.attr("Foo").attr("processFile");
// pass the python function a shared pointer instead of "this"
processFileFunc(shared_from_this(), "test.txt"); // can't use boost::shared_ptr(this) either
After updating the member function, the error went away and I could then send my Scene object to python inside and outside of the member function. Python now has access to my scene object and I can start exposing some more functions and variables inside my Scene class.
Below is a short working example of sending a shared pointer to python inside and outside of a member function (or you can look
here). Special thanks to the boost::python mailing list, which was very helpful in getting me going.
#include <iostream>
#include <boost/python.hpp>
#include <boost/python/class.hpp>
#include <boost/python/module.hpp>
#include <boost/python/def.hpp>
#include <boost/enable_shared_from_this.hpp>
using namespace boost::python;
object pyMainModule;
object pyMainNamespace;
#define EXAMPLE_PY_FUNCTION \\
\"from scene import Scene\\n\" \\
\"class Foo(object):\\n\" \\
\" @staticmethod\\n\" \\
\" def processFile(scene, filename):\\n\" \\
\" print(\'here\')\\n\"
std::string parse_python_exception();
class Scene : public boost::enable_shared_from_this<Scene>
{
public:
void sendYourselfToPython()
{
try {
object ignored = exec(EXAMPLE_PY_FUNCTION, pyMainNamespace);
object processFileFunc = pyMainModule.attr(\"Foo\").attr(\"processFile\");
processFileFunc(shared_from_this(), \"test.txt\");
} catch (boost::python::error_already_set const &) {
std::string perror = parse_python_exception();
std::cerr << \"Error in Python: \" << perror << std::endl;
}
}
};
typedef boost::shared_ptr<Scene> SceneP;
BOOST_PYTHON_MODULE(scene)
{
class_<Scene, boost::noncopyable>(\"Scene\");
}
main(int argc, char**argv)
{
std::cout << \"starting program...\" << std::endl;
Py_Initialize();
pyMainModule = import(\"__main__\");
pyMainNamespace = pyMainModule.attr(\"__dict__\");
boost::python::register_ptr_to_python< boost::shared_ptr<Scene> >();
PyImport_AppendInittab(\"scene\", &initscene);
SceneP scene(new Scene());
// sending Scene object to python inside member function
scene->sendYourselfToPython();
try {
object ignored = exec(EXAMPLE_PY_FUNCTION, pyMainNamespace);
object processFileFunc = pyMainModule.attr(\"Foo\").attr(\"processFile\");
// send Scene object to python using smart pointer
processFileFunc(scene, \"test.txt\");
} catch (boost::python::error_already_set const &) {
std::string perror = parse_python_exception();
std::cerr << \"Error in Python: \" << perror << std::endl;
}
}
// taken from http://thejosephturner.com/blog/2011/06/15/embedding-python-in-c-applications-with-boostpython-part-2/
namespace py = boost::python;
std::string parse_python_exception() {
PyObject *type_ptr = NULL, *value_ptr = NULL, *traceback_ptr = NULL;
PyErr_Fetch(&type_ptr, &value_ptr, &traceback_ptr);
std::string ret(\"Unfetchable Python error\");
if (type_ptr != NULL) {
py::handle<> h_type(type_ptr);
py::str type_pstr(h_type);
py::extract<std::string> e_type_pstr(type_pstr);
if(e_type_pstr.check())
ret = e_type_pstr();
else
ret = \"Unknown exception type\";
}
if (value_ptr != NULL) {
py::handle<> h_val(value_ptr);
py::str a(h_val);
py::extract<std::string> returned(a);
if(returned.check())
ret += \": \" + returned();
else
ret += std::string(\": Unparseable Python error: \");
}
if (traceback_ptr != NULL) {
py::handle<> h_tb(traceback_ptr);
py::object tb(py::import(\"traceback\"));
py::object fmt_tb(tb.attr(\"format_tb\"));
py::object tb_list(fmt_tb(h_tb));
py::object tb_str(py::str(\"\\n\").join(tb_list));
py::extract<std::string> returned(tb_str);
if(returned.check())
ret += \": \" + returned();
else
ret += std::string(\": Unparseable Python traceback\");
}
return ret;
}
Compiling the Code
To compile the code, I used
python-config to get the includes and flags.
python-config is a simple utility that queries the path of your python headers and libs depending on which version of python is installed and designated on you system. It's a useful utility as I usually have several versions of python on my machine at a time. It's especially nice not having to hard code your application's build system to a particular version of python.
python-config --includes
python-config --libs
g++ test.cpp -I/usr/include/python2.7 -I/usr/include/python2.7 -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -lpthread -ldl -lutil -lm -lpython2.7 -lboost_python
