Monday, June 30, 2008

boost::python examples

Continued from the previous post.

These are equivalent version of the examples on the previous post using boost::python.

[main2boost.cpp]


#include <boost/python/detail/wrap_python.hpp>
#include <boost/python/object.hpp>

using namespace boost::python;

int main()
{
Py_Initialize();

object o_main = object(handle<>(borrowed(PyImport_AddModule("__main__"))));
object o_int10000(10000);
o_main.attr("tamtam") = o_int10000;
PyRun_SimpleString("print tamtam");

Py_Finalize();
return 0;
}


[main4boost.cpp]

#include <boost/python/detail/wrap_python.hpp>
#include <boost/python/object.hpp>
#include <boost/python/dict.hpp>

using namespace boost::python;

int main()
{
Py_Initialize();

object o_main = object(handle<>(borrowed(PyImport_AddModule("__main__"))));
dict o_dict;
o_dict["foo"] = "bar";
o_main.attr("tamtam") = o_dict;

PyRun_SimpleString("print tamtam");

Py_Finalize();
return 0;
}


[main5boost.cpp]

#include <boost/python/detail/wrap_python.hpp>
#include <boost/python/object.hpp>
#include <boost/python/dict.hpp>
#include <boost/python/extract.hpp>
#include <iostream>
#include <string>

using namespace boost::python;

int main()
{
Py_Initialize();

PyRun_SimpleString("tamtam = {'foo':'bar'}");
object o_main = object(handle<>(borrowed(PyImport_AddModule("__main__"))));
dict o_dict(o_main.attr("tamtam"));
object o_value = o_dict["foo"];

//Oops I couldn't find a boost::python way of type checking.
if(PyObject_IsInstance(o_value.ptr(), (PyObject*)&PyString_Type))
{
std::string valstr = extract < std::string > (o_value);
std::cout << valstr << std::endl;
}

Py_Finalize();
return 0;
}


[main6boost.cpp]

#include <boost/python/detail/wrap_python.hpp>
#include <boost/python/object.hpp>
#include <boost/python/extract.hpp>
#include <iostream>
#include <string>

using namespace boost::python;

int main()
{
Py_Initialize();

PyRun_SimpleString(
"def myfunc(a, b, c):¥n"
" print a¥n"
" return b + c¥n"
);

object o_main = object(handle<>(borrowed(PyImport_AddModule("__main__"))));

object o_func = o_main.attr("myfunc");

object o_int(10000);
object o_foo("foo");
object o_bar("bar");

object o_result = o_func(o_int, o_foo, o_bar);
//object o_result = o_func(10000, "foo", "bar"); also works.

std::string resultstr = extract < std::string > (o_result);
std::cout << resultstr << std::endl;

Py_Finalize();
return 0;
}


- o_ prefix is my naming convension

- To use Python C/API function directly, you need to include boost/python/detail/wrap_python.hpp instead of including Python.h. Include it before any other, just like Python.h

- borrowed() is used when you are passing a Python object that you are not responsible to dereference. That's all you need to be careful around reference counter, boost::python does everthing else.

- You'll find a better tut here.
http://www.boost.org/doc/libs/1_35_0/libs/python/doc/tutorial/doc/html/index.html
Document about "Exception handling" in the "Embedding" chapter is a must to read.

- Though I didn't used here, there's a function exec() that can replace PyRun_*().

Python C/API tutorial

Upon my friend's request, I wrote a small Python C/API tutorial. I don't really recommend you use Python C/API directly, without the help of tools like boost::python, swig, pyrex, ... unless your code is really performance sensitive, or very small, but still it'll be good to know what it is like for a better understanding of Python/C interaction.

The reader is expected to know
- Python scripting
- Basic C++ programming
- How CPython handles object lifetime using reference counter

This tutorial is only for embedding. Extending (making a Python module in C) also uses Python C/API but it's much more tedious and I don't even want to think of it without a tool. It also doesn't cover threading stuff. Basically these are simple example programs that are not thread safe. You'll need to read Python C/API reference manual closely (global interpreter lock etc.) to make it thread safe.

I used Linux to write this doc but everything except compilation is platform independent.

[main1.cpp]



#include <Python.h>

int main()
{
Py_Initialize();
PyRun_SimpleString("print 'Hello Python C/API'");
Py_Finalize();
return 0;
}


$ gcc main1.cpp -I/usr/include/python2.5 -lpython2.5 -lstdc++
$ ./a.out
Hello Python C/API

This is the first Python C/API program. You can find anywhere about this program on the net so I'll skip explanation (and it's fully self explanatory after all). Just several things,

- Don't include Python.h like #include <python2.5/Python.h> even if you feel its better.

- When you use Python C/API, don't use names that start Py and _Py because it's reserved for Python C/API.

- Probably you will want to use -Wall and -O3 options when you compile.

- For Maya guys: If you are using Python C/API in a Maya plug-in, it'll be better compiling it with Maya Python
$ gcc main1.cpp -I/usr/autodesk/maya2008/include/python2.5 -L/usr/autodesk/maya2008/lib -lpython2.5 -lstdc++ (or something like this. I don't have maya2008. I'm just guessing the directory structure will be like this)

- For Maya guys: You don't need Py_Initialize(); because Python is already running. You don't need Py_Finalize() either because you don't want to terminate Python interpreter.

(Maya is a commercial software that is used for computer graphics)

[main2.cpp]

#include <Python.h>

int main()
{
Py_Initialize();

PyObject* po_main = PyImport_AddModule("__main__");
PyObject* po_int10000 = PyInt_FromLong(10000);
PyObject_SetAttrString(po_main, "tamtam", po_int10000);
Py_XDECREF(po_int10000);
PyRun_SimpleString("print tamtam");

Py_Finalize();
return 0;
}


$ ./a.out
10000

It gets a PyObject* to the main module, create an int Python object, set it to an attribute "tamtam" in the main module, and print it.

- You need to dereference po_int10000 after using it with Py_XDECREF() macro (or Py_DECREF if you are sure po_int10000 is non-null).

- PyObject_SetAttrString increments po_int10000 referene counter so when you execute Py_XDECREF its reference count still doesn't become zero, that's why subsequent PyRun_SimpleString("print tamtam") can print out that value.

- You can use PyImport_AddModule only when you know Python has the module already (You can use it when the module is not imported, but no module is imported and it returns an empty new module). Use PyImport_ImportModule() instead. Unlike PyImport_AddModule, you need to dereference it after using it. Usually you can see whether you need to dereference an object or not by looking at the Python C/API reference manual.

- Usually Python C/API functions that returns PyObject* returns NULL on failure. You'll need to check it. Though I omitted it for simplicity in this tut, you should at least take a look at Exception Handling chapter o the Python C/API manual.

- Many Python C/API functions which name end with "String" have a sibling. For example, PyObject_SetAttrString() has a sibling PyObject_SetAttr(), which takes PyObject* for attribute name instead of const char*. If you use it, it'll be like

PyObject* po_int10000 = PyInt_FromLong(10000);
PyObject* po_tamtam = PyString_FromString("tamtam");
PyObject_SetAttr(po_main, po_tamtam, po_int10000);

- More about Python reference lator (see [main5.cpp])

- "po_" prefix is just my naming convention indicating it's a PyObject*. You can follow it, or you can ignore it, either.

[main3.cpp]

#include <Python.h>
#include <iostream>

int main()
{
Py_Initialize();

PyObject* po_main = PyImport_AddModule("__main__");
PyObject* po_int10000 = PyInt_FromLong(10000);
std::cout << po_int10000->ob_refcnt << std::endl;
PyObject_SetAttrString(po_main, "tamtam", po_int10000);
std::cout << po_int10000->ob_refcnt << std::endl;
Py_XDECREF(po_int10000);
PyRun_SimpleString("print tamtam");

Py_Finalize();
return 0;
}


$ ./a.out
1
2
10000

It demonstrates how to print out an object(here an int object)'s reference counter. Usually it's enough to look at the manual but sometimes you will want to confirm it.

- include Python.h before any standard libraries

- when you change 10000 to 10, the result will change.

./a.out
12
13
10

That's because Python is using 'int 10' object somewhere else already (it may be for internal purposes). Also when Python interpreter has started, it already has a number of frequently used immutable objects so that it doesn't have to be recreated when requested.

[main4.cpp]

#include <Python.h>

int main()
{
Py_Initialize();

PyObject* po_dict = PyDict_New();
PyObject* po_bar = PyString_FromString("bar");
PyDict_SetItemString(po_dict, "foo", po_bar);
Py_DECREF(po_bar);

PyObject* po_main = PyImport_AddModule("__main__");
PyObject_SetAttrString(po_main, "tamtam", po_dict);
Py_DECREF(po_dict);

PyRun_SimpleString("print tamtam");

Py_Finalize();
return 0;
}


$ ./a.out
{'foo': 'bar'}

A little bit more practical example that uses Python dictionary.

- Refer to Python C/API reference manual to see what functions exists for dictionary objects.

- A Python dictionary is a Python object (ofcourse), so those functions in the Object Protocol an be used for a dictionary too. And if you see the manual, you'll see the documentation for dictionay is under "Mapping Objects". It means you can use functions in the "Mapping Protocol" section for a dictionary object.

- As you may have guessed, PyDict_SetItemString() has a sibling PyDict_SetItem().


[main5.cpp]

#include <Python.h>
#include <iostream>
#include <string>

int main()
{
Py_Initialize();

PyRun_SimpleString("tamtam = {'foo':'bar'}");
PyObject* po_main = PyImport_AddModule("__main__");
PyObject* po_dict = PyObject_GetAttrString(po_main, "tamtam");

PyObject* po_value = PyDict_GetItemString(po_dict, "foo");
if(PyObject_IsInstance(po_value, (PyObject*)&PyString_Type))
{
std::string valstr = PyString_AsString(po_value);
std::cout << valstr << std::endl;
}

Py_DECREF(po_value);
Py_DECREF(po_dict);

Py_Finalize();
return 0;
}


$ ./a.out
bar

How to get a value from a dictionary.

- If you see a document for PyObject_GetAttrString() and PyObject_GetAttrString(), you'll see it says

return Value: New reference.

It means you are responsible to dereference it after using it. Sometimes you'll see

return Value: Borrowed reference.

It means you don't have to dereference it after using it. If you see PyImport_AddModule() doc, it says so, that's why the code doesn't have Py_DECREF(po_main);

Sometimes you may find a term "steal a reference". It means when you pass an object that you are responsible to dereference to such a Python C/API function, Python does it so you don't have to do it any more.

- PyObject_IsInstance(po_value, (PyObject*)&PyString_Type) This is how to type check an object. PyString_Type is also a Python objecjt (everything is a Python object in Python) so you can cast it but I will not go into details. You can find type objects like PyString_Type in the manual.


[main6.cpp]

#include <Python.h>
#include <iostream>
#include <string>

int main()
{
Py_Initialize();

PyRun_SimpleString(
"def myfunc(a, b, c):¥n"
" print a¥n"
" return b + c¥n"
);

PyObject* po_main = PyImport_AddModule("__main__");
PyObject* po_func = PyObject_GetAttrString(po_main, "myfunc");

PyObject* po_int = PyInt_FromLong(10000);
PyObject* po_foo = PyString_FromString("foo");
PyObject* po_bar = PyString_FromString("bar");

PyObject* po_result = PyObject_CallFunctionObjArgs(po_func, po_int, po_foo, po_bar, NULL);
std::string resultstr = PyString_AsString(po_result);
std::cout << resultstr << std::endl;

Py_DECREF(po_result);
Py_DECREF(po_bar);
Py_DECREF(po_foo);
Py_DECREF(po_int);
Py_DECREF(po_func);

Py_Finalize();
return 0;
}


$ ./a.out
10000
foobar

Function call example.

- PyObject protocol has a set of function/method call API functions.
- These functions return a PyObject* that represents the return value of the python function. If the function fails (either the function call itself fails or the function call throws an exception and nobody catchs), it returns NULL. Note that it doesn't return NULL if None is returned, in that case a PyObject* that points to the None object is returned.



To be continued to the next post which shows how you can write these examples using boost::python.

Sunday, June 29, 2008

Shake Command Window Release 0.94 preannouncement

Release 0.94 preannouncement:
- Open window by pressing macro button
- At-least-not-crash level fail safe on pointer/floating point/pipe related runtime errors
- Fix freeze bug when pressing macro button while settings panel is open
- Fix freeze bug when trying to display non-ascii string
- Maybe it'll come with several utility functions.

These are already implemented. I need to pack it in a new release and write docs for them

Friday, June 27, 2008

Footprint

Please leave any comment if you've come here.

Wednesday, June 25, 2008

Mercurial v.s. Subversion

This post is not quite accurate, strictly speaking

I had been using subversion and I started using Mercurial. Mercurial is one of distributed version control systems. In Mercurial, repository informations are distributed to each developer's environment, i.e. local machine. So each has its own repository and you can merge repositories to share changes done by multiple developers. (A repository is a place or data to hold/manage source code. In subversion, it usually exists on a server)

Mercurial has many advantages over subversion but it has also downsides.

Pros:

- You can commit changes without connecting to the server.
You can commit changes to the local repository, which is at the same directory where the source code exists.

- You can commit changes locally without affecting your co-workers' works
If you commit changes to the local repository and do not reflect it to the shared repository, others don't have to know your changes. So you can commit whenever you want without worrying about stopping people's works by your bug.

- Repository location is relocatable
You can clone a repository as many times as you want.

- Making branches, merges are fully traceable
When you make a branch, the branch knows where it is forked from.

- Commit is fast.
It doesn't need server communication.

- Written in Python
Easy customize

Cons:

- To share your changes to others, you need to update everything.
you'll often need to
1) commit your modification to the local repository
2) reflect your co-worker's changes to your repository.
3) merge them
4) if there's a conflict, resolve it
5) commit
6) Reflect your repository to the shared repository.

- Changes between you and others can be big.
It may take a long time until a conflict gets unrevealed after it's been created.

- Ability of local commit can make people not tempted to share their works for a long time
A bad developer may keep their changes not being shared for a year.

- Not good at binary differencing,
A repository can be big

- You need to have a full size of repository clone in the local disk
It takes lots of disk space if the project is large
(but one guy will try to implement partial cloning this summer. hurrah)

- Cannot commit an empty directory
To do that, you need to make a dummy file in a directory, or write a script to make an empty directry. I think it's important. As long as you version control only source codes it won't be a big problem, but when you start managing data that is associated with the sources, it becomes a problem. What if we want to include an example Maya project?

- More complicated than subversion


These are what I'm seeing now. There will be lots of other pros and cons.

Thursday, June 19, 2008

Testing pyshake

It's working O.K. (though it displays an error due to command window's bug). It's good to know that NRiNode and NRiPlug are wrapped, including static methods.
NRiSys::error() is not wrapped (probably because it has variable number of args), so I cannot print messages on the history area. Making a print function by myself will be much easier than modifying py++ or gccxml. I'll make it when I need it.

Tuesday, June 17, 2008

Various ways to integrate Python and C (C++)

This is originally what I wrote in a mail I sent to a friend of mine. I modified it a little bit to post it here.

There are many tools for that.

- boost::python
- swig
- Py++
- pyrex/cython
- ctypes
- pyInline, scipy::weave, etc

boost::python, swig, pyrex, Py++ can be used for extending, which means writing a Python extension module (.pyd file on windows and .so file on Linux and Mac) in C (C++). That lets a Python script access to functions written in C (C++) , passing arguments to c and getting results from C (C++).

For embedding, you can use boost::python and pyrex/cython. boost::python will be better because pyrex/cython is mainly for extending. Embedding means starting a Python interpreter in your C (C++) program and running Python scripts inside the program. Extending is said to be better than embedding but I don't really care when embedding seems to be better for my specific problem.

boost::python is easy to use for a C++ programmer but it needs some knowledge of C++ template, and it's a little bit troublesome until you get the right compiler options. It is good if you don't have any code yet and you start writing a Python extension from a scratch. boost::python is sometimes good because all you need to write is c++ code and there's no additional script, interface definition file, etc.

If you already have a big amount of C++ code (usually library) and you want to make a Python wrapper for the library, swig is the one to choose. It is what Autodesk has used to make Maya's Python API wrapper. What I think is not good about swig is that essentially it cannot make an extension module which contains a Python class. It can only create functions in an extension. It still can generate c++ wrapper but to wrap a c++ class, it creates an extension that contains methods (i.e. functions), and it creates a module written in Python that has a class which methods are redirected to the functions in the extension module. So two files (.pyd and .py) are generated to wrap a single class. Still it's a defacto standard for wrapping a C (C++) library in my opinion.
To wrap it you need to write an interface definition file. For a very simple program, C (C++) header file can be used as an interface definition file.

Py++ is another option to wrap C (C++) library. it generates C++ source code which uses boost::python, and when you compile it, you'll get a Python extension that wraps the C (C++) library. It depends on several libraries . I don't recommend it for beginners since it takes some work to get it working. I haven't used it a lot.

If you already have Python scripts and want to make them faster withough rewriting it in C (C++), pyrex/cython is good for it. It needs some experience to use it (two/three weeks?) but if you get used to it you can convert a Python module to an extension module quickly, so in the long run it may worth the effort. Apart from that, pyrex/cython is Pythonic! I would use cython because it lets me use list comprehension (I love it!) but first I would compare them some more to see if cython has any big downside.

ctypes is part of Python standard library. You can use it from Python2.5. It calls shared libraries in a platform dependent way. I'd rather not use it because it's platform dependent. It's good if some functionality is given as .dll or .so, and you want to use it from a Python script.

pyInline, scipy::weave, etc is somewhat quite different. You can embed c code inside a Python script. I wrote an article about it on my blog.
http://koichitamura.blogspot.com/2008/05/embedding-c-code-in-python.html
If you are mainly making something in Python and want to convert just a little piece of the script in c for performance reason, I think it's a good idea to use it. I think It's good for Python plug-ins (though I haven't done it myself). Some of them are unix only.

July 23 added.
skuda gave me an information on SIP.
http://www.riverbankcomputing.co.uk/news
http://www.controlvideo.de/sip/out/index.html
Looks it covers what swig lacks.

Monty Hall problem

From Wikipedia,

"Suppose you're on a game show, and you're given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who knows what's behind the doors, opens another door, say No. 3, which has a goat. He then says to you, "Do you want to pick door No. 2?" Is it to your advantage to switch your choice?
"




The answer is.
you can double the chance by switching your choice to No.2.

Saturday, June 7, 2008

Shake Command Window Doc

See also, So how do I use Shake Command Window?

Shake command window (SCW) is a Maya script editor like command window with which you can write and execute a script interactively. It also lets you write a plug-in code in C++ and SCW does everything (save the source to file, compile/link, load, execute, unload, delete files) for you. Shake's console message is redirected to the UI.


Running three types of codes (pyshake, Shake script, gcc)



Compatibility

Currently it only works on Mac OSX 10.5. (Maybe it works on 10.3 and 10.4 if you install the latest wxPython and Python2.5 if they are not installed yet, but I don't have test environment so I cannot say for sure)

Download
Link
Oct. 29th, added.
Please see also this comment for release 0.94


Release 0.942 (bug fix version of 0.94)
Release 0.94
Release 0.93
Release 0.92
Release 0.91
Release 0.9
Release 0.8

When you load the tool, you'll see an option in the File menu. If you select it, SCW's main window will pop up.
















The upper area of the UI is history, where every Shake console message, and SCW's messages are displayed here. Lower area has four tabs. script tab is where you can write Shake script, and gcc tab is where you can write C++ plug-in code. pyshake tab is where you can write Python script (new in version 0.9). It is only available when pyshake is installed on your machine. settings tab is for tool settings. To execute Shake script, Python script, or C++ plug-in code, simply write the script/code and press a button in the Other tab in Shake.








When the script tab is open, the script written in the tab gets executed, and when gcc tab is open, the code gets executed, so does Python script.
settings tab has many buttons and text controls layouted nicely.












erase temporary files after using
If not selected, no temporary files are deleted after execution.
add script It affects how to tread a Shake script. If it is selected, it treats the script like you select File-Add Script... in the Shake menu and load the script. If it is not selected, it is directly passed to Shake's compiler (NRiCmplr). In most cases, leaving it on will do but if you need to turn it off to use extern declaration
show command It just copies a script/code to be executed to the history.
makefile Show Makefile. SCW uses the makefile to compile the plug-in code. Editable
show shell log Standard input/error message when compiling the code using the Makefile. Read only
unload after exec Unload the plug-in from Shake's process after execution.
header include/footer includeThese texts are added to the plug-in code. Editable.
Text area 1 (defaluts to "shakeCommandWinEntryFunc")
When loading the plug-in, SCW tries to search this symbol and execute it
Text area 2 (defaluts to 12) Font size
clear history
Clear the history area
save settings Save all these settings to file. Next time you open SCW, the settings will be loaded automatically

Currently it only works on Mac. I don't have Linux Shake so I can't make it for it but I tried not to use Mac specific things so it won't be difficult to port it.

Please leave a comment here if you like it (or not).



screen capture (using pyshake)

Friday, June 6, 2008

Added background gcc compile functionality to Shake Command Win

O.K. I could add an additional feature to Shake Command Window.
This time you can write a plug-in source code directly in the text field and this tool does the following things for you.
1) compile it using gcc
2) load it in shake
3) execute it
4) (optionally) unload it from shake
5) (optionally) delete files used for the execution














These are settings tab and several windows. Texts in the header include and footer include windows are just added to the c++ code in the gcc tab so that you don' t have to write these #include and extern stuff every time. shellLog is a log window when the tool compiles your code, and make is the makefile to compile. These texts are editable except shellLog. There are some more settings. I'll summerize it lator.

Wednesday, June 4, 2008

Shake Command Window

Shake is a good script based application but somehow I couldn't find a script window. So I made it. It's just like Maya's script editor.















Upper text area is the history: every output to the shake console is redirected to this area, and below one is where you can write a shake script.
























I'm planning to add another feature with which you can write a Shake plug-in c++ source code and it is compiled with gcc, then loaded immediately (borrowing the idea from PyInline, scipy.weave, etc ;).
I don't know if I can make what I am thinking now due to Shake's restriction but it'll be fun if I can.