Binary Hacks

There are not many good books written in Japanese especially around programming (most of them are entry level) but this is one of the rare exceptions.



















You'll see how good it is by taking a glance at its contents.
There's no equivalent book in English but the closest one will be linkers and loaders with the flavor of cookbook, though there are wide variety of hacks, e.g. hack#37 is 4 ways to implement singleton in c++, hack#68 demangling c++ symbol names at run time.

Another exception I know is Henrik Wann Jensen's photon mapping book. The translator of the book himself is a computer graphics researcher and he added lots of lots of footnotes that help the reader understand it!

How to access to Maya python objects from a plug-in written in C++.

This is a modified copy of my old document (you can still see the original somewhere on the net).
When I wrote this, Maya8.5 was latest, and it is still my latest unfortunately, so in this doc Maya8.5 is used. If you use Maya2008, you need to use Python2.5 (e.g. Python25.dll). And I was using windows at that time. I will add to this post how to do it on Linux and Mac soon. (Encouraging me to do so by posting a comment will become a good motivation :)

------------------------------------------------------------------------------------

Maya8.5 comes with Python. It is really not a must to know how to access to it from a plug-in written in C++ since you can make a plug-in in Python, but for some reason e.g. performance, you may want to know the way to access to it directly from your Maya .mll/.so/.bundle plug-in.

Python itself is a shared library which you can find at Maya8.5/bin/python24.dll (on windows) so if Maya can access to Python, our plug-ins can get access to it.

First you need to include Python.h before every includes like any other python programs. The version of Python must be 2.4 since Python inside Maya is version 2.4.3. Other version of Python e.g. 2.5 uses different shared library e.g. python25.dll and you cannot use the same Python as the one Maya uses.

Maya8.5 has already initialized Python interpreter so you don't need to (and you shouldn't) call Py_Initialize(), but in the doIt() (in case of a command plugin) of your plug-in, you need to call
PyGILState_Ensure()
Python interpreter is not thread safe and this function call locks the interpreter properly so that only the thread that is running your plug-in can access to Python. In the test plug-in code show below, I used PyRun_SimpleString() to access to Python. See this post to know how to use Python C/API. When you leave doIt(), you need to call
PyGILState_Release()
It is needed even if you don't use Python thread modules. Forgetting this freezes or crashes Maya easily.

#include <Python.h>
#include <maya/MGlobal.h>
#include <maya/MSimple.h>
DeclareSimpleCommand( maya85py, "", "8.5");

MStatus maya85py::doIt( const MArgList& args )
{
PyGILState_STATE state = PyGILState_Ensure();
PyRun_SimpleString("tamtam = 1");
PyGILState_Release(state);
return MS::kSuccess;
}

Here's the result of above test plug-in.

tamtam
# Error: name 'tamtam' is not defined
# Traceback (most recent call last):
#   File "<maya console>", line 1, in ?
# NameError: name 'tamtam' is not defined #

import maya
maya.mel.eval("maya85py")

tamtam
# Result: 1 #

Debug build:
On windows, the compiler tries to link python24.lib or python24_d.lib automatically (on Linux and Mac you need to link the appropriate one manually). python24_d.lib is a debug version and you need to build python itself to get it. I surfed the web but couldn't find it, and tried to build python but full build was a little bit bothering (install tcl/tk, brabrabra...), so I just modified pyconfig.h in my Python24/include directory. It's all what you need to debug build your plug-in (You don't need this if you don't use the debugger). Here I just used Python header files and import library that comes with standard Python distribution, since Maya doesn't come with them. Recently I heard one of my friend made an import library using dumpbin command. It'll be better if you have time to do it (still not safe though).

//#                pragma comment(lib,"python24_d.lib")
#                pragma comment(lib,"python24.lib")
#            else
#                pragma comment(lib,"python24.lib")

Finally, in an actual plug-in PyRun_SimpleString("tamtam = 1"); can be as complicated as you want, missing PyGILState_Release() execution by some exception is quite dangerous. I would use try-catch clause.

Houdini HDK notes

These are some notes when I was testing Houdini HDK last year.

Houdini notes on my hidden blog

Houdini topics are mostly written in English, the rest of blog entries are written in Japanese though. It is not so well organized and it's just a day by day notes so older posts (and even newer posts may) have wrong comments due to my misunderstanding. That's why I was declined to show it but it may still help somebody who has started learning HDK/HOM.

Houdini HDK class hierarchy is just part of them.


By the way, "orz" is a Japanese emoticon so when you see orz in my blog you can expect my pose like this.

Time for job hunting!

Finally I started looking for a job.

If you have an interesting information, please let me know.
(job offer, advices, etc.)

And in case you've seen my resume at the office, please recommend me to your boss!

Stackless Python serialize test

You can pickle the Python execution state with Stackless Python.

import stackless
def a():
    c = 1
    while(True):
        print c
        c += 1
        stackless.schedule()

tskOrg = stackless.tasklet(a)()

print "----first run"
stackless.schedule()
stackless.schedule()
stackless.schedule()

print "----now serialize the task"
import pickle
p = pickle.dumps(tskOrg)

print "----second run"
stackless.schedule()
stackless.schedule()
stackless.schedule()

print "----remove the task from the scheduler and insert the pickled one"
tskOrg.remove()
tskCopy = pickle.loads(p)
tskCopy.insert()

print "----run the pickled task"
stackless.schedule()
stackless.schedule()
stackless.schedule()

----first run
1
2
3
----now serialize the task
----second run
4
5
6
----remove the task from the scheduler and insert the pickled one
----run the pickled task
4
5
6


Note:
Stackless Python is not a standard Python.
It's a variant of Python that doesn't use C stack for Python function calls.
You need to download it from http://www.stackless.com/
and customize Python installed on your machine.

Photos taken with my mobile phone

Tokyo



















Tokyo (cheating)



















Tokyo


Nagoya (through the Shinkansen window)














Osaka














Osaka














Tokyo

















Shikine




















and something different?

-undefined dynamic_lookup (Mac)

I was a little bit confused about MacPython's extension dependency.

Extensions that are part of the standard library are in
/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-dynload (On my machine)
Basically those extensions seemed to be dependent on only libgcc_s.1.dylib and libSystem.B.dylib, not Python at all.

$ otool -L array.so
array.so:
/usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 111.0.0)

I was wondering why it was not dependent on Python, while in the extension Python C/API funcs were used.
Python C/API functions are in
/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/config/libpython2.5.a (again on my machine)
which is actually a shared library (dylib) although its file extension is .a, and is a symbolic link to ../../../Python. (P is capital)

$ cd /System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/config
$ nm libpython2.5.a | grep "T "
...(lots of API funcs)...

Should extensions be dependent on it?
If a symbol is not bound to a filename how can we avoid name conflict?
I tested further and found the way to know what file a functions is dependent on.

$ cat a.cpp
#include <stdio.h>
void f()
{
 printf("a\n");
}
$ cat b.cpp
#include <stdio.h>
void f()
{
 printf("b\n");
}
$ cat main.cpp
int main()
{
 void f();
 f();
 return 0;
}

$ gcc -dynamiclib -arch i386 -o a.dylib a.cpp
$ gcc -dynamiclib -arch i386 -o b.dylib b.cpp
$ gcc main.cpp -dylib a.dylib -dylib b.dylib -lstdc++
$ nm -mg a.out
0000200c (__DATA,__data) external _NXArgc
00002008 (__DATA,__data) external _NXArgv
(undefined [lazy bound]) external __Z1fv (from a)
...

So each function does have a dependency to a certain file, yeah of course
I used the command to array.so,

$ cd /System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-dynload
$ nm -mg array.so
(undefined [lazy bound]) external _PyErr_BadArgument (dynamically looked up)
(undefined [lazy bound]) external _PyErr_Clear (dynamically looked up)

What's (dynamically looked up)?
The answer was a linker option -undefined dynamic_lookup first supported in OSX10.3.
http://developer.apple.com/documentation/DeveloperTools/Conceptual/MachORuntime/Reference/reference.html

$cat fakeb.cpp
$ gcc -dynamiclib -arch i386 -o a.dylib fakeb.cpp
$ gcc -dynamiclib -arch i386 -o b.dylib fakeb.cpp
$ gcc main.cpp -dylib a.dylib -dylib b.dylib -lstdc++
Undefined symbols:
"f()", referenced from:
_main in cca9JLe0.o
ld: symbol(s) not found
collect2: ld returned 1 exit status
$ gcc -dynamiclib -arch i386 -o a.dylib fakeb.cpp
$ gcc -dynamiclib -arch i386 -o b.dylib fakeb.cpp
$ gcc main.cpp -dylib a.dylib -dylib b.dylib -lstdc++ -undefined dynamic_lookup
$ nm -mg a.out
0000200c (__DATA,__data) external _NXArgc
00002008 (__DATA,__data) external _NXArgv
 (undefined [lazy bound]) external __Z1fv (dynamically looked up)
...
$ gcc -dynamiclib -arch i386 -o a.dylib a.cpp
$ gcc -dynamiclib -arch i386 -o b.dylib b.cpp
$ ./a.out
a
$ gcc -dynamiclib -arch i386 -o a.dylib fakeb.cpp 
$ ./a.out
b

Phew...

Pyro

I'm now testing Pyro (Python Remote Objects).
Its concept is "transparent" networking, while twisted perspective broker is called "translucent".
Pyro will be good where networking overhead is not so painful.


Besides its technical features, it is
- written 100% in Python
- matured enough
- well documented, lots of useful examples (this is important!)
- MIT licensed
See other features Pyro has.