Christmas

Merry Christmas.



Every year the first floor of the building I live in becomes like this. It's a building owner's preference ;)

OpenReality SDK first look

It is a large SDK so It'll need some time to get used to it but it seems to be a straight forward implementation and doesn't seem weird (like Houdini SDK :p), so it won't be difficult.

I still don't know several most basic stuff such as what a plug owner means, but it'll be enough for now until I can use motionbuilder itself (now I use only PLE). I'll test some on it, look at and run the sample programs which come with it then.

Class hierarchy

FBWrapperHolder
  FBPlug
      FBComponent
          FBBox etc. A lot other
      FBProperty
          FBPropertyBasicList : abstruct
              FBPropertyBaseList<> : template, abstruct
                  FBPropertyListFCurveKey, etc. A lot other
              FBPropertyListComponentBase
                  FBPropertyListBox, etc. A lot other
                  FBPropertyListComponent
                      FBPropertyListCamera, etc. A lot other
          FBPropertyBase : template
              FBPropertyBaseComponent
              FBPropertyBaseEnum
          Other (FBPropertyStringList, FBPropertyAnimatable, FBPropertyEvent)


-----------------------------------------------------------
[FBWrapperHolder]
Probably Python wrapper related? Let's ignore this for now.


-----------------------------------------------------------
[FBPlug]
Connect*()
Disconnect*()
  Standard plug implementation

HFBPlug GetOwner()
  Nested hierarchy. Looks a plug can be owned by another. Not the same as parent.
  Maybe used for a plug(component) to own a plug(propery)? as property/component has
  no methods to get "who owns it"/"owned by whom" info.

int GetOwnerCount()
  Implies multiple owners. A plug can be shared by multiple plugs.

virtual char* ClassName()
virtual int GetTypeId()
  These two implies property is typed (different from Property type)

bool MoveSrcAt(int pIndex, int pAtIndex): Move source connection at pIndex to pAtIndex.
  Multiple sources allowed and its order is important.


-----------------------------------------------------------
[FBComponent]
  Represent objects (camera, curve, constraint...). Property holder.
  Components with callback funcs have a FBPropertyEvent type property,
  which has add()/remove() methods that accept a pointer to a user defined callback function.
  e.g. FBLayout::OnIdle/OnResize etc.

virtual int PropertyAdd(FBProperty *Property)
void PropertyRemove(FBProperty *Property)
  Behaves as a property holder

FBObjectFlag GetObjectFlags()
  selectable, savable, deletable, etc.


void SetObjectStatus(FBObjectStatus pStatus, bool pValue)
bool GetObjectStatus(FBObjectStatus pStatus)
  creating, deleting, storing, etc.
  Used within a callback? Why is there set? Can the SDK user set it?

FBPropertyManager PropertyList   Read Only Property: Manages all of the properties for the component.

FBPropertyListComponent Component   List: List of components.
  What's this? Children?

FBPropertyListComponent Parents   List: Parents.
  Similar hierarchy to Property? Why no get/setChildren() stuff?

FBPropertyBool Selected   Read Write Property: Selected property.

FBPropertyString Name   Read Write Property: Unique name of object.
  Component is Named, Component has a concept of namespace.


-----------------------------------------------------------
[FBProperty] holder for function callbacks into the internals of the application
  Holder for func callbacks? Not just a data representation? What are the correspondent methods?

char* GetName()
void SetName(char *pName)
  Has name

virtual FBPropertyType GetPropertyType()
char* GetPropertyTypeName()
  int, float, ... (different from Plug type)

int AsInt()
bool SetInt(int pInt)
virtual char* AsString()
virtual bool SetString(char *pString)
  Value get/set.
  Why can set*() fail?
  Why string methods are virtual? expression related?

FBPropertyFlag GetPropertyFlags()
  notset, hide, forcestatic, disable, animated, notsavable, readonly, notuserdletable

void SetMinMax(double pMin, double pMax)
  Similar to Maya

void* GetParent()
  Not the same as owner. Why void*?

virtual void SetData(void *pData)
virtual void GetData (void *pData, int pSize, FBEvaluateInfo *pEvalInfo=NULL) const
  Looks for more complex type of data


-----------------------------------------------------------
[FBPropertyBasicList]
  Abstruct class. Property container.
  Type independent methods are here.

virtual int GetCount ()=0   Get the number of properties in the list.
virtual void RemoveAt (int pIndex)=0   Remove property at pIndex.
  Methods to act as a container.


-----------------------------------------------------------
[FBPropertyBaseList<>]
  Abstruct template class. Type independent methods are in FBPropertyBasicList.
  Container of non-component properties.
  Base class of FBPropertyListRigidBody, FBPropertyListMotionClip, etc.

virtual int Add(tType pItem)=0   Add a property to the list.
virtual tType operator[](int pIndex)=0   [] operator overload.
virtual int Find(tType pItem)   Locate a property in the list.
virtual int Remove(tType pItem)   Remove pItem from the list.
virtual tType GetAt(int pIndex)   Get a property at pIndex.
  Methods to act as a container.


-----------------------------------------------------------
[FBPropertyListComponentBase]
  Abstruct *non-template* class. Type independent methods are in FBPropertyBasicList.
  Container of component properties (i.e. FBComponent).

virtual int Add(FBComponent *pItem)
virtual void RemoveAt(int pIndex)
virtual FBComponent* GetAt(int pIndex)=0
FBComponent* operator[](int pIndex)
virtual int GetCount()
virtual int Find(FBComponent *pItem)
virtual int Remove(FBComponent *pItem)
  Methods to act as a container.
  It doesn't inherit FBPropertyBaseList but all the methods are implemented here with it's template type tType=FBComponent.
  Bbefore FBPropertyBaseList was used for components as well. It is epareted for some reason.


-----------------------------------------------------------
[FBPropertyListComponent]
  Implements only GetAt() method.

virtual FBComponent* GetAt(int pIndex)


-----------------------------------------------------------
[FBPropertyBase, FBPropertyBaseComponent, FBPropertyBaseEnum]
  No idea yet.

Using doctest module for if __name__ == '__main__' test

Many people including me write a test code in __name__ == '__main__' like

def somefunction(intvalue):
 return intvalue * 2

if __name__ == '__main__':
 #Tests whatever.
 if somefunction(2) != 4:
     print "NG"

Previsously I did something like this.

if __name__ == '__main__':

 def test(cmd):
     print cmd,
     exec cmd

 ph = PreferenceHolder(pref1=1.0, pref2=100, pref3='tamtam')
 prefs = ph.getPreferences().items()
 test("print prefs == [('pref1', 1.0), ('pref2', 100), ('pref3', 'tamtam')]")
 test("print ph.pref1 == 1.0")

I wanted a nicer way so looked around documents for Python unit test modules. Python has two modules for unit testing, unittest and doctest. unittest is kind of overkill for a test in __name__ == '__main__' so I looked at doctest, which is better because it's purpose is to offer tools to write "readable" tests. I serched for a function to let me write a test code like

if __name__ == '__main__':
 teststr = """
 >>> ph = PreferenceHolder(pref1=1.0, pref2=100, pref3='tamtam')
 >>> ph.getPreferences().items()
 [('pref1', 1.0), ('pref2', 100), ('pref3', 'tamtam')]
 >>> ph.pref1
 1.0
 """
 import doctest
 doctest.teststring(teststr) #Note: it doesn't exist

Unfortunately I found I cannot do this. I could find a function that takes a filename to get a docstring but couldn't find one that takes a Python string directly like above code. I think this function is not provided because doctest module is designed to write the whole test in docstrings (It is provided as a class but not as a userfriendly function. Readability is important because everybody looks at this test code as an example). I was just looking for an easy way to give the users usage examples (primary interest) and have it work as a test (secondary), so I didn't want to write the whole tests in each doc string. I found an alternative function run_docstring_examples() which tests only a docstring for a specified object given by one of its arguments. The document says

There’s also a function to run the doctests associated with a single object. This function is provided for backward compatibility. There are no plans to deprecate it, but it’s rarely useful

but it's nearly perfect for my purpose. So now, I can write the test like this.

if __name__ == '__main__':
 def test():
     """
     >>> ph = PreferenceHolder(pref1=1.0, pref2=100, pref3='tamtam')
     >>> ph.getPreferences().items()
     [('pref1', 1.0), ('pref2', 100), ('pref3', 'tamtam')]
     >>> ph.pref1
     1.0
     """
 import doctest
 doctest.run_docstring_examples(test, globals(), False, __name__)

Because it's a doctest, it prints nothing if all the tests finish successfully (unless you set True to the third argument of run_docstring_examples). If it fails, it prints an error message

$ python preferenceholder.py
**********************************************************************
File "preferenceholder.py", line 94, in __main__
Failed example:
 ph.pref1
Expected:
 0.0
Got:
 1.0
$

Nice isn't it?
Please see the previous blog entry for the actual usage.

By the way, one common pitfall of using doctest is you don't update comments in a docstring when you modify its implementation. I've seen lots of wrong comments in existing dectests. Try to write only comments about specs and not about implementation details unless it is necessary.

Preference class (beta)

===BETA VERSION===

I made a generic preference (or config) data class. It is designed to hold system wide preference data. It has a flag "isPrefValueChanged" flag which is set when a preference is added/deleted/modified, and reset when it is saved/loaded. All the property data must be picklable.



You can set/get/delete preference attributes directly.
Direct data setting to a property will set isPrefValueChanged flag.

In addition there are four method.

isPrefValueChanged(self):
Return true if preference has been changed or not saved.

savePreference(self, filename):
Save preferences to a file. isPrefValueChanged flag will be reset.

loadPreference(self, filename):
Load preferences from a file. isPrefValueChanged flag will be reset.

getPreferences(self):
Get preference data in a {preference name : value} dictionary.
It returns a deep copy of the preference data. Changing a value will not modify
the preference value held in this object.

import copy
import cPickle as pickle

class PreferenceHolder(object):
  """System wide preference holder class.
  """

  def __init__(self, *args, **kwargs):
      self.__dict__['_pa'] = pa = type("", (), {})() #Private attributes
      pa.hcls = hcls = type("", (), {})
      pa.hobj = hcls()
      pa.isPrefValueChanged = True
      pa.filename = ''
      for name, value in kwargs.items():
          setattr(self, name, value)

  def __setattr__(self, name, value):
      pa = self._pa
      if not hasattr(pa.hcls, name):
          privatename = '__' + name
          def getvalue(self):
              return getattr(self, privatename)
          def setvalue(self, value):
              setattr(self, privatename, value)
          def delvalue(self):
              delattr(self, privatename)
          p = property(getvalue, setvalue, delvalue)
          setattr(pa.hcls, name, p)

      setattr(pa.hobj, name, value)
      pa.isPrefValueChanged = True

  def __getattr__(self, name):
      return getattr(self._pa.hobj, name)

  def __delattr__(self, name):
      pa = self._pa
      delattr(pa.hobj, name)
      delattr(pa.hcls, name)
      pa.isPrefValueChanged = True

  def isPrefValueChanged(self):
      """Return true if preference has been changed or not saved.
      """
      return self._pa.isPrefValueChanged

  def getPreferences(self):
      """Get preference data in a {preference name : value} dictionary.
      It returns a deep copy of the preference data. Changing a value will not modify
      the preference value held in this object.
      """
      result = {}
      for name in [n for n in dir(self._pa.hcls) if n[0] != '_']:
          result[name] = getattr(self, copy.deepcopy(name))
      return result

  def savePreference(self, filename):
      """Save preferences to a file. isPrefValueChanged flag will be reset.
      """
      pa = self._pa
      if filename == pa.filename and not self.isPrefValueChanged():
          return

      try:
          fileobject = open(filename, 'w')
          pickle.dump(self.getPreferences(), fileobject)
          pa.isPrefValueChanged = False
          pa.filename = filename
      finally:
          fileobject.close()

  def loadPreference(self, filename):
      """Load preferences from a file. isPrefValueChanged flag will be reset.
      """
      pa = self._pa
      try:
          fileobject = open(filename, 'U')

          for name in self.getPreferences().keys():
              delattr(self, name)

          savedprefholder = pickle.load(fileobject)
          for name, value in savedprefholder.items():
              setattr(self, name, value)
          pa.isPrefValueChanged = False
          pa.filename = filename
      finally:
          fileobject.close()

if __name__ == '__main__':
  def test():
      """
      >>> ph = PreferenceHolder(pref1=1.0, pref2=100, pref3='tamtam')
      >>> ph.getPreferences().items()
      [('pref1', 1.0), ('pref2', 100), ('pref3', 'tamtam')]
      >>> ph.pref1
      1.0
      >>> ph._pa.isPrefValueChanged = False #Do not do it!
      >>> ph.pref4 = 'new pref'
      >>> ph.pref4
      'new pref'
      >>> ph.isPrefValueChanged()
      True
      >>> ph._pa.isPrefValueChanged = False #Do not do it!
      >>> del ph.pref1
      >>> ph.pref1
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
        File "preferenceholder.py", line 34, in __getattr__
          return getattr(self._pa.hobj, name)
      AttributeError: '' object has no attribute 'pref1'
      >>> ph.isPrefValueChanged()
      True
      >>> #Tests that create files on disk
      ...
      >>> #ph.pref1 = 1.0
      >>> #ph.pref1
      1.0
      >>> #ph.savePreference('preffile')
      >>> #ph.isPrefValueChanged()
      False
      >>> #ph.pref1
      1.0
      >>> #ph.isPrefValueChanged()
      False
      >>> #ph.loadPreference('preffile')
      >>> #ph.isPrefValueChanged()
      False
      >>> #ph.pref1
      1.0
  """

  import doctest
  doctest.run_docstring_examples(test, globals(), False, __name__)

Started looking at Open Reality SDK

Getting a couple of documents MotionBuilder Open Reality SDK Help and MotionBuilder Python Scripting Help from an Autodesk page, I started looking at Open Reality SDK. I've just started learning it so I don't have a lot to tell except that it lets us to execute Python scripts over tcp port 4242. I'll write a review when I've read them more.

A very good site and the author's blog Stumbling Toward ‘Awesomeness’.

Japanese has no subject

I found a book about Japanese grammar titled "Japanese needs no subject". I haven't read it but I agree with what the title says, since when I studied Japanese grammer at school I couldn't understand why we had to distinguish subject from others. Many Japanese grammar books say that in spoken Japanese subjects are often omitted. But in my opinion Japanese has no concept of "subject". To explain that I'll modify English slightly (modifying English again, sorry ;). The new English grammar has no subject, but it has additional proposition "sub" which is followed by a noun indicating the subject of the sentence.

Went to school sub me in the morning.
Should see a dentist in the afternoon sub you.
Is hungry sub her.

They are quite weird but all makes perfect sense. These sentences written in new English have all semantic subject but there's no special grammatical subject. And just like other propositional phrases, it is not added to a sentence if you don't need to clarify it. I think it simulates Japanese sense of subject quite correctly.

Ideal version control in a production

This is an entry that follows this entry and related to this.
I've been thinking what an ideal software version control in a production looks like.

In my opinion an ideal way of version control should satisfy all of these.

a) A software bug fix must be reflected to the work environment automatically.
b) An interface modification of a software must be version controlled and artists must be able to chose which version of the software to use for each project.
c) An artist must be able to open a scene data as it was when it was modified last time. It means softwares (plug-ins etc.) must be the same as what it was including bugs.

I think a) and b) can be solved by defining naming rules carefully referencing this idea. For c) we will need to use a version control system. It'll be good to version control all the softwares in a file server and offer artists, access it through a symbolic link, and offer some utility tool with GUI for artists to retrieve a snapshot of the softwares and change the symbolic link.

Of course I know there are lots of reasons that cannot be simply done, historical reason, human resource, data size, data convert, artist preference, tools/scene data that depend on existing systems, network requirement, and sometimes a boundary between data and tools are not clear ... but it's still important to think about a symple ideal case to get a picture of a good production infrastructure.

Python3.0 final released!

http://www.python.org/download/releases/3.0/

Pythonic English

I'm learning English for decades and still haven't mastered it yet. What confuses me most are articles (a, an, the), tense, and pronunciation like "l", "r" difference. Instead of learning English, let's modify English referencing The Zen of Python.

Simple is better than complex.
In Japanese, there's no future tense nor perfect form. When we talk about something in the future we use present tense, like "I do homework soon" and when we need to mean that it's not present, we use a word indicating time, e.g. "We go shopping next Sunday". The fact that there's a language without them means it's not necessary. Let's remove them from English. How about past tense? Do we need them? "I went to school yesterday" can be "I go to school yesterday". No problem, let's remove it as well.

Explicit is better than implicit
Articles are nightmares to me. What people knows from articles are,

the: if something is specific or not.
a, an: if the number is one or not.

Let's make them explicit by replacing "a" and "an" to "one", and "the" to "this" or "that". "I have an apple" can be "I have one apple", "The earth" can be "this earth" or "that earth". Good.

If the implementation is hard to explain, it's a bad idea.
Native people often have difficulty in explaining the difference between "l", "r" pronunciations. Let's remove "r" sound from English.

Isn't it a good idea? Let's make it world common. Now is better than never.

Version control/Bug tracking at Rising Sun Pictures

You may have seen it because it's a famous slide but I'll link to it anyway.
"Adelaide University takes on the Earth" blog entry

Each movie might be running a different version of each piece of software

How they do it? I'm interested in its detail.
The only way I know is using an interface name (soname in this post).
but in a production it is not always good to fix a bug whenever you have find it, since what is the most important is sometimes to open a file later and see exactly the same scene as the one when it was created including bug behaviors, but it is troublesome to have artists update a software manually when you find a bug in it. It's a contradiction. Maybe a 3d software uses the latest software (plug-in etc.) automatically, but once a scene ihas been created, it keeps using the same version of the softwares? but how?

PyPy

PyPy is a project to implement Python(CPython compatible) in Python(RPython). What interests a PyPy end user like me is that it's good for converting Python code to various languages. According to PyPy document, it currently translates a Python script into C(full featured), llvm(full), .Net intermediate language(full), Java(not full but getting close), and JavaScript. Due to Python's super flexible features, not all the Python scripts can be translated. They introduced restrictions to CPython specification and defined RPython, a subset of CPython (well, they don't say they have clearly *defined* RPython spec since those restrictions introduced to RPython are evolving). While They make RPython, they have removed every Python features that makes scripts difficult to be statistically compiled. You can read details about RPython here, for example, all module global variables are constant.
PyPy has lots of documents but some of them are not easy to read because it deals with PyPy internal architecture as well as PyPy usage, several comments are only for implementing Python on Python, while others are more generic and for every RPython scripts, and there are lots of levels of things, CPython v.s. RPython, application level objects v.s. interpreter level objects ... so I will suggest in what order you read PyPy documents.
The order I recommend is, you first read getting-started just briefly ignoring every stuff you cannot understand, then architecture then coding-guide then translation. Do not read documents regarding new features such as What PyPy can do for your objects first, though it's the first thing you see in the PyPy documentation list.


Here I will make some more comments on different PyPy topics randomly.

One good thing about outputting llvm code is that it can take advantage of llvm's powerful optimization, both llvm bit code level and processor dependent. From the llvm point of view, PyPy is a llvm Python front end(more precisely RPython frontend). I haven't tested a lot but it's definitely a cool topic.

PyPy translates RPython into other languages from a Python object, not the script directly. It means what you pass to the translator is your function object, not a string nor a file path that contains RPython script.

They have added some to PyPy that doesn't exist in CPython such as optional stackless features. Lazy computed objects is another.

Talking about PyPy implementation, "object space" is interesting. PyPy uses a "standard object space" to execute Python scripts. When it generates a control flow graph (which is used when it translates RPython into another language), it just replaces the standard object space with "flow object space", so for the rest of PyPy, it looks as if the script is executed but the result is a control flow graph, not script execution. I'm not interested in the details but understanding the concept made it easier to read the document.

I just found a tiny bug.

def f():
  for a in [1, 2, 3]:
          print a,
>>> f()
1 2 3

f() prints nothing after it is c-compiled. It works as expected if I remove comma after "print a".

pypy first test

I installed pypy and tested just a little bit.
my RPython code is

def myfunc():
    for i in range(10000):
        print "tamtam", i

and it took 0.06 sec to print 10000 lines, not bad.

$ python bin/translatorshell.py
>>> import translator.test.mytest as mytest
>>> t = Translation(mytest.myfunc)
>>> t.rtype()
>>> f = t.compile_c()
>>> print time.time();f();print time.time()
1227481538.02
tamtam 0
tamtam 1
...
tamtam 9999
1227481538.08

I also wanted to use pypy as a llvm python frontend. The sample code worked but my function didn't. The pypy getting-started doc says I should use llvm1.9 but I installed the latest (ver 2.4). It is said that llvm doesn't pay big attention to backward compatibility so maybe it's due to llvm version mismatch, or my wrong pypy/llvm usage.

FBX SDK

I've been having fun with FBX SDK. FBX SDK is seemingly well documented but what you really need are not so much. I'll write some of them I found briefly (I could write them much more in detail but not sure if the licence grants it). I didn't write basic stuff which you can find easily, I rather focused on the data structure. The FBX SDK version I used is 6.1.0


- KFbxLink is obsolete. If you find it in a document, the document is old.

Now KFbxLink is a typedef of KFbxCluster. But in the KFbxCluster document and method name, a "link" (typically seems to) means a node which has a KFbxSkeletion object as its attribute.

- A patch has no information on deformation (skeleton, etc.) by any means.

- A skeleton has no information on geometry (patch, etc.) by any means.

- A cluster connects a patch and a skeleton. It has control point weights, as well as its binding information such as transform matrix of a patch and transform matrix of a skeleton. which are all kept in the world coordinate.

You can register
a skin to a Node that has KFbxGeometry with AddDeformer(), and
a cluster to a skin with AddCluster(), and
a node that has KFbxSkeleton to a cluster with SetLink()

- You can get the transform matrix of the patch with GetTransformMatrix() which is used to bind the patch and the skeleton (not animation)

- You can get the transform matrix of the skeleton with GetTransformLinkMatrix() which is used to bind the patch and the skeleton (not animation)

- A cluster doesn't have a skeleton animation data.

- A node itself can have its animation data, not somewhere else.

More precisely, A node can have multiple KFbxTakeNode objects and each KFbxTakeNode object can contain a set of animation curves which are needed for the animation of the node corresponding to the take.

- Every KFbxTakeNode has its name in it. The whole animation of a take is obtained by summarizing all the information each node has in its KFbxTakeNode corresponding to the take (which is identified by the take name).

Important Notice

I'm NOT a Japanese tourist in Killer Tomatoes Eat France!

Chalana - ASF to MEL converter

http://code.google.com/p/chalana/ via http://www.mento.info/dekekincaid

What is Chalana ?

Chalana is a freeware program which you can use to convert AMC and ASF file to MEL so you can import the motion capture data into Maya.

How does it work ?

Chalana works by using the Biomichanics inc's AMC to MEL and ASF to MOV executables which you can download for free at CMU Graphics Lab motion capture database. Chalana also includes the same programs in the application directory.

With Chalana you can convert the MEL files generated by the above programs in to your Maya projects frame rate and/or grid scale.

For whom ?

Chalana can be used by Anyone who wishes to insert motion capture data to their animation. Especially for game developers and crowd animators.

Hair plug-in

This is my hair plug-in.
At first I didn't make the plug-in for hair (the third movie is the closest one to the original purpose) but called it so later since it looked like hair, when I made it Maya was 4 and there was no hair.

Why are there so many symbolic links?

Why are there so many symbolic links? Beta

This document is still beta.
If you find anything is wrong or inaccurate, please let me know.

You can see lots of symbolic links to shared libraries on Linux. Why are there so many and how it works? I couldn't find a detailed explanation on the web so I'll try to make a very easy tut here.

In short they are there for version control of the shared libraries. The key is that there are two types of library modifications, implementation modification with no interface change of a library, and interface modification. An "interface" of a function, classes, etc. is the meaning of it. If the declaration has been changed, like

old: void foo(int bar);
new: void foo(double bar);

You can say the interface of the function has been changed. And even if there's no change of the declaration, the interface may have been changed. It gets changed if the meaning of the argument or return value has been changed.

old: void foo(int bar); //bar means your age.
new: void foo(int bar); //bar means the number of chocolate bars you have eaten today.

and an "implementation" means how the function that satisfies the interface is made. You can make a sort function using bubblesort, heapsort, quicksort, or bogosort ;) all have the same interface but implementations are different. A modification to a function implementation with no interface change is often made for bug fix and performance tuning.


I will first tell you how it works and then why it works.

How it works

[Three types of names]

There are three types of names for one shared library.

(1)libfoo.so
(2)libfoo.so.1
(3)libfoo.so.1.0

Usually (1) and (2) are symbolic links. libfoo.so -> libfoo.so.1 -> libfoo.so.1.0



(1)libfoo.so
It is called "linker name". It is used when building (linking) a shared library like

gcc -lfoo -o myapp *.o

(2)libfoo.so.1
It is called "soname". th number indicates interface version. It is incremented on every interface modification of the library. A program that uses the library doesn't always assume the latest interface, since the program may only work on old interface. Remember an interface means the meaning of the function. In a shared library, its soname can be embedded (i.e. information about the interface version of the library is held in the library itself) by building (linking) like

gcc -shared -Wl,-soname,libfoo.so.1 -o libfoo.so.1.0 *.o

(the reason is described later).


(3)libfoo.so.1.0
It is called "real name". It is (usually) not a symbolic link. The first number (in this case 1) is the interface version of the library which is correspondent to the soname, and the second number (and sometimes the third) indicates the implementation version of the library, which gets incremented on every implementation modification of the library, like libfoo.so.1.1, libfoo.so.1.2 All the programs that use the library assume the latest implementation version of the library.


[When linking]
When a program (myapp) is built, like

gcc -lfoo -o myapp *.o

libfoo.so is searched and used (unless libfoo.a is found first). libfoo.so is a symbolic link: libfoo.so -> libfoo.so.1 -> libfoo.so.1.0 so what is really used is libfoo.so.1.0 The linker finds its soname libfoo.so.1, which is embedded in the libfoo.so.1.0, and put the soname to the executable (myapp) as a library that the executable is dependent on. It will sound a roundabout way but you'll see why the linker do such a thing when you've finished reading this blog entry.

[At the start up of myapp execution]
When myapp is started, libfoo.so.1 is searched because the name (soname) is recorded in myapp as a dependent library. Because libfoo.so.1 is a symbolic link to libfoo.so.1.0, libfoo.so.1.0 is loaded and used.


Why it works
Why is it good for the library's version control? I'll explain it by taking examples.


[When libfoo has a bug]
The library's author fixes the bug and release it as libfoo.so.1.1

gcc -shared -Wl,-soname,libfoo.so.1 -o libfoo.so.1.1 *.o

Because the interface is not changed, soname is not changed. Now libfoo.so.1.1 is the latest and the libfoo.so.1 's link should be updated.
old: libfoo.so.1 -> libfoo.so.1.0
new: libfoo.so.1 -> libfoo.so.1.1
This link is done by the system automatically (When you restart linux a program called ldconfig is executed, ldconfig finds wrong links and updates them). Now myapp uses libfoo.so.1.1 because libfoo.so.1 is a symbolic link to libfoo.so.1.1 In this way my app uses the latest implementation version of the library.

[When libfoo has a interface modification]

The library's author makes a new libfoo which has a different interface and release it as libfoo.so.2.0

gcc -shared -Wl,-soname,libfoo.so.2 -o libfoo.so.2.0 *.o

Because the interface has been changed, its soname should be updated (libfoo.so.2). Now a symbolic link libfoo.so.2 -> libfoo.so.2.0 is created. If he rebuilds myapp like before

gcc -lfoo -o myapp *.o

and runs it, libfoo.so.1.1 is still used, because myapp is still told to be dependent on libfoo.so.1 (because libfoo.so -> libfoo.so.1 when building) and libfoo.so.1 is still a symbolic link to libfoo.so.1.1

[When myapp's author wants to test the new library]

He wants to test the new library. He wants to make "yourapp" which uses the new libfoo. He can do so by changing the symbolic link libfoo.so

old: libfoo.so -> libfoo.so.1
new: libfoo.so -> libfoo.so.2

When he builds yourapp

gcc -lfoo -o yourapp *.o

and runs it. Now libfoo.so is a symbolic link to libfoo.so -> libfoo.so.2 -> libfoo.so.2.0, soname libfoo.so.2 is embedded in libfoo.so.2.0, and libfoo.so.2 is recorded in yourapp as a dependent library. On yourapp application startup, libfoo.so.2 -> libfoo.so.2.0 is loaded and used. myapp still uses libfoo.so.1 unless he rebuilds it after libfoo.so -> libfoo.so.2 relinking.


Conclusion
Every executable uses the latest implementation of a fixed interface which is recorded as a "soname" in the executable.
The soname which is recorded in an executable is the one that the symbolic link which name is "linker name" links to.

Hooking library calls on Mac using DYLD_INSERT_LIBRARIES

Mac offers a way to override functions in a shared library with DYLD_INSERT_LIBRARIES environment variable (which is similar to LD_PRELOAD on Linux). When you make a twin brother of a function that is defined in an existing shared library, put it in you a shared library, and you register your shared library name in DYLD_INSERT_LIBRARIES, your function is used instead of the original one. This is my simple test. Here I've replaced f() in mysharedlib.dylib with f() in openhook.dylib.

$ cat mysharedlib.h
void f();
$ cat mysharedlib.c
#include <stdio.h>
#include "mysharedlib.h"

void f()
{
 printf("hello");
}
$ cat main.c
#include <stdio.h>
#include "mysharedlib.h"

int main()
{
 f();
 return 0;
}
$ cat openhook.c 
#include <stdio.h>
#include <dlfcn.h>
#include <unistd.h>
#include "mysharedlib.h"

typedef void (*fType)();
static void (*real_f)() = NULL;

void f()
{
 if ( ! real_f)
 {
  void* handle = dlopen("mysharedlib.dylib", RTLD_NOW);
  real_f = (fType)dlsym(handle, "f");
  if ( ! real_f) printf("NG");
 }
    printf("--------zzz--------");
 real_f();
}
$ cat bat
#!/bin/bash
gcc -flat_namespace -dynamiclib -o openhook.dylib openhook.c
gcc -dynamiclib -o mysharedlib.dylib mysharedlib.c
gcc mysharedlib.dylib main.c
export DYLD_FORCE_FLAT_NAMESPACE=
export DYLD_INSERT_LIBRARIES=openhook.dylib
./a.out
$ ./bat
 --------zzz--------hello

You also need to define DYLD_FORCE_FLAT_NAMESPACE (doesn't matter what value it has). In general it makes the command (in this case a.out) unstable, not a lot in my opinion if we use it just for debugging purpose, but it increases the chance of symbol name conflicts.

You can use the same technique to override a method in a C++ class. Say there's a method named "fff" in a class AAA, like

class AAA  
{  
public:  
    int m;  
    AAA(){m = 1234;}  
    void fff(int a);  
};

To override it, you first need to know the mangled symbol name of the method.

$ nm somelibrary.dylib | grep "T "  
00000ed6 T __ZN3AAA3fffEi

Then what you need to define is _ZN3AAA3fffEi. Don't forget removing the first '_'. If you see multiple symbols in the shared library and not sure which one to override, you can check it by demangling a symbol like

$ c++filt __ZN3AAA3fffEi  
AAA::fff(int)  

Now you can override it like this.

$ cat mysharedlib.h
class AAA
{
public:
 int m;
 AAA(){m = 1234;}
 void fff(int a);
};
$ cat mysharedlib.cpp 
#include <stdio.h>
#include "mysharedlib.h"

void AAA::fff(int a)
{
 
 printf("--ORIGINAL:%d--", a);
}
$ cat main.cpp 
#include <stdio.h>
#include "mysharedlib.h"

int main()
{
 AAA a;
    printf("--------main1--------");
 a.fff(50);
    printf("--------main2--------");
 return 0;
}
$ cat openhook.cpp 
#include <stdio.h>
#include <dlfcn.h>
#include <unistd.h>
#include "mysharedlib.h"

typedef void (*AAAfffType)(AAA*, int);
static void (*real_AAAfff)(AAA*, int);

extern "C"
{

void _ZN3AAA3fffEi(AAA* a, int b)
{
    printf("--------AAA::fff--------");
    printf("%d, %d", b, a->m);
 void* handle = dlopen("mysharedlib.dylib", RTLD_NOW);
    real_AAAfff = (AAAfffType)dlsym(handle, "_ZN3AAA3fffEi");
 if (real_AAAfff) printf("OK");
 real_AAAfff(a, b);
}

}
$ cat bat
#!/bin/bash

gcc -flat_namespace -dynamiclib  -lstdc++ -o openhook.dylib openhook.cpp
gcc -dynamiclib -lstdc++ -o mysharedlib.dylib mysharedlib.cpp
gcc -lstdc++ mysharedlib.dylib main.cpp
export DYLD_FORCE_FLAT_NAMESPACE=
export DYLD_INSERT_LIBRARIES=openhook.dylib
./a.out
$ ./bat
 --------main1----------------AAA::fff--------50, 1234OK--ORIGINAL:50----------main2--------

Note that the first argument of the function call is this pointer, just like Python passes self to a bound method. C++ just does it implicitely. I believe this is compiler (in this case gcc) implementation specific, and there may be a case it is not true. Please use this technique at your own risk.

Here I assumed you have access to the header file that declares the function to get (the size of) argument data being passed to the function. If the size of arguments are all known (int, float, pointer, ...) you can wrap it even if you don't have the header but if not, you'll need to write assembler to modify the stack to pass the arguments to the original function.

So how do I use Shake Command Window?

I use it for plug-in development. You don't have to restart Shake every time you modify a code. You can see the internal state of nodes interactively, you can test your plug-in after modifying the node state, and you can see how things work, monitor plug changes to see if node evaluation chain is working as you expected. There are two programs I was mainly using. I made these two programs bit by bit interactively with the Command Window.

[dispTree] Displays a node tree.
name: the root tree
recurseNode: set true if you want to show child node
showPlug: set true if you also want to show plugs and their values
recursePlug: set true if you want to show child plugs and their values

























[dispPlugInfo] displays informations about a plug
nodename: node name which has the plug to display
plugname: plug name to display



























You may need to add include directives to the header include (settings - header include).
#include <string>, etc.

[dispTree]

const char* name = "NRiScript1";
bool recurseNode = true;
bool showPlug = false;
bool recursePlug = false;

#define print(x) NRiSys::error((NRiName() + x).getString());

const NRiName operator +(const NRiName lhs, const std::string rhs)
{
    return lhs + rhs.c_str();
}

void printSpace(int nestcount)
{
 for (int n = 0; n < nestcount; ++n)
 {
  print("  ");
 }
}


void dispPlug(NRiPlug* p)
{
 NRiName val;
 switch(p->getType())
 {
 case kString:
  val = p->asString();
  break;
 case kInt:
  val = p->asInt();
  break;
 case kFloat:
  val = p->asFloat();
  break;
 case kDouble:
  val = p->asDouble();
  break;
 case kPtr:
  val.sprintf("0x%x", p->asPtr());
  break;
 default:
  break;
 }
 print(".." + p->getName() + "=" + val + "\n");
}

void dispPlugTree(NRiPlug* plug, int nestcount, bool recursePlug)
{
 printSpace(nestcount);
 dispPlug(plug);
 if (recursePlug)
 {
  int ncp = plug->getNbChildren();
  for (int i = 0; i < ncp; ++i)
  {
   dispPlugTree(plug->getNthChild(i), nestcount + 1, recursePlug);
  }
 }
}

void dispNodeTree(NRiNode* node, int nestcount, bool showPlug, bool recurseNode = false, bool recursePlug = false)
{
 printSpace(nestcount);
 print(node->getName() + ":" + node->getClassName() + "\n");
 if (showPlug)
 {
  int np = node->getNbPlugs();
  for (int ip = 0; ip < np; ++ip)
  {
   dispPlugTree(node->getNthPlug(ip), nestcount + 1, recursePlug);
  }
 }

 if (recurseNode)
 {
  int numkids = node->getNbChildren();
  for (int i = 0; i < numkids; ++i)
  {
   dispNodeTree(node->getNthChild(i), nestcount + 1, showPlug, recurseNode, recursePlug);
  }
 }
}

void shakeCommandWinEntryFunc()
{
 NRiNode* node = NRiNode::findNode(name);
 dispNodeTree(node, 0, showPlug, recurseNode, recursePlug);
}

[dispPlugInfo]

const char* nodename = "NRiScript1.Gamma1";
const char* plugname = "In";


#define print(x) NRiSys::error((NRiName() + x).getString());


NRiName value(NRiPlug* p)
{
 NRiName val;
 switch(p->getType())
 {
 case kString:
  val = p->asString();
  break;
 case kInt:
  val = p->asInt();
  break;
 case kFloat:
  val = p->asFloat();
  break;
 case kDouble:
  val = p->asDouble();
  break;
 case kPtr:
  val.sprintf("0x%x", p->asPtr());
  break;
 default:
  break;
 }
 return val;
}

NRiName ioString(NRiPlug* p)
{
 const char* types[]  = {"ng", "in", "out", "inout"};
 return types[p->getIO()];
}

NRiName typeString(NRiPlug* p)
{
 switch(p->getType())
 {
 case kInt:
  return "int";
 case kFloat:
  return "float";
 case kPtr:
  return "ptr";
 case kString:
  return "string";
 default:
  return "ng";
 }
}

NRiName flagString(NRiPlug* p)
{
 NRiPlug::Flags flags[] = {NRiPlug::kInternal, NRiPlug::kNotify, NRiPlug::kNotifyConnect,
  NRiPlug::kInherit, NRiPlug::kDisconnectOnSet, NRiPlug::kPersistent,
  NRiPlug::kRecompile, NRiPlug::kLoadable, NRiPlug::kPreUpdate,
  NRiPlug::kOwnerScope, NRiPlug::kIgnoreConnect, NRiPlug::kAutoPlug,
  NRiPlug::kLocal, NRiPlug::kIgnoreDependencies, NRiPlug::kDeleting,
  NRiPlug::kXReadWrite, NRiPlug::kIgnoreType, NRiPlug::kAlwaysCallOwner,
  NRiPlug::kMonitor, NRiPlug::kNoExpr, NRiPlug::kInterrupt, NRiPlug::kFiltered,
  NRiPlug::kSerializeChildren, NRiPlug::kLookupSrc, NRiPlug::kExpressionSrc,
  NRiPlug::kUserDefined, NRiPlug::kPassThrough, NRiPlug::kCurveDefined,
  NRiPlug::kLocked};

 const char* flagStrings[] = {
  "Internal", "Notify", "NotifyConnect", "Inherit", "DisconnectOnSet",
  "Persistent", "Recompile", "Loadable", "PreUpdate", "OwnerScope",
  "IgnoreConnect", "AutoPlug", "Local", "IgnoreDependencies", "Deleting",
  "XReadWrite", "IgnoreType", "AlwaysCallOwner", "Monitor", "NoExpr",
  "Interrupt", "Filtered", "SerializeChildren", "LookupSrc", "ExpressionSrc",
  "UserDefined", "PassThrough", "CurveDefined", "Locked"};
 NRiName result;
 for (int i = 0; i < 29; ++i)
 {
  if (p->getFlag(flags[i]))
  {
   result += flagStrings[i];
   result += " ";
  }
 }
 return result;
}

NRiName hasErrorString(NRiPlug* p)
{
 if (p->hasError())
 {
  return "Yes";
 }
 else
 {
  return "No";
 }
}

void printPlug(NRiPlug* p)
{
 if (p)
 {
  print(p->getOwner()->getFullName() + ".." + p->getFullName());
 }
 print("\n");
}

void printPA(const NRiPArray < NRiPlug >&  pa)
{
 for(unsigned i = 0; i < pa.getLength(); ++i)
 {
  print("  ");
  printPlug(pa[i]);
 }
}

void shakeCommandWinEntryFunc()
{
 NRiNode* node = NRiNode::findNode(nodename);
 if ( ! node)
 {
  print("Node not found.\n");
  return;
 }
 NRiPlug* plug = node->getPlug(plugname);
 if ( ! plug)
 {
  print("Plug not found.\n");
  return;
 }

 print("<<<");
 print(node->getFullName() + ".." + plug->getFullName() + ">>>\n");
 print(NRiName() + "[Value] " + value(plug) + "\n");
 print(NRiName() + "[Expr] " + plug->asExpr() + "\n");
 print(NRiName() + "[IO] " + ioString(plug) + "\n");
 print(NRiName() + "[Type] " + typeString(plug) + "\n");
 print(NRiName() + "[Flags] " + flagString(plug) + "\n");
 print(NRiName() + "[HasError] " + hasErrorString(plug) + "\n");
 print("[Input] ");
 printPlug(plug->getInput());
 print("[LogicalInput] ");
 printPlug(plug->getLogicalInput());
 //print("[Ouput] ");
 //printPlug(plug->getOutput());
 print("[Outputs]...\n");
 NRiPArray < NRiPlug >  pa;
 plug->getOutputs(pa);
 printPA(pa);
 pa.clear();
 print("[getLogicalOutputs]...\n");
 plug->getLogicalOutputs(pa);
 printPA(pa);
 pa.clear();
 print("[getDependencies]...\n");
 const NRiPArray < NRiPlug > * pap;
 pap = plug->getDependencies();
 printPA(*pap);
 print("[getDependents]...\n");
 pap = plug->getDependents();
 printPA(*pap);
}

Nice blog

Today I found a nice blog which was made by members of feeling software. I knew the company but didn't know they have started a blog. They are especially interested in image recognition including computer vision (reconstructing a model from images etc.), and interactive technologies. Because they have a sound knowledge of theories behind them, their words are quite realistic. Just browsing the blog will make you fun if you have no time to read all.

enlighten3d - An expert blog on computer vision and 3D graphics

Shake Command Window ver. 0.94

As I wrote looong time ago, I released ShakeCommandWindow 0.94

New features are:

- Open window by pressing macro button
- At-least-not-crash level fail safe on pointer/floating point/pipe related runtime errors (1)
- Fix freeze bug when pressing macro button while settings panel is open
- Fix freeze bug when trying to display non-ascii string
- Created settings dialog (2)
- Modified default gcc header/footer includes

1.
I signal trapped inside the SCW plug-in so now it doesn't crash when your gcc code has a null-pointer bug, /0 bug, etc. This is not perfect, because newer gcc compilers do not let me raise an exception inside the signal handler, class destructors are not called property, it will probably cause memory leaks and some other side effects. That's why I call it At-least-not-crash level fail safe. But I was happy with this when I was using SCW to make another plug-in.

2.
For unknown reason, my wxPython wouldn't work anymore when a button is in a tab, so I made a separate settings dialog and "settings" button to open the dialog.


You can download it here.

Mayapad

A friend of mine is making a nice tool.













- syntax highlighting.
- undo/redo
- opening a file by drag&drop
- print messages and Python traceback on the output window
- other

Oct, 27 added.
- auto completion

Massive seminar

As many of the readers probably know, Massive is a software for crowd simulation which was first developed for The Lord of the Rings. I have a strong interest in the software because my last work was crowd simulator development (which concept is completely different from massive). I went to a seminar held by Crescent because this time Stephen Regelous, the founder and CTO of Massive Software was the presenter. He explained some of new features such as agent field with which you can feed boid force (each component separately) to the brain with lots of parameters to set weight and select which neighbor agents to be considered. At the party after the seminar I talked with him and found he is a very nice and friendly guy (I heard he spoke some Japanese, can he speak Japanese?).













Stephen, at the party


Another surprise was that I met a girl I know for the first time in 6 years! She is now working for Prometech Software, which I think is the best R&D based CG company in Japan. We talked lots of technical and non-technical stuff and had fun. She gave me the answer of one of my questions (Why does the free surface boudary condition used in one of particle based fluid simulations works) which had been puzzling me for a long time.

Porting Python2.x script to Python3.0

Because Python3.0 (aka Python3000) has no backward compatibility, it is supposed to cause some trouble when porting existing code to Python3.0. According to PEP3000 (PEP stands for Python enhancement proposal) , there is a recommended procedure to port 2.x code to 3.0. This is a quote from PEP3000

0. You should have excellent unit tests with close to full coverage.
1. Port your project to Python 2.6.
2. Turn on the Py3k warnings mode.
3. Test and edit until no warnings remain.
4. Use the 2to3 tool to convert this source code to 3.0 syntax. Do not manually edit the output!
5. Test the converted source code under 3.0.
6. If problems are found, make corrections to the 2.6 version of the source code and go back to step 3.
7. When it's time to release, release separate 2.6 and 3.0 tarballs (or whatever archive form you use for releases).

I recommend you read PEP3000 for upcoming Python3.0 release. Unlike many PEPs, it is quite easy to read. It's not urgent since Python3.0 is not released yet (according to PEP361, Python3.0 is planned to be released on Dec. 03 2008) , and even after its first release, the safest way is not using Python3.0. But if you stick to 2.x for years, you'll be left out from the rest of the world in the meantime.


For Python experts: did you know there's one byte code assigned just for PRINT_NEWLINE? No wonder Guido wanted to renew Python interpreter. You can see the list of byte codes here. (and compare it with Python3.0's byte code instructions, you'll see lots of them are simplified or generalized with arguments.)


by the way, is anybody reading this blog regularly? I'm just curious.

Factory / Ruin photos

I don't know if it's a world trend but in Japan beautiful factory photos are popular since several years ago.
This is a site made by a professional photographer who has published a photo book which has pictures of only industrial complexes.
http://www.kasako.com/0802keihinfoto.html

And ruin photos are getting popular as well.
Maybe he is not a professional photographer but his pictures are awesome.
http://urasoku.blog106.fc2.com/blog-entry-530.html

If you've seen "life after people" and you liked it, you'll also like these.

Ray Tracing Project

While browsing my friend's blog, I found he's mentioned a site that introduces a simple raytracer written in Haskell. That reminded me another site where you can see lots of simple raytracers implemented in various programming languages.
Currently they are written in Awk, C, C++, Java, JavaScript, Lisp, Mel, Perl, Python, Ruby, csh, and Tcl.
Somewhere I found a raytracer written in PostScript but I've forgotten where it was.

The eyeballing game

Another game for artists

















My score:

1st time: Couldn't see the rule.
2nd time: Mouse pointer jumped.
3nd time:


Parallelogram 8.2 4.1 1.0
Midpoint 2.2 4.2 4.2
Bisect angle 2.4 6.6 2.2
Triangle center 1.3 5.1 4.8
Circle center 0.0 2.8 4.5
Right angle 4.4 3.9 4.6
Convergence 2.0 2.2 2.2

Overall score: 3.47

It's more difficult than the previous one. Be prepared!

Python code reading

I made a presentation at a meeting called "Python code reading" held by a Python community in Japan. It's a meeting to read one standard Python module each time (a volunteer makes a presentation) to steal knowledge and techniques by reading scripts written by Python gurus. We read inspect module this time.

Here are several topics I mentioned during the code reading.

- basic usage of inspect module
- method generation mechanism by descriptor and decorator
- custom import (PEP302, which is used by importing zipped modules)
- code object and its meaning
- various special attributes (co_filelineno etc.)
- .pyo file
- file open mode 'U'
- imp module
- negative lookbehind assertion
- method resolution order
- how to clear the cache of the source code which is used by inspect.getsource(), etc.
(if the cache is kept uncleared, getsource() etc. still returns an old source after you modify the script and reload the module)

And what I studied during its preparation (but not talked during the presentation) are

- Python VM and byte code
- disassembling using dis module
- Python dev bug tracking

I didn't mention VM related stuff so that the meeting wouldn't be only for experts (and there was no time to explain them anyway).

Useful inspect module

Inspect module offers various useful functions to Python developers.
Here are four most basic and useful functions inspect module has.

Before showing them, I will just make a simple module (mymodule.py) for the example usages shown below. It has just one function and two classes.

#file mymodule.py


#f() just prints "hello"
def f():
return "hello"

class A(object):
def ma(self):
print "A_m"

class B(A):
def mb(self):
print "B_m"

getsource(object)
This function is probably the most useful one in the module, along with getabsfile().
It returns the source code where object is defined.
object can be a class, method, function, etc (not instance because an instance gets created dynamically).

>>> print inspect.getsource(mymodule.B)
class B(A):
def mb(self):
print "B_m"

getabsfile(object, _filename=None)
It returns the absolute path to the file where object is defined.

>>> print inspect.getabsfile(mymodule.A)
/Users/tamurakouichi/mymodule.py

getcomments(object)
Get lines of comments immediately preceding an object's source code.
the object can be a class, function, etc...

>>> print inspect.getcomments(mymodule.f)
#f() just prints "hello"

classify_class_attrs(object, predicate=None)
It returns a detailed information of attributes a class has.
It includes in what class a method is defined.

>>> for m in inspect.classify_class_attrs(mymodule.B):
...     print m
...
...snip...
('ma', 'method', <class 'mymodule.A'>, <function ma at 0x260c30>)
('mb', 'method', <class 'mymodule.B'>, <function mb at 0x260c70>)

There are many many functions. inspect module is a friend of every Python users from beginners to byte code analysers.

Ragel State Machine Compiler

Have you seen this?

Ragel State Machine Compiler

Ragel compiles executable finite state machines from regular languages. Ragel targets C, C++, Objective-C, D, Java and Ruby. Ragel state machines can not only recognize byte sequences as regular expression machines do, but can also execute code at arbitrary points in the recognition of a regular language. Code embedding is done using inline operators that do not disrupt the regular language syntax.

Interesting, and looks so useful !



Oct. 9 added.

I found another tool.
The State Machine Compiler
While Ragel is more suitable for text processing, this one is something more general purpose. Though it is interesting, I wonder if it is so useful. I know it's tedious to turn a state diagram into a code when the number of states/actions gets larger but it's just tedious and not confusing. Just wondering if adding a tool for it in the project is worth while, trading off building and porting simplicity.
Well, I haven't looked through the document. I'm just saying that after the first look and may change my opinion later.

By the way bringing the concept of state machine in visual programming can be interesting. It's an easy to learn concept and naturally easy to visualize (thinking about something weird again ;)

Light-Paint Piano Player

Light-Paint Piano Player from Ryan Cashman on Vimeo.

Nice!

Let's not use 3D pie chart

From this blog,

This is a share of textbooks for Tokyo metropolitan high schools.










The share of Nichibun (brown, 14.5%) is twice as much as Keirincan (blue, 7.2%) but the center angle is smaller than Keirincan, and those look nearly the same in size.

I would like people teach students not to use 3D pie chart in their information literacy class.
Pie chart itself is not good, as is described in R help,

Pie charts are a very bad way of displaying information. The eye is good at judging linear measures and bad at judging relative areas. A bar chart or dot chart is a preferable way of displaying this type of data.

Cleveland (1985), page 264: “Data that can be shown by pie charts always can be shown by a dot chart. This means that judgements of position along a common scale can be made instead of the less accurate angle judgements.” This statement is based on the empirical investigations of Cleveland and McGill as well as investigations by perceptual psychologists.

3D pie chart is even worth than pie chart.


--------------------
This document is licensed under the Creative Commons Attribution 2.1 Japan License.
copyright by Haruhiko Okumura

Python virtual machine document

I've finished reading an excellent document on Python virtual machine and related stuff.
http://www.nasuinfo.or.jp/FreeSpace/kenji/sf/python/virtualMachine/PyVM.htm (Japanese)

It covers

- disassembling with inspect module and dis module
- FrameObject, FunctionObject, CodeObject and their relationship
- data stack
- local, enclosure, global, builtin scope ... how they are implemented
- how to read Python Virtual Machine source code (ceval.c)
- other

I don't know if Japanese-English machine translation works well but if it does it'll be a good start to understand Python VM.

p.s
I'm currently reading inspect module to make a speech on "Python code reading", a meeting held by a local Python community in Tokyo.

matplotlib

matplotlib is a python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms. matplotlib can be used in python scripts, the python and ipython shell (ala matlab or mathematica), web application servers, and six graphical user interface toolkits.

http://matplotlib.sourceforge.net/

Looks nice.
Found the info from a blog entry on Ramblins about Python

multiprocessing — Process-based “threading” interface in Python2.6

http://docs.python.org/dev/library/multiprocessing.html#module-multiprocessing
Haven't read it but looks it's a similar approach to POSH.
I'll update this post when I've read it through.

Visual programming

I have a strong interest in visual programming. Many programmers say it is not practical but they are ignoring its big success in CG. Most highend CG softwares have a node tree/graph and it is a visual programming environment in a sense.

Currently people are still making small ad-hock scripts for daily works if there's no appropriate tool exists, renaming files, file type conversion, registering something to another, opening a file, add some comment on it, save it and close it, find something in a file, replacing it, brabrabra... and the repeated execution of mixture of them. In big productions lots of works are automated but still they are doing a lot like this for non-procedual works daily.

In my opinion it's absolutely ridiculous to make scripts for them. Why do we have to make scripts for such small things? But currently there's no easier way.

There are several reasons why Python itself is not perfect for this reason.
- A module is too big to back-up dozens of smallest code snippets for later use.
- If a module has been changed, it'll be affected by all the scripts that uses it.
- Scripting is not intuitive.
- Python itself is not a tool to find a code snippet quickly.
- Try and error is not easy. Often you need to re-run a script.
(You may think "No, it's not. Everything listed here is so easy in Python", but I'm thinking of them from a point of view of non-technical people)

Several years ago I made a prototype of a Python based visual programming framework. It's target is artists who are not so much familiar with scripting. You can change the node network dynamically and see the result immediately, registering the node for later use, and reusing it by cloning. A node itself is a code snippet as well as a GUI representation tool so the user can relatively easily find where to look at when he is modifying the network .
















You can find more information, and several capture movies here.

I'm making it with a German guy. Currently the project is almost frozen but I'm still hoping to release ver.1
I cannot say a lot cause we may release it commercially but he is looking at the tool from a different aspect, i.e. for professional programmers so it'll be a much larger system than what you would imagine after reading this post.

Test your color IQ

Test your color IQ

Found the test looking at my friend's SNS.
Drag&drop the tiles and reorder them.















I got perfect at the second try.

Why does a method becomes bound method when it is accessed with an instance?

Short answer:

Because it is part of Python specification.

Long answer:

It is something to do with __get__ special method.
If an attribute has a method __get__, it is called when the attribute is accessed with . (dot)

>>> class C(object):
...     def __get__(*arg):
...             print "hello", arg
...
>>> class D(object):
...     d = C()
...
>>> D.__dict__["d"]
<__main__.C object at 0x00B32CB0>
>>> D.d
hello (<__main__.C object at 0x00B32CB0>, None, <class '__main__.D'>)
>>> D().d
hello (<__main__.C object at 0x00B32CB0>, <__main__.D object at 0x00B32CD0>, <cl
ass '__main__.D'>)

When you define a function, Python creates a function object and a function object has __get__ method.

>>> def f():pass
...
>>> f
<function f at 0x00B2EF30>
>>> dir(f)
['__call__', '__class__', '__delattr__', '__dict__', '__doc__', '__get__', '__ge
tattribute__', '__hash__', '__init__', '__module__', '__name__', '__new__', '__r
educe__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', 'func_closure',
'func_code', 'func_defaults', 'func_dict', 'func_doc', 'func_globals', 'func_na
me']

and when you call __get__ of the function object, guess what will be returned?

>>> f.__get__(D())
<bound method ?.f of <__main__.D object at 0x00B32D90>>

Yeah it returns a bound method. That's why a method becomes bound method when it is accessed with an instance.

The short answer is a Python specification and the long answer is its implementation. property is also implemented with __get__.

>>> dir(property(f))
['__class__', '__delattr__', '__delete__', '__doc__', '__get__', '__getattribute
__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__
', '__set__', '__setattr__', '__str__', 'fdel', 'fget', 'fset']

Sep. 5 added:

> This is called the descriptor protocol.
>
> See the following for a long description of descriptors:
>
> http://users.rcn.com/python/download/Descriptor.htm

Thanx Fuzzyman!

X-Ray shader

Just a simple shader I made long long ago.



























more info (Japanese)

Binary Hacks

There are not many good books written in Japanese especially around programming (most of them are entry level) but this is one of the rare exceptions.



















You'll see how good it is by taking a glance at its contents.
There's no equivalent book in English but the closest one will be linkers and loaders with the flavor of cookbook, though there are wide variety of hacks, e.g. hack#37 is 4 ways to implement singleton in c++, hack#68 demangling c++ symbol names at run time.

Another exception I know is Henrik Wann Jensen's photon mapping book. The translator of the book himself is a computer graphics researcher and he added lots of lots of footnotes that help the reader understand it!

How to access to Maya python objects from a plug-in written in C++.

This is a modified copy of my old document (you can still see the original somewhere on the net).
When I wrote this, Maya8.5 was latest, and it is still my latest unfortunately, so in this doc Maya8.5 is used. If you use Maya2008, you need to use Python2.5 (e.g. Python25.dll). And I was using windows at that time. I will add to this post how to do it on Linux and Mac soon. (Encouraging me to do so by posting a comment will become a good motivation :)

------------------------------------------------------------------------------------

Maya8.5 comes with Python. It is really not a must to know how to access to it from a plug-in written in C++ since you can make a plug-in in Python, but for some reason e.g. performance, you may want to know the way to access to it directly from your Maya .mll/.so/.bundle plug-in.

Python itself is a shared library which you can find at Maya8.5/bin/python24.dll (on windows) so if Maya can access to Python, our plug-ins can get access to it.

First you need to include Python.h before every includes like any other python programs. The version of Python must be 2.4 since Python inside Maya is version 2.4.3. Other version of Python e.g. 2.5 uses different shared library e.g. python25.dll and you cannot use the same Python as the one Maya uses.

Maya8.5 has already initialized Python interpreter so you don't need to (and you shouldn't) call Py_Initialize(), but in the doIt() (in case of a command plugin) of your plug-in, you need to call
PyGILState_Ensure()
Python interpreter is not thread safe and this function call locks the interpreter properly so that only the thread that is running your plug-in can access to Python. In the test plug-in code show below, I used PyRun_SimpleString() to access to Python. See this post to know how to use Python C/API. When you leave doIt(), you need to call
PyGILState_Release()
It is needed even if you don't use Python thread modules. Forgetting this freezes or crashes Maya easily.

#include <Python.h>
#include <maya/MGlobal.h>
#include <maya/MSimple.h>
DeclareSimpleCommand( maya85py, "", "8.5");

MStatus maya85py::doIt( const MArgList& args )
{
PyGILState_STATE state = PyGILState_Ensure();
PyRun_SimpleString("tamtam = 1");
PyGILState_Release(state);
return MS::kSuccess;
}

Here's the result of above test plug-in.

tamtam
# Error: name 'tamtam' is not defined
# Traceback (most recent call last):
#   File "<maya console>", line 1, in ?
# NameError: name 'tamtam' is not defined #

import maya
maya.mel.eval("maya85py")

tamtam
# Result: 1 #

Debug build:
On windows, the compiler tries to link python24.lib or python24_d.lib automatically (on Linux and Mac you need to link the appropriate one manually). python24_d.lib is a debug version and you need to build python itself to get it. I surfed the web but couldn't find it, and tried to build python but full build was a little bit bothering (install tcl/tk, brabrabra...), so I just modified pyconfig.h in my Python24/include directory. It's all what you need to debug build your plug-in (You don't need this if you don't use the debugger). Here I just used Python header files and import library that comes with standard Python distribution, since Maya doesn't come with them. Recently I heard one of my friend made an import library using dumpbin command. It'll be better if you have time to do it (still not safe though).

//#                pragma comment(lib,"python24_d.lib")
#                pragma comment(lib,"python24.lib")
#            else
#                pragma comment(lib,"python24.lib")

Finally, in an actual plug-in PyRun_SimpleString("tamtam = 1"); can be as complicated as you want, missing PyGILState_Release() execution by some exception is quite dangerous. I would use try-catch clause.