CPU/GPU memory abstraction

When you write a program that uses GPU (either CUDA of OpenCL), you may want to implement both CPU code and GPU code, and use only one of them depending on the user's choice. For this reason I wanted to know how to treat different kinds of memories generically, i.e. the way  to abstract host and device memory.

 The easiest one is to follow the approach of (or just use) Thrust. It has host_vector and device_vector which interfaces are quite similar to std::vector. When you have one of them, you can copy it to another with assignment. Host/device copy will be done automatically under the hood.

 I wanted another way of abstraction to debug the program easily. What interested me was to have a mechanism where input data was either in host or device memory, and the the code does not have to know where it is. After several try and error I implemented one like this;

template < typename T >
class Buffer
{
public:
    T* get(MemoryType mType);
    const T* get(MemoryType mType) const;
    void setClean(MemoryType mType, bool isClean=true);
    void sync(MemoryType mType) const;
    void allocate(MemoryType mType) const;
    void free(MemoryType mType);

private:
    mutable MemoryType m_cleanState; //Clean state. Bitwise OR of HOST and DEVICE.
    mutable void* m_addrs[2]; //Host and device.
};

It can have both host and device memory, and knows if the data stored in the host/device memory is up to date or not. If the one in the host memory is up to date and one in the device memory is not, it copies data from the host memory to the device by calling sync(DEVICE). if the device memory is already up to date, sync() does nothing.
 You can use this class like this,

someCalculationCpu(const Buffer < float > * input, Buffer < float > * output)
{
    input.sync(HOST);
    float* ip = input.get(HOST);
    float* op = output.get(HOST);
    op[0] = ip[0]; /*Do some calculation with CPU*/
    output.setClean(HOST); //Tell the buffer that the data stored in the device memory is up to date.
}

someCalculationGpu(const Buffer < float > * input, Buffer < float > * output)
{
    input.sync(DEVICE);
    float* ip = input.get(DEVICE);
    float* op = output.get(DEVICE);
    op[0] = ip[0]; /*Do some calculation with GPU*/
    output.setClean(DEVICE); //Tell the buffer that the data stored in the device memory is up to date.
}

anotherCalculationCpu(const Buffer < float > * input, Buffer < float > * output)
{
    /*Same style as someCalculationCpu() with another calculation.*/
}

anotherCalculationGpu(const Buffer < float > * input, Buffer < float > * output)
{
    /*Same style as someCalculationGpu() with another calculation.*/
}

Now the these are all valid,

   someCalculationCpu(input, output);
   anotherCalculationCpu(input, output);

   someCalculationCpu(input, output);
   anotherCalculationGpu(input, output);

   someCalculationGpu(input, output);
   anotherCalculationCpu(input, output);

   someCalculationGpu(input, output);
   anotherCalculationGpu(input, output);

I've already implemented so I'll keep using it but just wonder if there is already a tool or a way with Thrust. Please leave a comment if you know.

Then only thing I expect to every programmer

is to know the difference between interface and implementation.
It includes, but not limited to spec.

Two-Scale Particle Simulation

Two-Scale Particle Simulation

It's a straightforward, practical idea to reduce simulation time. First simulate fluid with roughly, and simulate again only within some regions (in the paper, water surface and camera frustum is taken as examples) that needs accracy which boudary conditions given by the rough simulation. Good for multi-threading.

Maya API charts

These are a couple of charts that tells what methods you can use to get an object you need. I wrote them when I was writing a Maya programming book (which ended up not being published), when Maya version was 5 so they're old and the interface and behavior written there may be changed but most of them are still valid.

The first chart shows the relationship between MDagPath, MObject, MFn*, etc. The second one focuses on plugs and attributes. The term "attribute" and "plug" are distinguished strictly in the API. You can say a node *type* has attributes, not a node, so

correct: a mesh has an attribute translateX.
wrong: pCone1 has an attribute translateX.

It indicates you need two informations to get a plug for "pCone1.translateX". First, the node is "pCone1", and second, the attribute is "translateX". You'll see it on the second chart.


Disclaimer: I do not guarantee the accuracy of the information. Refer to it at your own risk.

SPH second test

18000 particles.

One more video

Find .mov and .ogv from a directory and its sub directories, convert them to .avi files and store them to another directory. 1080p available.

Zamami pre-alpha

This is a software I've been making.
Watch them on full screen mode with 720p to see how the tool works in detail.

Basic usage


Batch execute and built-in debugger


sqlite


Detect missing files in an image sequence


job dispatch


fancy template


I made a twitter client on it for a complecated node-network example but twitter has turned off Basic auth last year and it doesn't work now. I'll upload it when I've made it working.

Tips to obfuscate your code.

I often realize people make their code hard to maintain. It's a very good idea to raise your value in the company since nobody else can maintain it. Let's learn from their code. Often you can even get more efficient code since you don't need to write extra lines to make your code nicer. It doesn't cover basic skills like using magic numbers, not making a method name readable, give an object more than one name, delete all comments, etc (Do all of them!). Ideas are listed in order of importance.

(1) Store the same value in more than one places.

If you store the same parameter in more than one places, you need to synchronize them. When a maintainer write a code that changes the value, he will have a change to miss the fact and change only one of them. Every code that uses the other variable which wasn't changed will behave mistakenly. Hopefully the program will be in an inconsistent state.

(2) Store lots of data in an object and every method depends on them

If lots of methods are dependent on the state of the object, the user of the class will need to set all of them properly before using the method and he will be confused. A nice side effect is that it makes the program hard to debug and test cause the behavior of a method can have lots of right and wrong cases. Try packing lots of unnecessary parameters, make the state of objects complicated, and make it hard to maintain.

(3) Reference an object from lots of other objects.

People who look at it will be puzzled, "Is the object I can get from A is the same as what I can get through B?". This strategy is good with (1) above. Unnecessary references are good!

Dec. 24:
Just tried to tell what will happen if you do them. Don't do them of course :)

Coq Proof Assistant

From the homepage.

Coq is a formal proof management system. It provides a formal language to write mathematical definitions, executable algorithms and theorems together with an environment for semi-interactive development of machine-checked proofs.

I studied basic symbolic logic to learn Coq, but its still very difficult. I still have no idea about correspondence between "A's type is B" and "A is a proof of B" (Curry–Howard correspondence).

Re-opened hohehohe2's OpenMaya tutorial

It's moved here!
http://www.daisukemaki.com/archive/koichi/0mokuji.html

Thank you Maki for hosting it.

Qt Designer bad knowhows (*)

"bad knowhow" is a Japangrish meaning it is something you have to use knowing it is not a nice way.

Qt designer is a good tool but there are several things you need to know until you get familiar with it.

1) Making widgets expand within a window
It is the first thing most people takes a few hours googling to get the answer. The answer is tricky.

- Drag and drop arbitrary widget (button or spacer is enough)
- Right click the background of the window (not the widget you have dropped)
- Select Layout - Layout Horizontally (or Layout Vertically) in the pop-up menu

You cannot select Layout- Layout Horizontally (Vertically) until you create a widget on the window.

2) Moving a layout from its parent to another.
When the UI gets complicated, it is often difficult to move the right layout to the right place.

To move the right layout:
- Select the layout in the Object Inspector.
- Click the layout
Qt Designer will pick the selected layout.

To the right place:
- When clicking the layout, click near the top left corner of the layout
- Drag and drop it to the new parent in the Object Inspector
Unless you click new the top left corner, you'll be puzzled when dragging it to the Object Inspector

3) Try to copy and paste a widget and get error "Cannot paste widgets. Designer could not find a container without a layout to paste into."

My solution is to create a new dummy window (with ctrl-N) and copy&paste to it. then drag and drop it to wherever you wish to. There may be other ways.

I will add a screen shot on request.
When using the qt-designer generated file (*.ui), never modify automatically generated file. For example, I use Python and I convert *.ui to *.py with pyuic4 tool. I never modify the .py file by hand, which would halve the qt designer's strength. You will quite often want to update the GUI design and once you have modified the generated file, your modification to the source code would be lost. Insted, use the file from another. Same applies to C++ users to.

By the way qt designer is in qt-devel yum package (e.g. qt-devel-4.6.3-8.fc13.i686). It tool a while for me to find it.

Scons

This is my log of start using SCons, you don't need to read this if you are familiar with SCons already. Most of my knowledge comes from SCons User Guide, and many of examples shown here were taken from the guide.


I looked for a build tool (like make) for my project. Like everybody else in the world, I'm sick of traditional make and needed a more elegant one.OMake's concurrent build (-P switch) attracted me a lot but it seemed to need some time to use Omake with emacs. Scons is a build tool (like make) fully utilizes Python, and there is no need to code in OCaml (omake) (*).

Once you have installed scons (if you use macport to install, you also need py25-hashlib to be installed), the first step of using Scons is to write a SConstruct file (Makefile equivalent), which is actually a Python script (so you can write a Makefile equivalent in Python).
If you just have two files hello.cpp which uses hello.h, a simple SConstruct file is

Program('hello.cpp')

to build, you just type

scons

Scons automatically looks at the source code hello.cpp and it is dependent on hello.h, so whenever hello.h changes, scons knows it needs to rebuild hello.

SCons User Guide is very well written and easy to read document but it is targeted for non-Python programmers and a little bit verbose, so I'll just write down the most important parts (for me) of the guide here. It doesn't cover throughout hte guide, I only read the first several chapters, up to 7.2.1 and I felt I can already use Scons to some extent.


-Program(['foo.cpp', 'bar.cpp']) if there are two sources.

-You can write Program('program', Glob('*.c')).

-SharedLibrary('foo', ['f1.c', 'f2.c']) and StaticLibrary('foo', ['f1.c', 'f2.c']) for shared and static library.

-Program('prog.c', LIBS=['foo', 'bar'], LIBPATH='.')

-Program('hello.c', CPPPATH = ['include', '/home/project/inc']) for -Iinclude -I/home/project/inc

-Program('prog.c', LIBS = 'm', LIBPATH = ['/usr/lib', '/usr/local/lib']) for -L/usr/lib -L/usr/local/lib -lm

-You can write SConstruct script like,

     hello_list = Object('hello.c', CCFLAGS='-DHELLO')
    goodbye_list = Object('goodbye.c', CCFLAGS='-DGOODBYE')
    Program(hello_list + goodbye_list)

Object means "Object file" (i.e. *.o file). Not Python object, nor any other object.


-Explict dependency.

      hello = Program('hello.c')
     Depends(hello, 'other_file')

The following scripts are also accepted.

Program('hello.c')
Depends('hello', 'ho.cpp')

-Construction Environment

env = Environment(CC = 'gcc', CCFLAGS = '-O2')
env.Program('foo.c')

To make it default settings (so that you can write Program('foo.c') instead of env.Program('foo.c')),

DefaultEnvironment(CC = 'gcc')

- SConscript(['subdirectory/SConscript']) for hierarchical build,

- Program(['foo.cpp', 'bar.cpp']) makes build target foo (looks scons gets the name from the first item in the list, not where the main function exists). Default(Program(['foo.cpp', 'bar.cpp'])) to set it as the default target.



(*) OMake official site says "There is no need to code in Perl (cons), or Python (scons)." ;)

Recipe 577237: Prevent star imports (Python)

Recipe 577237
Use this code in your module to prevent people using the "from foo import *" syntax with your module.

@apply
class __all__(object):
    def __getitem__(self, _):
        raise ImportError("Star imports not supported")

Path class

I hope one of these stuffs is in the standard library.

>>> p = Path('/path/to/some/text.txt')
>>> p.parent()
Path('/path/to/some')
>>> p.parent().parent()
Path('/path/to')
>>> p.parent().parent().parent()
Path('/path')
>>> p.parent().parent().parent().parent()
Path('/')
>>> p.parent().parent().parent().parent().parent()
Path('/')
>>> p.parent().child('other').child('text.txt')
Path('/path/to/some/other/text.txt')
>>> p[:-1]
Path('/path/to/some')
>>> p[1:-1]
Path('path/to/some')
>>> p[2:-1]
Path('to/some')

You cannot write e.g. p[2:3] = p('aaa') since conceptually the Path class is immutable.

import os

class Path(object):
    def __init__(self, path):
        if isinstance(path, Path):
            path = path.path
        self.path = path

    def parent(self):
        return Path(os.path.dirname(self.path))

    def child(self, name):
        if isinstance(name, Path):
            name = name.path
        childPath = os.path.join(self.path, name)
        return Path(childPath)

    def base(self):
        return Path(os.path.basename(self.path))

    def abspath(self):
        return Path(os.path.abspath(self.path))

    def __str__(self):
        return self.path

    def __repr__(self):
        return self.__class__.__name__ + '(' + repr(self.path) + ')'

    def __add__(self, rhs):
        if isinstance(rhs, Path):
            return self.child(rhs.path)
        else:
            return Path(self.path + rhs)

    def __eq__(self, rhs):
        if isinstance(rhs, Path):
            rpath = rhs.path
        elif isinstance(rhs, unicode):
            rpath = rhs
        else:
            return False
        return os.path.abspath(self.path) == os.path.abspath(rpath)

    def __ne__(self, rhs):
        return not self.__eq__(rhs)

    def __getitem__(self, key):
        this, parent = self, self.parent()
        bases = [this.base()]
        while this != parent:
            bases[:0] = [parent.base()]
            this, parent = parent, parent.parent()
        bases[0] = this
        bases = bases[key]
        retPath = bases[0]
        for base in bases[1:]:
            retPath = retPath.child(base)
        return retPath

    @staticmethod
    def getCurrent():
        return Path(os.getcwd())

To extract the Python string a Path object has, you can pass the object to str() or unicode(). A friend of mine gave me an idea to make the class to be a subclass of str/unicode so that a Path object can be used as a Python string. But the meaning of __eq__, __ne__, and __getitem__ is quite different between str/unicode and my class.

SIKULI is fun.

This is my first SIKULI script which makes a sphere, duplicate it, and move it.











I like its "what you see is how it works" style scripting :)

The Third & The Seventh

Amazing!

A FULL-CG animated piece that tries to illustrate architecture art across a photographic point of view where main subjects
are already-built spaces. Sometimes in an abstract way. Sometimes surreal.
.Fullscreen it, please. (comment by the Author)

The Third & The Seventh from Alex Roman on Vimeo.

What every programmer should know about memory

I started reading What every programmer should know about memory.
It's much more advanced than "L2 cache is 10 times faster than memory access" stuff.

Not a nice way to interrupt a sub thread

In my application, the main thread in in charge of UI drawing with wxPython and starting worker threads which runs user defined Python code. If a user defined Python code is buggy the application user needs to stop it. There's a way to raise a KeyboardInterrupt exception in the main thread but unfortunately not the opposite. What I needed to do is interrupting a sub thread from the main thread. I don't know why it doesn't exist. Probably it is a well considered decision but what I need is what Python misses. The API provides a function PyThreadState_SetAsyncExc() which takes a thread id and exception object and raises exception in the thread. So I had to make a wrapper extension.

pySubthreadInterruptTest.c

#include <Python.h>

static PyObject* interrupt(PyObject* self, PyObject* args)
{
    long threadid;
    PyObject* po_exception;
    if(! PyArg_ParseTuple(args, "lO", &threadid, &po_exception))
    {
        return NULL;
    }

    int result = PyThreadState_SetAsyncExc(threadid, po_exception);
    return Py_BuildValue("l", result);
}

static PyMethodDef MethodsDefs[] = {
    {"interrupt", interrupt, METH_VARARGS},
    {NULL, NULL, 0},
};

void initpystit(void){
    (void) Py_InitModule("pystit", MethodsDefs);
}

setup.py

from distutils.core import setup, Extension

module1 = Extension('pystit',
                    sources = ['pySubthreadInterruptTest.c'])

setup (name = 'PackageName',
       version = '1.0',
       description = 'This is a demo package',
       ext_modules = [module1])

test.py

import time, thread
import pystit
def f():
    print 'thread start'
    try:
            for i in range(1000):
                    print i
                    time.sleep(0.001)
    except:
            print 'interrupted'
            raise
    print 'thread end'
tid = thread.start_new_thread(f, ())
time.sleep(0.001)
pystit.interrupt(tid, ValueError)
time.sleep(1)

$ python setup.py build
running build
running build_ext
building 'pystit' extension
gcc -pthread -shared build/temp.linux-i686-2.6/pySubthreadInterruptTest.o -L/usr/lib -lpython2.6 -o build/lib.linux-i686-2.6/pystit.so
$ cd build/lib.linux-i686-2.6/
$ ls
pystit.so  test.py
$ python test.py
thread start
0
1
2
3
interrupted
Unhandled exception in thread started by <function f at 0xb8043e64>
Traceback (most recent call last):
  File "test.py", line 8, in f
    time.sleep(0.001)
ValueError
$ 

It works but I can't say I'm quite satisfied with the solution.

A module file can be loaded twice as two different modules (in Python 2.x)

I'll show you a case where one file (mymodule.py) gets loaded twice as two different modules. I'm sure this is trivial to some people, if you know how to do this, you don't have to read the rest of this entry. I will write it anyway hoping it will help somebody since it could bring a huge confusion if he doesn't know why it happens (huge confusion, according to my experience).

Modules and packages in directories found in sys.path are called toplevel. Those which are not in the directories found in sys.path but in a package are still accessible but not toplevel.

Say you have a package named mypackage in a directory you can find in the PYTHONPATH, and it has a module mymodule. My package is a toplevel and mymodule is not.

$ pwd
/home/kotamura/mytest
$ echo $PYTHONPATH
/home/kotamura/mytest/mytoplevel
$ tree mytoplevel/
mytoplevel/
`-- mypackage
|-- __init__.py
`-- mymodule.py

Let's run Python interactively.

$ python
Python 2.6 (r26:66714, Jun  8 2009, 16:07:26)
[GCC 4.4.0 20090506 (Red Hat 4.4.0-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import mypackage.mymodule as mm
>>> mm.__name__
'mypackage.mymodule'

You can see the module is not a toplevel from its name (it's under mypackage.)
Go down the directories and do it again.

$ python
Python 2.6 (r26:66714, Jun  8 2009, 16:07:26)
[GCC 4.4.0 20090506 (Red Hat 4.4.0-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import mypackage.mymodule as mm
>>> mm.__name__
'mypackage.mymodule'

It's not surprising at all. But you can also import mymodule

>>> import mymodule as m
>>> m.__name__
'mymodule'

Now mymodule is toplevel!, This is because if you run Python interactively Python adds the current directory to sys.path. What is important is that mm and m are different objects even if it is created from the same file mymodule.py

>>> mm.a = 1
>>> m.a = 2
>>> mm.a
1
>>> mm is m
False

It's not the case you will see only when you use interactive console. The same thing happens when you import a module from another module without specifying it from the toplevel.

$ cat weirdImporter.py
import mypackage.mymodule as mm
import mymodule as m
print mm.__name__
print m.__name__

You don't have to be in the directory to run weirdImporter.py. You can run it from anywhere

$ python /home/kotamura/mytest/mytoplevel/mypackage/weirdImporter.py
mypackage.mymodule
mymodule

Again mymodule.pygot imported twice.

import mymodule as m

Python lets weirdImporter import mymodule because it is in the same directory, and since weirdImporter runs in the __main__ module (not mypackage.weirdImporter I mean), it imports mymodule as a toplevel. While

import mypackage.mymodule as mm

imports the mymoduel.py as mypackage.mymodule. So they are different.

A solution many people recommend is that you always import a package/module specicfying the path from the toplevel. It will solve every problem. You can also use "relative import" first introduced in Python2.5 but it brings another confusion until you get used to it.

Finally, Python3.x doesn't let weirdImporter import mymoduel.py only because it's in the same directory. But you still need to be careful not to have the same file be imported more than once from multiple toplevels. Don't forget executing 'python /path/to/foo.py' adds /path/to directory to sys.path.

Thread Manager

Last update Jun 10

import thread, threading

class _WorkerThread(threading.Thread):
    def __init__(self, sem, job, *args, **kargs):
        self._workSem = sem
        super(_WorkerThread, self).__init__()
        self.job = job
        self.args = args
        self.kargs = kargs
        self.isCanceled = False
        self.hasStarted = False
        self._cancelLock = threading.Lock()

    def run(self):
        if self._workSem:
            with self._workSem:
                with self._cancelLock:
                    if self.isCanceled:
                        return
                    self.hasStarted = True
                self.job(*self.args, **self.kargs)
        else:
            self.job(*self.args, **self.kargs)

class JobManager(object):
    def __init__(self, maxnumthreads = 1):
        """Set maxnumthreads to specify the max number of threads which runs concurrently."""
        self._workers = {}
        self._unmanagedWorkers = []
        self._workSem = threading.Semaphore(maxnumthreads)
        self.maxnumthreads = maxnumthreads
        self._lock = threading.RLock()
        self._nextJobIdCounter = 1

    def postJob(self, job, *args, **kargs):
        """job must be a callable, args and kargs are arguments passed to it."""
        self._gc()
        wt = _WorkerThread(self._workSem, job, *args, **kargs)
        wt.start()
        id = self._nextJobIdCounter
        self._workers[id] = wt
        self._nextJobIdCounter += 1
        return id

    def cancelJob(self, jobid):
        """Cancel a posted job. jobid must be an object returned by postJob().
        It returns True if the job gets canceled, False it it has started."""
        self._gc()
        worker = self._workers.get(jobid, None)
        if not worker: #It doesn't exist because it finished execution and removed from the _workers
            return False
        with worker._cancelLock:
            worker.isCanceled = True
            return not worker.hasStarted

    def waitOnIdle(self):
        """Blocks until the every worker thread terminates."""
        self._gc()
        while self._workers or self._unmanagedWorkers:
            wt = (self._workers.values() + self._unmanagedWorkers).pop()
            wt.join()
            self._gc()

    def getNumWaitingJobs(self):
        """Returns the number of jobs waiting. It includes threads currently running."""
        self._gc()
        return len(self._workers)

    def forceExecuteOnWorkerThread(self, job, *args, **kargs):
        """Execute the job immediately on a thread. It is not queued."""
        wt = _WorkerThread(None, job, *args, **kargs)
        wt.start()
        self._unmanagedWorkers.append(wt)

    def executeWhenNoWorkerThreadsRunning(self, job, *args, **kargs):
        """The calling thread execute the job (callable), ensuring no worker threads running.
        It blocks when a thread is running.It DOESN'T mean job is executed after the every
        worker threads has been terminated.
        (Though it looks the current Python implementation awakens a thread which called acquire()
        earlier.)
        It doesn't take jobs launched by forceExecuteOnWorkerThread() into account."""
        for i in range(self.maxnumthreads):
            self._workSem.acquire()
        try:
            job(*args, **kargs)
        finally:
            for i in range(self.maxnumthreads):
                self._workSem.release()

    def _gc(self):
        with self._lock:
            self._workers = dict([w for w in self._workers.items() if w[1].isAlive()])
            self._unmanagedWorkers = [wt for wt in self._unmanagedWorkers if wt.isAlive()]


if __name__ == '__main__':
    import time
    def somejob(i, wait = 0.1):
        time.sleep(wait)
        print 'somejob', i
        time.sleep(1)
    def endMessage(msg):
        print msg

    jm = JobManager()
    jobIds = []
    for i in range(5):
        id = jm.postJob(somejob, i + 1)
        jobIds.append(id)
        print 'posted', id
    time.sleep(2)
    print "cancel", jobIds[0], jm.cancelJob(jobIds[0])
    print "cancel", jobIds[-1], jm.cancelJob(jobIds[-1])
    #print jm.getNumWaitingJobs()
    jm.forceExecuteOnWorkerThread(somejob, 'unmanaged1', 4)
    jm.waitOnIdle() #It waits for unmanaged1 termination
    jm.forceExecuteOnWorkerThread(somejob, 'unmanaged2')
    jm.executeWhenNoWorkerThreadsRunning(endMessage, 'done') #It doesn't wait for unmanaged2