Using Blender with openMosix

2004-04-09 19:05:41

What you will need for this bite-sized example

• 2 (or more) networked machines running dyne:bolic 1.3
• storage space for the blender models, renders, etc.
• a .blend scene ready to batch render
• Marc O. Gloor's render script

Multiple d:b's

In addition to the obvious solution of burning multiple CD's, you can run d:b from a hard drive by 'docking' dynebolic. Simply copy the 'dyne' directory from the CD to the root directory of a readable hard drive partition. Only read access is needed, so any filesystem d:b can read will work. For example, on a Windows machine, copy the dyne directory to C:, creating a C:\dyne. In a linux environment copy straight to /, creating a /dyne. Now reboot with the d:b CD and it will locate the new docked install and eject the CD. Repeat the process on each machine that will form the cluster. Assuming the machines are connected to the same network (the faster the better) the cluster will be configured by the time you finish booting. Remember, too, that you can add nodes at any time, just by booting another d:b on the network.

Storage space

You will need to have at least 60 megabytes of free space for the scene elements, rendered animation frames and a few other odds and ends. In d:b, the local hard drive partitions are known as /vol/hd#, where # is the number representing the order it which it was encountered. For example /vol/hd1 would be d:b's name for the typical Windows C: drive, or /dev/hda1 under most Linux flavors.

A blender sceen to render

We'll use the time honored blacksmith scene from download.blender.org . Pick the .tgz version. Explode the downloaded file with:

tar xvzf blacksmith.tgz

This will create a 'blacksmith' directory. We will need to make a few minor tweaks to set things up to render frames as expected for this example. Inside the blacksmith directory, create a 'rendered' directory. This will be used to hold the animation frames we are going to render. Now, launch Blender and load the blacksmith.blend. Make sure you have scene 04_06 selected. Bring up the render settings panel to set these options:

• frame size, pick 640x480
• file type, pick jpeg
• output directory, our 'rendered'
• turn on file extentions

This screenshot highlights the relevant changes. Check out one of the excellent Blender tutorials on the web to build an understand of all of these options. Save your work as a new .blend file, so you can go back to the original if needed. Call it blacksmith-jpeg.blend. You can exit Blender at this point.

The render script

Blender needs a little bit of coaxing to use the cluster resources. OpenMosix works by moving processes from a heavily loaded machine to a more lightly loaded machine. Blender runs as a single process, so openMosix can't do much, other than move the one process to another machine. If you split the work to be done into 2 or more process, openMosix will handle the rest. You should split the work into at least as many processes as you have nodes on the network. You could split the workload manually, but for this example, we will use a pre-made script to do the splitting. Download Marc O. Gloor's excellent render script (local mirror - render.gz). Ungzip the script and make it executable with:

You can store this script anywhere, but If you copy it into your 'blacksmith' directory, it will be in the search path for this example, since by default in dyne:bolic's zsh shell the current directory is in the path.

openMosix basics

There are a lot of options available in openMosix but dyne:bolic is preconfigured to use auto-discovery, so you don't need to do much. From an xterm, cd to /usr/mosix and these commands are available:

openmosix status

This will verify that you have a cluster, and show the nodes that are available. Here is a sample output from a 4 node system:

[d:b] /usr/mosix #openmosix status
openMosix: No map-file found.
openMosix: Falling back to autodiscovery mode using /usr/mosix/sbin/omdiscd
This is OpenMosix node #261
Network protocol: 2 (AF_INET)
OpenMosix range   261-261   begins at 10.0.1.5
OpenMosix range   407-407   begins at 10.0.1.151
OpenMosix range   266-266   begins at 10.0.1.10
OpenMosix range   258-258   begins at 10.0.1.2
Total configured: 4

There are other useful options. Display a complete list by executing openmosix with no arguments.

bin/mtop

mtop is an openMosix aware top command. Note the N# column indicating what node the process is running on.

 10:37pm  up 21:50,  0 users,  load average: 0.32, 0.15, 0.05
45 processes: 44 sleeping, 1 running, 0 zombie, 0 stopped
CPU states: 160.8% user,  7.1% system,  0.0% nice,  0.0% idle
Mem:  254936K av, 167160K used,  87776K free,      0K shrd,  17812K buff
Swap: 642560K av,   5344K used, 637216K free                 83440K cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT N# %CPU %MEM   TIME COMMAND
 1169 root      17   0 39328  38M   676 S    407 85.5 15.4   0:17 blender
 1172 root      14   0 37700  36M  1624 S    266 42.6 14.7   0:11 blender
 1166 root      18   0 42436  41M  1656 S    266 38.3 16.6   0:11 blender
 1175 root      14   0 26592  25M  3408 S     0  0.7 10.4   0:01 blender
 1181 root      18   0   848  848   692 R     0  0.3  0.3   0:00 mtop
 1098 root      10   0  2208 2208  1576 S     0  0.1  0.8   0:24 sshd
    1 root       9   0   476  476   420 S     0  0.0  0.1   0:06 init
...

bin/mps

Just as mtop is to top, so is mps to ps.

[d:b] /usr/mosix #bin/mps
  PID TTY NODE STAT TIME COMMAND
 1100  ?     0 S    0:00 -zsh
 1146  ?     0 S    0:00 xterm
 1147  ?     0 S    0:00 zsh
 1159  ?     0 S    0:00 /bin/sh render blacksmith-jpeg.blend 1 235 4
 1166  ?   266 S    2:40 (blender)
 1169  ?   266 S    5:51 (blender)
 1172  ?   258 S    4:16 (blender)
 1175  ?   258 S    1:48 (blender)
 1199  ?     0 R    0:00 bin/mps

It is interesting to note in this display that openMosix has moved 4 jobs to just 2 nodes, and the node that started the processes is not running any of the jobs. The machines on the cluster were a mixture of older slow machines, and newer faster machines. openMosix handles balancing the load it is given. In cases like this, you might squeeze more out of your cluster by increasing the number of jobs running.

bin/mosctrl

mosctrl offers even more detail information and control for your cluster.

[d:b] /usr/mosix #bin/mosctl
Usage: bin/mosctl command
Available commands: stay/nostay, lstay/nolstay, block/noblock, quiet/noquiet,
                    mfs/nomfs, expel, bring, gettune, getdecay,
                    whois [mosix_no|IP-address|hostname],
                    getload [node-number], getspeed [node-number],
                    status [node-number], getmem [node-number],
                    getfree [node-number], getutil [node-number],
                    getyard, setyard [node-type], setspeed speed,
                    setdecay interval(seconds) slow(/1000) fast(/1000),
                    isup [node-number]
zsh: 1208 exit 1     bin/mosctl
[d:b] /usr/mosix #bin/mosctl getmem 266
free memory = 217960448 of 234815488
[d:b] /usr/mosix #bin/mosctl getmem 261
free memory = 247595008 of 268423168
[d:b] /usr/mosix #bin/mosctl getspeed 266
speed=26842.
[d:b] /usr/mosix #bin/mosctl getspeed 261
speed=8240.

The last two commands shown offer a clue why node 266 gets more work than 261.

Ready to Render

In an xterm, cd to your 'blacksmith' directory. Issue the render command as shown. Change the final 4 to match the number of nodes in your cluster.

[d:b] ../blacksmith #render blacksmith-jpeg.blend 1 235 4
 
Rendering 234 images from blacksmith-jpeg.blend on 4 nodes.
Tasks forked, network rendering in progress.
Job started at: 10-04-04 22:35:46
Please wait while rendering...

Now comes the waiting, which will depend on the number of nodes and the speed and memory of each node. While the jobs run, try out the mtop, mps and mosctrl commands in another xterm to get the feel for what is happening and what the capabilities of your cluster are. Eventually you will be presented with the completion message.

Rendering successfully finished.
Job ended at: 10-04-04 22:56:15

Follow up with a ls rendered to list your output files. Assuming you find 230 some .jpg files, congratulations on your cluster render!

Building a movie from jpeg files

Now that everything is rendered, you might want to see this animation, right? d:b has the tools for that, too. Just cd into your rendered directory and issue this:

In just a few moments, you should have an 'output.avi' file, ready to play. For more info on this, and similar capabilities in mencoder check in the MPlayer manual under Encoding with MEncoder .

Now you've got power. Go create something wonderful!

Article 12 was last updated 2004-08-14 12:07:28