Taskfs with one to one mapping

Taskfs is now properly working with one to one mapping between Taskfs resource and single remote node. Current implementation of Taskfs allows users to mount multiple remote nodes inside /remote as subdirectory with any name. Taskfs will find all valid remote resources, and reserve one for current job execution. Currently scheduling between remote resources is not implemented as no information about load or remote session is available. But once this information is available, making informed scheduling decision is trivial modification to this code. Also, the validity of remote resource is checked by presence of local/clone file. The revision no. of this code is 5476644bd5.

Next target for Taskfs is support for one to many mapping. This will allow single Taskfs resource to manage and utilize multiple remote resources. By this mechanism, by sending single command to ctl file in Taskfs resource, that command can be executed on multiple remote resources simultaneously, and output will be automatically aggregated back into single stdout file of Taskfs resource.

Following is the demo of Taskfs with one to one mapping.

; cd /task/remote
; cat clone
cat: cannot open clone: Remote resources not present at [/remote]
; mount -A tcp!127.0.0.1!6666 /remote/host1
; cat clone
0; cd 0
; echo exec pwd > ctl
; cat stdio
/home/pravin/projects/inferno/ericvh/ericvh-brasil
; cd ..
; mount -A tcp!127.0.0.1!6667 /remote/host2
; cat clone
0; cd 0
; echo exec pwd > ctl
; cat stdio
/usr/share/lxr/source/inferno/ericvh-brasil
; 

This demo shows that the output of two pwd commands is different. This is because taskfs selected host1 node when only one remote node was available. But it selected host2 node when there were two nodes available.

taskfs : executing commands on remote host

The first working version of taskfs is complete. This version allows user to execute host command on host which is running remote inferno instance. And these commands are executed using filesystem interface. Revision 49060921b4 of inferno-rtasks implement these modifications.

Here are the steps about how to run and test it.

Firstly, you will need to run one remote instance of inferno with devcmd2 support. ericvh-brasil has this support. here are the commands used for this remote instance.

$ pwd
/home/pravin/projects/inferno/ericvh/ericvh-brasil/
$ ./Linux/386/bin/emu -r `pwd`
; styxlisten -A tcp!*!6666 export /cmd2

Now, following are the commands used with inferno-rtasks.

$ pwd
/usr/share/lxr/source/inferno/inferno-rtasks/
$ ./Linux/386/bin/emu -r `pwd`
; cd /task/remote
; cat clone
cat: cannot open clone: '/remote/local' does not exist
; mount -A tcp!127.0.0.1!6666 /remote
; cat clone
0;
; cd 0
; ls
args
ctl
env
ns
status
stderr
stdin
stdio
stdout
wait
; echo exec pwd > ctl
; cat stdio
/home/pravin/projects/inferno/ericvh/ericvh-brasil
; cat status
cmd/0 3 Done /home/pravin/projects/inferno/ericvh//ericvh-brasil/ pwd

You can see in the output above that command was executed by ericvh-brasil, which is remote inferno instance and not by inferno-rtasks.

Now, few comments about implementation.
The read call on /taskfs/remote/clone is modified to first perform a read on /remote/local/clone and save the path of reserved resource in REMOTE_RESOURCE_PATH string.

Also, the write call on /taskfs/remote/n/ctl are captured and then written to /remote/local/r/ctl. Similarly, the read on stdio,stderr and status are pulled from their same remote counterpart at /remote/local/r/.

One major flow in this implementation is the way handle to remote resource is stored. In this implementation, this handle is stored in REMOTE_RESOURCE_PATH string which contain absolute path. With help of namec function from emu/port/chan.c, Channel is created separately with each above mentioned remote files.

Correct way to implement this would be to store the handle to the Channel pointing to the /remote/local/r/, and use it to locate all the files without depending on absolute paths. Next version of taskfs will attempt to do this.

Pitfall : Executing wrong emu

I have wasted 5 hours today in executing wrong emu image and wondering why my changes are not producing any results. This was such a stupid and unexpected mistake that I never considered it.

I have copied the source code from it's initial compilation location to the new location. I did modified the script which starts the emu to point at correct emu image, and also use correct inferno root fs. But I forgot to change the root path variable in mkconfig. All my changes were affecting emu image in old location, while I was running emu image in new location for testing purpose.

So, whenever you copy inferno source-code base, remember to adjust following paths.

1. The root path variable in mkconfig file. This is absolute path, which means you have to edit it.


2. $PATH variable that you modify to include Linux/386/bin so that compilation tools like mk can be found.


3. For running, you need path to the Linux/bin/emu generated after compilation.


4. For running, you also need a path to the inferno_root filesystem as command line parameter to emu

Remote mounting and using devcmd2 filesystem

This blog documents how to mount and use devcmd2 filesystem remotely.

For this, first step is to bind and export devcmd2. In my case, devcmd2 is automatically mounted at /cmd2 location. You can read previous blog for how to do this manually. Now for exporting this filesystem, use following command.

styxlisten -A tcp!*!6666 export /cmd2

And you are done!!!

Any inferno system that wants to mount this filesystem can do it by following command.

mount -A tcp!127.0.0.1!6666 /mpoint

This command mounts the filesystem exported at address tcp!122.0.0.1!6666 on location /mpoint. -A disables any authentication as we are not using any.

Now lets execute commands on remote inferno filesystem.

; cd mpoint
; ls
local
; cd local
; ls
arch clone env fs ns status
; cat clone
0
; cd 0
; ls
args ctl env ns status stderr stdin stdout wait
; echo exec hostname > ctl
; cat stdio
BlackPearl
; 

Wow!!! We just executed remote command using Inferno.

This mounted filesystem can be unmounted by unmount command.

unmount /mpoint

Next step is to mount this inferno devcmd2 from Linux. For this you will need to install 9mount, which can be done as follows for debian/Ubuntu based systems.

sudo apt-get install 9mount

And, now use following command to mount it.

9mount 'tcp!127.0.0.1!6666' mpoint

You can unmount this filesystem by

9umount mpoint

command. But before unmounting it, lets try and execute some remote commands using this mount point.

$ cd mpoint
$ cd local
$ ls
arch clone env fs ns status
$ cat clone
0
$ cd 0
$ ls
args ctl env ns status stderr stdin stdio stdout wait
$ echo exec hostname > ctl
bash: ctl: Unknown error 526
$ cat stdio
$ cat stderr
$ cat status
cmd/0 0 Closed /usr/share/lxr/source/inferno//ericvh-brasil/ ''

Well, this is bad news as I am not able to execute commands on inferno from Linux. I also tested with devcmd filesystem, but it gave same error.

bash: ctl: Unknown error 526

Somehow, ctl is not willing to accept commands when sent from sent from mount point on Linux. I don't know the reason behind it, but it seems some more work is needed on 9mount.

Executing host commands from Inferno using cmd filesystem interface ( '#C' )

This blog is about using ability of inferno to execute commands on host OS using filesystem interface. As this project plans to use Inferno as middleware for distributing jobs transparently this interface is important.

The easiest way to execute host command on inferno is by using os command.

; os ls /

Above is the example of executing ls / on host OS.

For using filesystem interface, you can use devcmd or devcmd2 synthetic filesystems.
devcmd2 is extension of devcmd with few improvements. We will stick to devcmd
for purpose of this blog.

You can go into this filesystem by either doing cd or by binding it.

; cd '#C'

This filesystem can be binded to your namespace by following command.

bind '#C' /cmd
cd /cmd

Now, lets see how we can execute commands using this filesystm.

; cd cmd
; ls
clone
; cat clone
0
; cd 0
; ls
ctl
data
status
stderr
wait
; echo exec fortune > ctl
; cat data
Hope that the day after you die is a nice day.
;

What has happened above is we created new resource by catting clone. This resource was 0. So, we cd to 0 for using this resource. We execute the fortune command by echoing it into ctl file. You can execute any command here. After this you can cat the data file for command output and stderr for any errors.

For devcmd2 output will be in stdio file instead of data.

If you don't have fortune command on your system, then go ahead and install it using apt-get install fortune. It is brilliant way to waste time :-).

Setting up lxr for Inferno

Understanding and studying the code is much easier if you have source code cross reference set up. There are public repositories for most well known operating systems. You can find them at http://fxr.watson.org/.

Unfortunately, Inferno source code is not hosted here. So, I had to go through the endeavor myself. There are few tutorial out there which gives simple steps for installing and configuring lxr for Linux source code. here are few examples that I referred.
http://tread.wordpress.com/2007/08/25/howto-setup-lxr-on-linux-ubuntu-feisty/
http://orangespike.ca/content/ubuntu-710-lxr-linux-cross-referencing-howto

But I could not find anything which tells how to do it for generic code. So, here I will describe the steps I took for configuring generic code in lxr. Inferno code is taken as example here. It should be easy to replace Inferno with other system code in following steps.

sudo apt-get install apache2 libapache2-mod-perl2
sudo apt-get install libcgi-simple-perl libcgi-ajax-perl libhtml-parser-perl libtemplate-perl

Install LXR.

sudo apt-get install lxr

If possible, install glimpse also. It provides free-text search capabilities. apt-get did not worked for this one in my case. And I did not fight to get it installed also. If following command does not work for you then you can try to get it from http://webglimpse.net/.

sudo apt-get install glimpse

Add the following lines to /etc/apache2/httpd.conf at the end:

# Linux Cross Reference Stuff
Alias /lxr /usr/share/lxr

Options All
AllowOverride All

Create a file /usr/share/lxr/http/.htaccess which contains:

SetHandler cgi-script

Restart Apache

/etc/init.d/apache2 restart

Create /usr/share/lxr/source if it doesn’t exist.

sudo mkdir -p /usr/share/lxr/source
cd /usr/share/lxr/source
sudo mkdir inferno
hg clone https://ericvh-brasil.googlecode.com/hg/ ericvh-brasil
echo "inferno" > /usr/share/lxr/source/versions
sudo ln -s /usr/share/lxr/source/inferno /usr/share/lxr/source/defversion

I performed following steps to get ownership of these file, so that I can edit them directly. You may not have to do these steps. Just make sure that the code directory is accessible to everyone for read and browse.

cd /usr/share/lxr/source
chown -R pravin.pravin inferno
chmod -R +x inferno
chmod 777 /usr/share/lxr/source/inferno/fileidx

Now comes the most important step. You need to modify /usr/share/lxr/http/lxr.conf
to suit your source-code. This file is set-up for linux source-code organization, so, does not work out of the blue for other source-codes. Here is the link to my lxr.conf file. Lines which are modified are highlighted. Most important is to change srcrootname to the root directory of your code. In my case, this entry was modified to

srcrootname: ericvh-brasil

Once you configure /usr/share/lxr/http/lxr.conf, perform following steps.

cd /usr/share/lxr/source/inferno
genxref

Now, your lxr is ready to browse.
Visit http://localhost/lxr/http/source/ericvh-brasil/ to start browsing the code.

If you have system other than Linux or inferno, just try and replace inferno and ericvh-brasil with your version name and source-code root directory and hopefully it will work.

I am quite sure that, this is not the most optimal way to get lxr working. Few of the above steps can be avoided. But, above steps worked for me. Lets hope it helps others as well.

Adding new filesystem to Inferno

I have uploaded the codebase that I will be using on google-code at
http://code.google.com/p/inferno-rtasks/.
Feel free to download and play with it.

My next objective is to add new synthetic file-system similar to /net or /prog to Inferno. Here is the approach that helped me.


First step was to find out where are most of the synthetic fie-systems are residing.
The location was emu/port/. New synthetic filesystem was created by copying
emu/port/devcmd.c with new name emu/port/devtask.c. This file has
one structure Dev cmddevtab at bottom of the file which needs modifications.
I modified this structure as follows

Dev cmddevtab = {
 'C',
 "cmd",

 cmdinit,

Above structure was modified as

Dev taskdevtab = {
 'T',
 "task",

 cmdinit,

You also need to tell compilation process for including this file into compilation path.
This can be done by modifying emu/Linux/emu file which is configuration file
controlling what all should be included in emu image. This file has sections for
dev, lib, link, mod, port, code, init and root. As my addition is supposed to be
synthetic filesystem, I added following line to the dev section of this file.

      task

With this modification, you can recompile the kernel and execute the new image.
To test if this new filesystem is present, you can use following command

; ls '#T'
#T/cmd

The output #T/cmd proves that this file-system is in kernel.


Now, lets concentrate on auto mounting this synthetic filesystem at boot time.
This involves few more steps.

First, we need place to mount this filesystem. Lets edit emu/Linux/emu
again. This time, we add following line in root section.

     /task   /

Refer other entries for the syntax expected in this field. This will create a
placeholder directory /task were we can mount out filesystem.

Next step is to modify initialization script so that it will actually mount synthetic filesystem.
The initialization script can be found at appl/cmd/emuinit.b.
Locate the init() function which is responsible for initial binding of the
filesystem with their mount-points. Add following line to this function after
other filesystems are mounted using similar calls.

      sys-bind ("#T", "/task", sys->MBEFORE) ;

I repeated the same modification in appl/cmd/brasil.b as in few cases brasil.b is used
as initialization script.

After this, you should be able to recompile the kernel and test using following command
within inferno.

; ls /task
/task/cmd

Here, the output /task/cmd shows that our filesystem is properly attached to the
root filesystem :-)