Many to many support for Taskfs

This blog on this functionality was pending for some time. Finally posting it today. Hopefully it will create the base for understanding next functionality on which work is already started.

Lets start with what is many-to-many in this context. In one-to-many taskfs, once divided, these sub-tasks are not aware of presence of other subtasks. So they can't communicate with each other. Many-to-many taskfs provides a way by which subtasks can communicate with each other. This is achieved by binding all remote resources in the directory of current session of taskfs. Each subtask will be running into one of these pseudo subdirectories (which actually are compute resources on some remote nodes). In this setup, all subtasks can assume that all other subdirectories are other subtasks, and can directly communicate with these neighboring subdirectories without asking parent about actual locations of these resources.

Now, lets see how exactly it is implemented. The trick used here is whenever session in taskfs allocate new remote resource, it will bind this new resource directory as one of the sub-directory within current session directory. This also simplifies read and write aggregations. Reads and writes on taskfs session directory will just pass that read/write to corresponding file in all subdirectories. Taskfs does not bother to remember where exactly are the remote resources as this information is automatically encoded by bindings between subdirectories and remote resource directories.

Now, lets see the example of it. Following test-run assumes that there are remote deployments of inferno which are exporting devcmd2, and client inferno instance has mounted those exported remote taskfs on /remote/. This client inferno is also assumed to export the taskfs for others to use. Following are the commands executed on client inferno host.

styxlisten -A tcp!*!5555 export /task
mount -A tcp!9.3.61.180!6666 /remote/host1
mount -A tcp!9.3.61.180!6667 /remote/host2
mount -A tcp!127.0.0.1!6666 /remote/host3

Now, following was executed on Linux shell by mounting the taskfs exported by above inferno client. $ represents Linux shell prompt.

$ 9pfuse localhost:5555 mpoint
$ tree mpoint
mpoint
`-- remote
    |-- arch [error opening dir]
    |-- clone
    |-- env [error opening dir]
    |-- fs [error opening dir]
    |-- ns [error opening dir]
    `-- status [error opening dir]
$ cd mpoint/remote
$ cat clone
0
$ cd 0
$ tree ../
../
|-- 0
|   |-- args
|   |-- ctl
|   |-- env
|   |-- ns
|   |-- status
|   |-- stderr
|   |-- stdin
|   |-- stdio
|   |-- stdout
|   `-- wait
|-- arch [error opening dir]
|-- clone
|-- env [error opening dir]
|-- fs [error opening dir]
|-- ns [error opening dir]
`-- status [error opening dir]

6 directories, 11 files

$ echo res 4 > ctl
$ tree ../
../
|-- 0
|   |-- 0
|   |   |-- args
|   |   |-- ctl
|   |   |-- env
|   |   |-- ns
|   |   |-- status
|   |   |-- stderr
|   |   |-- stdin
|   |   |-- stdio
|   |   |-- stdout
|   |   `-- wait
|   |-- 1
|   |   |-- args
|   |   |-- ctl
|   |   |-- env
|   |   |-- ns
|   |   |-- status
|   |   |-- stderr
|   |   |-- stdin
|   |   |-- stdio
|   |   |-- stdout
|   |   `-- wait
|   |-- 2
|   |   |-- args
|   |   |-- ctl
|   |   |-- env
|   |   |-- ns
|   |   |-- status
|   |   |-- stderr
|   |   |-- stdin
|   |   |-- stdio
|   |   |-- stdout
|   |   `-- wait
|   |-- 3
|   |   |-- args
|   |   |-- ctl
|   |   |-- env
|   |   |-- ns
|   |   |-- status
|   |   |-- stderr
|   |   |-- stdin
|   |   |-- stdio
|   |   |-- stdout
|   |   `-- wait
|   |-- args
|   |-- ctl
|   |-- env
|   |-- ns
|   |-- status
|   |-- stderr
|   |-- stdin
|   |-- stdio
|   |-- stdout
|   `-- wait
|-- arch [error opening dir]
|-- clone
|-- env [error opening dir]
|-- fs [error opening dir]
|-- ns [error opening dir]
`-- status [error opening dir]

10 directories, 51 files
$ cd 3
$ pwd
/home/pravin/projects/inferno/mpoint/remote/0/3
$ ls -l
total 0
--w--w--w- 1 pravin pravin 0 2010-01-14 17:12 args
-rw-rw---- 1 pravin pravin 0 2010-01-14 17:12 ctl
--w--w--w- 1 pravin pravin 0 2010-01-14 17:12 env
-rw-rw---- 1 pravin pravin 0 2010-01-14 17:12 ns
-r--r--r-- 1 pravin pravin 0 2010-01-14 17:12 status
-r--r--r-- 1 pravin pravin 0 2010-01-14 17:12 stderr
--w--w--w- 1 pravin pravin 0 2010-01-14 17:12 stdin
-rw-rw---- 1 pravin pravin 0 2010-01-14 17:12 stdio
-r--r--r-- 1 pravin pravin 0 2010-01-14 17:12 stdout
-r--r--r-- 1 pravin pravin 0 2010-01-14 17:12 wait
$ cat status
cmd/1 1 Closed /home/pravin/projects/inferno/ericvh//ericvh-brasil/ ''
$ cd ..
$ pwd
/home/pravin/projects/inferno/mpoint/remote/0
$ cat status
cmd/0 1 Closed /home/pravin/projects/inferno/ericvh//ericvh-brasil/ ''
cmd/0 1 Closed /home/pravin/inferno/host2//ericvh-brasil/ ''
cmd/0 1 Closed /home/pravin/inferno/ericvh//ericvh-brasil/ ''
cmd/1 1 Closed /home/pravin/projects/inferno/ericvh//ericvh-brasil/ ''
$ echo exec hostname > ctl
$ cat stdio
BlackPearl
inferno-test
inferno-test
BlackPearl
$ cat stderr
$ cat status
cmd/0 5 Done /home/pravin/projects/inferno/ericvh//ericvh-brasil/ hostname
cmd/0 5 Done /home/pravin/inferno/host2//ericvh-brasil/ hostname
cmd/0 5 Done /home/pravin/inferno/ericvh//ericvh-brasil/ hostname
cmd/1 5 Done /home/pravin/projects/inferno/ericvh//ericvh-brasil/ hostname
$ cd ../../..
$ pwd
/home/pravin/projects/inferno
$ fusermount -u mpoint
$ ls mpoint
$

Now, lets see what exactly happened above.

1. The taskfs was mounted with 9pfuse.


2. The directory structure of initial taskfs was shown with tree command.


3. Changed the directory in side taskfs by cd mpoint/remote


4. taskfs sesion was created by cat clone command.


5. Changed the directory to taskfs session. in this case it iscd 0


6. Directory structure of taskfs after session creation is examined. You can see that there are no subdirectories to the session directory, which indicate that there are no remote resources allocated.


7. Remote resources were allocated by echo res 4 > ctl command.


8. Directory structure of taskfs after resource allocation is examined. You can see that there are four subdirectories, one for each remote resource.


9. Changed the directory to one of the subdirectory representing remote resource, and operated on it independently. Checked the status of that resource using cat status.


10. Checked the status of taskfs session by doing cat status on session directory. You can see that it shows the status of all remote resources.


11. Executed the command on taskfs session by doing echo exec hostname > ctl


12. Checked the output, stderr and status after execution. You can see that there are two different hostnames in those 4 outputs.


13. Came out of mounted directory, and unmounted it.

This functionality of taskfs is available in revision d125d391c1.