inferno programmer's notebook

lab 105 - automount

noreply@blogger.com (Eric Van Hensbergen) — Thu, 05 Nov 2009 01:11:00 +0000

NAME

lab 105 - automount

NOTES

A small modification to mntgen yields an automounter which automatically mounts a server based on path name. This is a fairly crude proof of concept with hardwired port numbers and top-level mount path -- but it would be fairly easy to make it a bit more robust. Essentially, I just added a mount command to the code which dynamically adds the directory node to the mntgen tree on reference. Then I made the file system handling call its own thread so that the mount could reference the synthetic without locking up the original single threaded synthetic file server. One problem is that unknown servers take some time to respond with file note found, which is less than desirable. Fixing this and other annoyances is left as an exercise for the reader.

EXAMPLE

% ls /amnt
% mount {automnt} /amnt
% ls /amnt
% ls /amnt/localhost/usr/inferno
/amnt/localhost/usr/inferno/9cpu
/amnt/localhost/usr/inferno/README
/amnt/localhost/usr/inferno/charon
/amnt/localhost/usr/inferno/lib
/amnt/localhost/usr/inferno/namespace
/amnt/localhost/usr/inferno/tmp
%

FILES

105/automnt.b

lab 104 - ducts: bi-directional mount channels

noreply@blogger.com (Eric Van Hensbergen) — Fri, 07 Aug 2009 01:08:00 +0000

NAME

lab 104 - ducts: bi-directional mount channels

NOTES

Hi there, I want to talk to you about ducts. Are your ducts old fashion, out of date?

In this lab, I'll walk through a little facility I built to allow me to have bi-directional mounts over a single file descriptor channel. The nature of the way I access my Blue Gene environment requires me to do an awful lot over a single communications channel. I wanted the ability to both access the target node's file system while simultaneously exporting mine, while only using a single file descriptor.

To facilitate this, I wrote a little 9P-knowledgeable multiplexor which directs T-msgs to an export thread and R-msgs to a mount thread while multiplexing the responses from those threads back onto the same file descriptor.

The initial implementation of the base mechanism was relatively simple until I had to start considering how to clean things up when I started unmounting directories on one side or both. The liberal use of sys->pipe throughout the code meant it was somewhat difficult to detect when things were finished. I ended up adding a signaling packet to the protocol to notify the remote Duct when I had closed my mount, and then implemented a channel based monitor infrastructure so I could track what was still running and appropriately shut down the various pieces as things unmounted (it is possible to unmount one side without unmounting the other).

The interface is fairly simple, if using the module interface you spawn an uplink, specifying an input file descriptor and an output file descriptor for the connection (which may be the same FD), the path you wish to export and the mount point for where you want to place the imported file hierarchy.

If you are running it from the command line, it uses stdin and stdout for the input and output FDs and you only need specify the export path and mount point.

EXAMPLES

host1% listen -A tcp!*!1127 duct /usr/ericvh /n/foo
host2% dial -A tcp!host1!1127 duct /usr/inferno /n/bar

FILES

104/duct.m
104/duct.b

lab 103 - python content assist

noreply@blogger.com (caerwyn) — Sun, 28 Jun 2009 16:08:00 +0000

NAME

lab 103 - python content assist

NOTES

Building on lab 102 I've started to write an acme client for python content-assist in python. I read the code for python's IDLE editor and pulled some pieces out of that to help with my implementation. The AutoComplete.py and CallTip.py modules are copied out of idlelib and reduced to just the functions I needed.

Just as in lab 102 the acme client is implemented using pyxp to connect to the acme file system.

In an acme editor window execute PyAssist. Start typing some python. After typing a dot the client will display available attributes for the module. Continue typing and it will reduce the list to those names with a matching prefix. Press the Tab key to complete a name or prefix depending on whether there is one or more names.

After typing an open parenthesis the client will show the call tip for the function.

The client implements an Import command so that more python modules can be added to the namespace. PyAssist doesn't attempt to parse the edited script. Idlelib includes code that could help to do this. But for now you need to pass a module name as argument to the Import command.

A next step for python integration to acme is to implement a debugger. Python comes with libraries for this too, so a debugger similar to the acme debugger adeb should be a future lab. I might then run edited scripts in a sub shell and inspect that process' namespace similar to the IDLE implementation, so that content-assist adapts to the script being edited.

FILES

inferno-lab/103

lab 102 - python acme client

noreply@blogger.com (caerwyn) — Wed, 24 Jun 2009 00:58:00 +0000

NAME

lab 102 - python acme client

NOTES

A recent post to 9phackers announced Pyxp, another implementation of Styx in Python.

I immediately downloaded Pyxp and tried it out. I had no trouble using it so I started thinking about python clients I could write. Python is still new to me so writing a styx client was an excuse to get more practice.

I started with an acme client. I translated the acmewin limbo module I use for most acme-sac clients to python. Below is a simple example opening a new acme window and doing some operations on it.

from acmewin import Acmewin

win = Acmewin()
win.writebody("hello, world!\n\n\n")
win.tagwrite("Hello")
win.writebody("goodbye")
win.replace("/goodbye/", "GOODBYE")
win.select(",")
win.show()

Remember to export the namespace before trying it out.

% styxlisten -A 'tcp!*!localhost' export /
% textclient.py

I recently saw on Hacker News a repost of Peter Norvig's spelling corrector. I thought this would make an easy first trial of my python acmewin code. I implemented a client to offer spelling suggestions to an editor window. It works somewhat like my earlier acme content assist code. This client opens the event file of another window it is assisting and writes text out to its own window. In this case it offers a suggested spelling for the word currently being typed.

Here's the implementation. Note that this is single threaded and it is not reading the event file of the second window. I haven't gotten that far in the Python book.

#!/dis/python26

import sys
from acmewin import Acmewin
import spell

win = Acmewin(int(sys.argv[1]))
outwin = Acmewin()

while True:
    (c1, c2, q0, q2, flag, nr, r) = win.getevent()
    if c2 in "xX":
        if flag & 2:
            win.getevent()
        if flag & 8:
            win.getevent()
            win.getevent()
        win.writeevent(c1, c2, q0, q2)
        if c2 == "x" and r == "Del":
            outwin.delete()
            break
    if c1 == "K" and c2 == "I":
        ch = r[0]
        if ch in " \t\r\n":
            outwin.replace(",", "")
            continue
        while q0 >= 0 and not (ch in " \t\r\n"):
            sss = win.read(q0, q0+1)
            if not sss:
                # print("empty sss %d" % q0)
                sss = " "
            ch = sss[0]
            q0 -= 1
        if q0 < 0 and not(ch in " \t\r\n"):
            q0 = 0
        else:
            q0 += 2
        ss = win.read(q0,q2)
        lastcorrect = spell.correct(ss)
        outwin.replace(",", lastcorrect)

To run this we need to know the id of the window we are assisting, so we need a wrapper to send the $acmewin environment variable as an arg to the script. For that I have a script called SpellAssist.

#!/dis/sh
$home/python/assist.py $acmewin

Now that I have a simple assist-like client working I'd like to develop it further. I'd like to try having content assist for python inside acme. It should be possible to adapt the python code that implements the IDLE editor to this purpose.

FILES

inferno-lab/102
big.txt the large text file used to train the spelling corrector. Note that the path is hardcoded in spell.py and should be changed locally.

lab 101 - limbo B+ tree

noreply@blogger.com (caerwyn) — Tue, 16 Jun 2009 01:19:00 +0000

NAME

lab 101 - limbo B+ tree

NOTES

This lab is an implementation of a B+ tree. It is code from a much older lab that was overly complicated and buggy. I pulled the B+ tree code out, tried to fix the bugs and clean it up. The interface is roughly the same as dbm(2). Here is a synopsis.

include "btree.m";
btreem := load Btreem Btreem->PATH;
Datum, Btree: import Btreem;

Btree: adt {
 create: fn(file: string, perm: int): ref Btree;
 open: fn(file: string, flags: int): ref Btree;
 
 fetch: fn(b: self ref Btree, key: Datum): Datum;
 delete: fn(b: self ref Btree, key: Datum): int;
 store: fn(b: self ref Btree, key: Datum, val: Datum):int;

 firstkey: fn(b: self ref Btree): Datum;
 nextkey: fn(b: self ref Btree, key: Datum): Datum;

 flush: fn(b: self ref Btree);
 close: fn(b: self ref Btree);
};

init: fn();

Like Dbm the keys and values are stored as arrays of bytes, and being a B+tree the values are stored only in the leaf nodes. The maximum key and val size is 255 each. The block size is 8192, so the maximum leaf node size is 515 making the minimum branching factor 15. The maximum number of records in an internal nodes assuming an int as key is 630.

The delete is not fully implemented; it doesn't merge nodes.

Here is some example code listing the full contents of a btree.

sys := load Sys Sys->PATH;
btreem := load Btreem Btreem->PATH;
Datum, Btree: import btreem;

btreem->init();
bt := Btree.open("index.bt", Sys->ORDWR);
for(key := bt.firstkey(); key != nil; key = bt.nextkey(key)){
 v := bt.fetch(key);
 sys->print("%s %s\n", string key, string v);
}

Here is a rough comparison of btree vs. dbm. This simple test is more of a sanity check that the btree doesn't do anything horribly wrong in its implementation (which it did originally, by making a syscall to get the daytime for every block it tried to get).

% awk '{print $1, NR}' < /lib/words | tr -d 'cr' > t1

% >index.bt
% >dbm.pag
% >dbm.dir

% time sh -c 'btree/store < t1' 
0l 4.172r 4.172t

% time sh -c 'dbm/store dbm < t1'
0l 35.25r 35.25t

% ls -l dbm.* index.bt
--rw-rw---- M 6 xcs0998 XCS0998    8192 Jun 04 12:38 dbm.dir
--rw-rw---- M 6 xcs0998 XCS0998 1048576 Jun 04 12:38 dbm.pag
--rw-rw---- M 6 xcs0998 XCS0998 770048 Jun 04 12:36 index.bt

% time sh -c 'btree/list > /dev/null'
0l 5.187r 5.187t

% time sh -c 'dbm/list dbm > /dev/null'
0l 9.438r 9.438t

FILES

inferno-lab/101

lab 100 - limbo tags

noreply@blogger.com (caerwyn) — Sun, 07 Jun 2009 18:46:00 +0000

NAME

lab 100 - limbo tags

NOTES

I've been playing more with exuberant ctags. It's possible to make ctags recognize limbo source code using regular expressions to identify symbols we want tagged.

Here's a command I defined called ltags. It will search for functions, assuming the function name is not preceded by spaces, and, more reliably, find adt and module definitions:

#!/dis/sh
os -t ( /n/D/ctags57/ctags.exe  -n 
 '--tag-relative=yes'
 '--langdef=limbo' 
 '--langmap=limbo:.b.m' 
 '--regex-limbo=/^([a-zA-Z][a-zA-Z0-9]+)\(.*\)/\1/m/' 
 '--regex-limbo=/([a-zA-Z][a-zA-Z0-9]+) *: *adt */\1/t/'  
 '--regex-limbo=/([a-zA-Z][a-zA-Z0-9]+) *: *module */\1/c/'  
 $* )

Try running it under /, and use relative path names:

 
% cd /
% ltags -R appl module

And open the acme client

Ctag /tags

I added a command "Tag" to the Ctag program that will find and print all matching symbols to the name given as argument. Mouse-chording makes this nice to use. Right-clicking on symbol names will open the source file at that location. Browsing the whole limbo source hierarchy is then made quite easy.

FILES

inferno-lab/100

lab 99 - catching up with inferno-os

noreply@blogger.com (caerwyn) — Sat, 30 May 2009 15:35:00 +0000

NAME

lab 99 - catching up with inferno-os

NOTES

I had fallen way behind in keeping acme-sac code up to date with inferno-os. I tried to correct that this week by merging changes and adding files from inferno-os.

I wrote a script, part of this labs files, to check the differences between inferno-os and acme-sac. I'm hoping by using this script I'll stay up to date with inferno-os as changes are made. The script compares sub-trees, such as the limbo source code hierarchy, the C source code, the manual pages, and the /lib directory. It prints out commands to diff the files or copy the file over.

% inf
inferno synchronization:
  appl    show changed limbo files
  sys     show changed C source files
  man     show changed man pages
  lib     show changed /lib files
  cmd     show changed sh scripts below /dis
  update  refresh file tree info for local and remote
  adiff   adiff selected file with inferno-os equivalent
  diff    diff selected file with inferno-os equivalent
  missing show files missing from local tree
  added   show files added to local tree
% inf update
% inf sys
diff -u /sys/emu/port/win-x11a.c /n/D/inferno-os/emu/port/win-x11a.c
cp /n/D/inferno-os/emu/port/win-x11a.c /sys/emu/port/win-x11a.c

%

There is a short list of directories I did not synchronize because I know every file has changed; I consider these a fork, though the changes to charon are not dramatic,

/appl/acme/  /appl/charon/  /acme/

There is a list of directories acme-sac has removed from the inferno-os hierarchy:

/FreeBSD/  /libbio/     /utils/
/Hp/       /libdynld/   /appl/alphabet/
/Inferno/  /libkern/    /appl/collab/
/Irix/     /liblogfs/   /appl/demo/
/NetBSD/   /libnandfs/  /appl/ebook/
/OpenBSD/  /libprefab/  /appl/spree/
/Plan9/    /libtk/      /appl/tiny/
/asm/      /tools/      /appl/wm/

And finally there is a sub-tree we need to actually synchronize:

Linux    appl/grid lib       libkeyring  libmp   module
Nt       appl/lib  lib9      libmath     libsec
appl     appl/math libdraw   libmemdraw  include
appl/cmd emu       libinterp libmemlayer man

After comparing these sub-trees I built a list of files that were intentionally removed and files locally changed. Any file found "missing", where inferno-os added new functionality, was copied over. Files that were not marked as locally changed but nevertheless contained changes were merged, usually overwritten with the latest from inferno-os. There is also a list of files added to acme-sac apart from anything below /acme.

I found it useful to use the side-by-side diff functionality from lab 95. I added some simple interactive merge functionality that worked if you start from the end of the file and stepped backwards through the diffs using a PrevDiff command on the adiff output and calling the command Merge to apply the currently highlighted change. The implementation of this Merge command was pretty simple because it involved just copying the selected text from the source file over to the target. Given acme's filesystem interface this is literally a file copy,

cp /mnt/acme/$b/rdsel /mnt/acme/$a/wrsel

The PrevDiff commands does the work of setting up the selected texts correctly, properly handling the case of added, changed, or deleted text.

The list of files in the changed list should be reduced. Some of these are changes that could be pushed to inferno-os. E.g., diff adds the -u flag, os has changes from rog implementing host file name translation. Some files will need to be continually merged, such as the man page indices.

The list of files and directories removed are mostly just functionality either specific to running inferno as a native os, or the Tk functionality. There shouldn't be anything there people would really miss in an acme-sac environment. Though if there is let me know.

FILES

inferno-lab/99

lab 98 - acme Ctag

noreply@blogger.com (caerwyn) — Tue, 19 May 2009 02:25:00 +0000

NAME

lab 98 - acme Ctag

NOTES

Ctag is a new client for acme that reads the tags file generated by ctags. In particular I worked with Exuberant Ctags on windows. While working on acme content-assist in lab 94 I thought a next possible step was using ctags for code completion. So in writing this program I wanted to explore using exuberant ctags by doing something easier, that is by providing assistance with code navigation.

Exuberant ctags understands many languages, but for the current effort I tried it on Java and Python.

For example, I generated tags for the python standard libraries,

cd:/Python30
ctags -R --excmd=number --exclude=test

Then run the client on the tags file,

Ctag /n/D/Python30/tags

What you start with is a blank window. In the tagline is a File command. Give it an argument of a filename which it will grep for in the ctags file.

File mailbox

It will then show all classes and methods within matching files. You can right-click on a method name to open the file at the line where the method is defined.

Ctag gives a top down view of a project. It is like the Outline view of some IDEs. A next step would be to implement the reverse direction, but probably as part of the content assist.

FILES

inferno-lab/98

lab 97 - acme Navigator

noreply@blogger.com (caerwyn) — Tue, 19 May 2009 01:20:00 +0000

NAME

lab 97 - acme Navigator

NOTES

I created a simple directory browser for acme. Called Navigator, it opens directories in the same window. This makes descending deep but unfamiliar directory hierarchies easier because it avoids the proliferation of windows that clutter the desktop.

The client understands the command Pin for creating a standard acme directory window for the current directory (to pin the current directory to the desktop). This is an example for a rather simple client for acme.

FILES

inferno-lab/97

lab 96 - acme color schemes

noreply@blogger.com (caerwyn) — Sun, 17 May 2009 23:59:00 +0000

NAME

lab 96 - acme color schemes

NOTES

The inferno version of acme always contained some code to configure the acme color scheme, but I'm not sure if it always, if ever worked.

I modified the code a little in acme-sac to make the color configuration work. And I came up with a few new schemes.

The schemes are checked into acme-sac under /acme/color. To use a different scheme add a line like this to your $home/lib/profile

run /acme/color/evening

Running the "evening" scheme will make acme look like this.

The configuration files sets a few environment variables of the following form:

acme-fg-text-0='#000000'

This sets the foreground text color in the body text to black. It's possible to mix two colors, as for the standard acme background.

acme-bg-text-0='#FFFFAA/#FFFFFF'

Another simple scheme is "bw" for black and white.

There is also a file /acme/color/standard for the default acme color scheme which can be copied tweaked.

lab 95 - acme side-by-side diff

noreply@blogger.com (caerwyn) — Sun, 17 May 2009 02:25:00 +0000

NAME

lab 95 - acme side-by-side diff

NOTES

Here's a script I wrote to help do side-by-side diffs in Acme. Run adiff as normal inside acme. And then in the adiff output window type the command NextDiff. Executing this command will step through each diff in the adiff output and highlight the changed region in each file.

The script will highlight each line, avoiding the cursor jump to highlighted regions as happens by plumbing or right-clicking on a file pattern. Though the cursor jump will occur the first time the diffed files are opened, for subsequent execution of NextDiff for diffs within the same files the cursor will remain over the NextDiff command and the highlighted regions will change with file scrolling to show at least part of the changed regions.

When the files first open you'll still need to manually arrange the files side by side. There is no acme API for window placement. However, the command will save some amount of scrolling, clicking, and mouse movement within the adiff output.

Here's the script for acme-sac. Some slight changes might need to be made for inferno, and something very similar should work in p9p acme.

#!/dis/sh

fn findwin {
 file = $1
 < /mnt/acme/index getlines {
  (w x y z zz f rest) := ${split ' ' $line }
  if {~ $f $file} {echo $w}
 }
}

fn highlight {
 file = $1
 lineno = $2
 a := `{findwin $file}
 if {! ~ $#a 0} {
  cd /mnt/acme/$a
  <> addr {
   echo -n $lineno > addr
   echo 'dot=addr' > ctl
   echo show > ctl 
  }
 }{
  plumb $file:$lineno
 }
}

cwd = `pwd
cd /mnt/acme/$acmewin
{
 echo -n 'addr=dot' >ctl
 (a b) := `cat 
 echo -n '/^[^\-<>].*\n/' > addr
 echo -n 'dot=addr' > ctl
 echo -n show > ctl
 (f1 cmd f2) := `{getline data}
 (f1 l1) = ${split ':' $f1}
 (f2 l2) = ${split ':' $f2}
 f1 = `{cleanname -d $cwd $f1}
 f2 = `{cleanname -d $cwd $f2}
 highlight $f1 $l1
 highlight $f2 $l2
} <> addr

The findwin and highlight functions I have used before in the acme debugger interface, adeb, in /appl/acme/acme/debug/adeb.b.

lab 94 - acme content assist

noreply@blogger.com (caerwyn) — Thu, 09 Apr 2009 00:21:00 +0000

NAME

lab 94 - acme content assist

NOTES

This lab explores a way of implementing content assist for acme. I've focused on the user interface and how that might work inside acme and not specific language support.

The command is called Assist. Launch it within an editor window and it will provide assistance for that window only. A new window will open and while you type in the edit window Assist will attempt to match the currently typed word against the content of /lib/words using the command look(1). The results are displayed in the +Assist window.

With results in the +Assist window and with focus still in the editor window type Ctrl-l to step down through the results. Each result will be selected in turn and the selection will wrap around to the top. To choose a selection and replace the currently edited word with the selected text type Ctrl-k.

The Assist command also supports file completions. Start typing a path in the edit window and type Ctrl-y to show file completions in the +Assist window. Use Ctrl-l and Ctrl-k to navigate and select from the results.

Assist assumes /lib/words exists and is the default for finding word completions. You can download a copy of /lib/words from plan9port. You can also run Assist with a filename for your own list of words. It should follow the format expected by look(1).

Some interesting things about the implementation. Acme isn't aware of the control keys. It adds the control character to the document and the Assist plugin reads it and removes it from the document. In this way plugins can define their own special keys as long as acme does not already process them in some special way (see table below).

The Assist plugin does cause the editor window to change its behaviour slightly. Assist has to resend all events so that normal editor commands like Put and Undo are understood. However, because we've opened the event file the normal refresh of the tag line with Undo and Put when the file is dirtied does not happen.

The +Assist window is off to the side, or it could be placed just below the edit window. This is pretty good. But your eye does need to travel to see completion suggestions. Maybe not as ideal as the info popup appearing just below the cursor position. However, this method fits in far better to the acme model.

Special Keys

While working on this I thought I'd map out the control keys already understood by acme. The ASCII control characters are mostly passed through to the document. However, a number are interpreted by acme in a special way. In the table below NA, non-assigned, means they have their standard ASCII assignment.


Ctrl Key Description
a move cursor to beginning of line
b NA
c copy
d NA (eof)
e move cursor to end of line
f file completion
g NA
h delete
i tab
j NA (carriage return)
k NA                
l NA                   
m newline
n NA
o NA
p move cursor to beginning of document
q move cursor to end of document
r NA
s NA     
t NA
u delete to start of line
v paste
w delete to start of word
x cut
y NA
z undo
[ escape
\ NA
] NA
^ NA
_ NA
? ?

Ctrl Key	Description
a	move cursor to beginning of line
b	NA
c	copy
d	NA (eof)
e	move cursor to end of line
f	file completion
g	NA
h	delete
i	tab
j	NA (carriage return)
k	NA
l	NA
m	newline
n	NA
o	NA
p	move cursor to beginning of document
q	move cursor to end of document
r	NA
s	NA
t	NA
u	delete to start of line
v	paste
w	delete to start of word
x	cut
y	NA
z	undo
[	escape
\	NA
]	NA
^	NA
_	NA
?	?

There are also mappings for the Home keys. These mappings are for the acme editor window in the most recent commits to acme-sac (the up/down arrow have been changed compared to other acmes).


Key Description

Home scroll to beginning of document
End scroll to end of document
PgUp scroll page up
PgDown scroll page down
Left Arrow move cursor left one char
Right Arrow move cursor right one arrow
Up Arrow move cursor up one line
Down Arrow move cursor down one line

Key	Description
Home	scroll to beginning of document
End	scroll to end of document
PgUp	scroll page up
PgDown	scroll page down
Left Arrow	move cursor left one char
Right Arrow	move cursor right one arrow
Up Arrow	move cursor up one line
Down Arrow	move cursor down one line

None of the function keys are interpreted or even passed to the console on windows inferno. The escape key has some special meaning, it selects the last entered text, or cuts the selected text if text is selected. The Alt key is also used for entering unicode keys.

FILES

inferno-lab/94

Google Summer of Code 2009

noreply@blogger.com (caerwyn) — Sun, 29 Mar 2009 18:46:00 +0000

NAME

Google Summer of Code 2009

NOTES

Plan 9 has been accepted this year as a mentoring organization for Google Summer of Code. The Plan 9 group accepts projects to do with the operating system Plan 9 from Bell Labs and from related technologies such as Inferno, Plan 9 from User Space, v9fs, 9vx, and Glendix.

If you're a student and you like any of these technologies you should apply immediately to the GSoC with your a project proposal. You can take any of the ideas you've seen explored in this blog and build on them, or develop your own ideas and take Plan 9/Inferno into new territory. I would love to see projects that extend Acme-SAC such as with content-assist or syntax highlighting. Or new Inferno applications in particular for the Nintendo DS. Inferno is an "integrating environment" so try integrating Inferno with Eclipse, Python, or Ruby.

The deadline for student project proposals is April 3rd, so you need to be quick. There is so much cool stuff you could be working on this summer; Don't let this opportunity slip away.

lab 92 - vxinferno

noreply@blogger.com (caerwyn) — Sun, 04 Jan 2009 03:14:00 +0000

NAME

lab 92 - vxinferno

NOTES

In this lab I create a new Inferno builtin module that calls the vx32 library and get a minimal system working that runs native x86 code, with system calls redirected to inferno's system calls and therefore making the inferno namespace visible to the sandboxed code.

Vx32 is a new user-level sandboxing library by Bryan Ford and Russ Cox. From the vx32 paper,

"Vx32 is a multipurpose user-level sandbox that enables any application to load and safely execute one or more guest plug-ins, confining each guest to a system call API controlled by the host application and to a restricted memory region within the host’s address space."

Inferno, being a virtual operating system, provides its own system call API to limbo applications. The same system calls are available as a C API for use by native libraries that appear as builtin modules or devices within the inferno environment. This C API is a natural fit for building a Vx32 sandbox allowing native code of all kinds to run within inferno, which has considerable flexibility over the namespace made available to the native code. This would allow inferno to be extended in new ways beyond limbo, builtins, or external processes that export styx protocol. It also extends the reach of native code that when run in this hosted environment its use of files take on even greater significance.

Please read the vx32 paper, download the code and play with it. I haven't included the vx32 code in the lab. Instead this lab is more tutorial in creating a new builtin module for inferno. This labs code, linked to in the steps below, is all the code necessary to make vx32 appear as a builtin. I've done enough to show some simple examples working, but I haven't defined the full system call interface.

So here are the steps in creating a new builtin module linkage.

module interface

Create the limbo module interface, e.g. /module/vxrun.m. I created the interface to closely resemble the vxrun application in the vx32 distribution. The module contains one function to load and run a native ELF executable.

Edit /module/runt.m to include new include the new module interface. This file includes all builtin modules and is used later to generate a runtime C struct.

incorporate library code

Copy library and header files into inferno-os tree. I copied vx32.h to /include/vx32.h. I created a new libvx32 folder at the root of the tree and created a dummy mkfile. I didn't copy all the source into the tree, I cheated and just copied libvx32.a to /Linux/386/lib. But the emu build will expect the folder and mkfile to exist. So this is a placeholder for now.

add builtin to libinterp

Implement the builtin linkage /libinterp/vxrun.c This is the bulk of the work, where we call the vx32 API and map the system calls defined in the codelet C library that comes with the vx32 distribution, libvxc, to inferno's API defined in /include/kernel.h.

The template can be generated by /dis/limbo from the vxrun.m interface

% limbo -T Vxrun /module/vxrun.m
#include <lib9.h>
#include <isa.h>
#include <interp.h>
#include "Vxrunmod.h"


void
Vxrunmodinit(void)
{
 builtinmod("$Vxrun", Vxrunmodtab);
}

void
Vxrun_run(void *fp)
{
 F_Vxrun_run *f = fp;
}

Which is actually wrong, it should be,

Vxrunmodinit(void)
{
 builtinmod("$Vxrun", Vxrunmodtab, Vxrunmodlen);
}

The hard work is the filling in the Vxrun_run() function. A lot of this code was taken from vx32/src/vxrun/vxrun.c.

The main work of running the native code and redirecting syscalls is in the following loop,

 for (;;) {
  int rc = vxproc_run(p);
  if (rc < 0){
   acquire();
   *f->ret = -4;
   return;
  }
  if (rc == VXTRAP_SYSCALL) {
   if(dosyscall(p, f->ret))
    continue;
   else
    break;
  }
 }

In dosyscall we switch on the syscall number and call the Inferno method, e.g.,

static int 
dosyscall(vxproc *proc, int* fret)
{
...
 switch (NUM) {
 case VXSYSREAD:
  fd = ARG1;
  addr = ARG2;
  len = ARG3;
  if (!vxmem_checkperm(proc->mem, addr, len, VXPERM_WRITE, NULL))
   print("bad arguments to read");
  ret = kread(fd, (char*)m->base + addr, len);
  break;
... other syscalls
 }
 RET = ret;
 return 1;
}

To get this to build we need to edit the /libinterp/mkfile. Add vxrun.$O to the list of OFILES, add vxrun.m to the list of MODULES, and add the following rules to ensure the module header, vxrunmod.h, is generated.

vxrunmod.h:D: $MODULES
 rm -f $target && limbo -t Vxrun -I../module ../module/runt.m > $target
 
vxrun.$O: vxrunmod.h

We can now compile libinterp.

edit emu config

The final step is to edit /emu/Linux/emu configuration file and add the dependencies on the vxrun module and the vx32 library. We can now build a new emu that has the vx32 vxrun as a builtin module.

test

We need a limbo command to call the module. I included vxinferno.b in the lab code. But it does nothing more than load the module and call it passing in any command line arguments.

init(nil:ref Draw->Context, args:list of string)
{
 vxrun := load Vxrun Vxrun->PATH;
 vxrun->run(tl args);
}

I used the vx32-gcc to compile the native code. I included one example, cat.c, that would test the system calls, open, read, write, from the inferno namespace. Note that the name of the executable to call from inside Inferno is the host pathname, because vx32 itself is not using the Inferno system calls. This could be fixed by either changing the elf loader, or by using the library call to load the ELF from memory.

$ cd ~/vx32/src/vxrun
$ vxrungcc cat.c 
$ emu -s -r ~/inferno-os
; vxinferno /home/caerwyn/vx32/src/vxrun/_a.out /dev/drivers
#/ root
#c cons
#e env
#M mnt
...

conclusion

This lab confirmed vx32 as a builtin to inferno would work. Now it needs to be implemented in full. I'd most like to see vx32 ported to windows. There is an effort to port vx32 to windows, but it seems to have stalled.

I think this is really cool!

FILES

inferno-lab/92

lab 91 - using freetype

noreply@blogger.com (caerwyn) — Wed, 24 Dec 2008 02:55:00 +0000

NAME

lab 91 - using freetype

NOTES

While fiddling with Charon's fonts and wondering what work would be involved to replace the whole set I decided to take a quick look at the freetype module. This lab documents some of my progress.

A recent post to the acme-sac mail-list pointed me to the DejaVu fonts. They are derived from Bitstream Vera Fonts but with more characters. It includes various styles: Sans, Serif, Italic, Oblique, Bold, and Mono, making it a good choice for Charon. At first I considered converting the whole set over to Inferno format.

There is a program to convert TrueType fonts to the inferno format. But the program is designed to run on Plan 9 and I don't have a ready Plan 9 environment anymore. So the effort of setting up an environment, compiling and fixing problems I know exist in the conversion tool, creating font files for all the styles, and in a variety of sizes, and I was ready to look for an easier solution.

The Freetype library is compiled into the inferno-os emulator and exports a builtin limbo interface. There are no programs in inferno-os that use Freetype. And there is no documentation describing the limbo interface. There have, however, been a few posts to the inferno-list describing its use. Also, the Freetype documentation from its source website is good. The tutorial basically describes what needs to be done within inferno to use the freetype module.

The first example from inferno-list, testfreetype.b, shows the use of the library for rotation and scaling.

% cd lab 91
% limbo testfreetype.b
% testfreetype fonts/DejaVuSans.ttf 'Hello World'

In this screenshot, the Inferno logo image is the background, and the word 'freetype' is scaled and rotated above it, with some transparency.

The example dbft2.b is a simplification of the above that demonstrates writing a string to a window. I took the dbft2.b code an tried to adapt it to the frame module, a port a libframe from Plan 9, and used by Acme in inferno-os,

First of my own demos is an application called term that accepts keyboard input and uses frame to display the entered text inside a window. This uses inferno fonts. Using frame requires some setup code,

 framem = load Framem Framem->PATH;
 
 # setup a window client
 ...   
 win := wmclient->window(ctxt, "Term", Wmclient->Appl);
 ...

 font = Font.open(display, "/fonts/lucidasans/unicode.8.font");
 textcols = array[NCOL] of ref Draw->Image;
 textcols[BACK] = display.black;
 textcols[HIGH] = display.color(Draw->Darkyellow);
 textcols[BORD] = display.color(Draw->Yellowgreen);
 textcols[TEXT] = display.color(Draw->Medgreen);
 textcols[HTEXT] = display.black;
 framem->init(ctxt);
 frame = framem->newframe();
 win.image.draw(win.image.r, textcols[BACK], nil, ZP);
 framem->frclear(frame, 0);
 framem->frinit(frame, win.image.r,  font, win.image, textcols);

Its not documented in Inferno, but see the Plan 9 manual page.

On input from the keyboard we append it to a buffer and pass the buffer to frame:

 c := <-w.ctxt.kbd =>
  buf[len buf] = c;
  framem->frinsert(frame, buf[len buf - 1:], 1, frame.p0);

Once I got that baseline working, the next program is frame2.b and term2.b that uses freetype. In the init function I load the freetype module and load a new face.

 freetype = load Freetype Freetype->PATH;
 face = freetype->newface("./fonts/DejaVuSerif-BoldItalic.ttf", 0);
 face.setcharsize(20<<6, 72, 72);
 glyphsimg = ctxt.display.newimage(Rect((0,0), (20,20)), Draw->GREY8, 0, Draw->Black);

I rewrote the three functions used by frame to display strings, stringx, charwidth, and strwidth. Stringx does the work of loading the glyph and drawing it.

stringx(d : ref Image, p : Point, f : ref Font, s : string, c : ref Image)
{
 origin := Point(p.x<<6, (p.y+face.ascent)<<6);
 for (i := 0; i < len s; i++)
 {
  g := face.loadglyph(s[i]);
  if (g == nil){
   sys->print("No glyph for char [%c]\n", s[i]);
   continue;
  }
  drawpt := Point((origin.x>>6)+g.left, (origin.y>>6)-g.top);
  r := Rect((0,0), (g.width, g.height));
  r = r.addpt(drawpt);
  glyphsimg.writepixels(Rect((0,0), (g.width, g.height)), g.bitmap);
  d.draw(r, c, glyphsimg, (0,0));
  origin.x += g.advance.x;
 }
}

In this screenshot, the term application is running inside inferno-os, with some typed text. (Note, there is a error in my render of lowercase 'f', the top of the 'f' has been chopped off.)

Acme uses frame so we can quickly experiment with using a TTF file for the Acme font. The above three functions are in another module in the acme source called graph.b. I've included a graph2.b file in this lab that can be bound over the existing graph.dis and will load a hardcoded TTF file for the font. Below is a screenshot of acme running the DejaVuSans.ttf font.

Now back to my original aim: changing the charon fonts. I created a module that defined a new Font ADT to replace the one defined in the draw module. I changed the interface slightly, to include the freetype face and added the stringx function.

All the code for drawing text in charon is in layout.b. In this labs code I've included a replacement that uses the freetype Font adt. I changed all calls to Image.text() to call a local function that itself calls Font.stringx(). Layout.b locally defines a Fontinfo that specifies all the font files and sizes. I modified it to point to all the DejuVu fonts with sizes as appropriate. To run this yourself inside inferno-os you'll need to extract the DejuVu ttf files directly under /fonts, then,

% cd /appl/charon
% bind -bc '/n/local/inferno-lab/91' .
% limbo layout.b
% limbo ftfont.b
% bind layout.dis /dis/charon/layout.dis
% charon&

And here is an screenshot of the results.

I haven't looked at converting Tk.

FILES

lab/91

lab 89 - electroquongton

noreply@blogger.com (caerwyn) — Fri, 05 Dec 2008 02:48:00 +0000

NAME

lab 89 - electroquongton

NOTES

The code for this lab was something I was playing with to display on a Nintendo DS. It uses the mux window manager and the prefab module builtin. Because of the dependency on the builtin, which isn't usually part of the inferno-os standard emu build, I've included a muxemu.exe for Windows in this labs code. To launch the code run the following,

% muxemu -r . -g256x384 /dis/mux/mux.dis

and on the screen you should see this,

Move up and down using keys 'i' and 'm'. Enter a selection using the 'Enter' key and return to the main menu by pressing the spacebar.

I'm not experienced with the draw(2) API so I started with screens for board games.

I was experimenting using transparency effects. The look I was going for was Electroplankton for Nintendo DS.

I tried to get shapes with edges that blended out into a graded background and some simple animations of shapes pulsing.

The one serious application I was trying to write was a QUONG keyboard for convenient touch input.

This application responds to the mouse to enter letters. The top half of the windows is an implementation of Plan 9's libframe. It was part of Inferno's acme implementation and I extracted it from acme's dependencies. If you take anything away from this lab it would probably be this one library, maybe to build an inferno based 9term.

A big disappointment is that these didn't actually work on the inferno-ds due to what I guess is a bug in the handling of graphics with masks. They did seem to work on the DS emulators, but slowly. Another difficulty is writing them so that they fit in the DS memory, the techniques of which I'm completely ignorant because I'm used to the great spaciousness of modern desktops.

FILES

inferno-lab/89

lab 90 - Multicast DNS and Zeroconf

noreply@blogger.com (Eric Van Hensbergen) — Mon, 10 Nov 2008 16:32:00 +0000

NAME

lab 90 - Multicast DNS and Zeroconf (client portion)

NOTES

While using synthetic file systems to publish services works great, you still need to know where your server is. This information can be provided from a shared ndb/local or even DHCP, but there are plenty of scenarios I have run into where I don't control the DHCP server and distributing ndb/local is tedious. In working with Blue Gene the problem becomes a bit worse in that we don't know apriori which portion of the machine (and therefore which IP address) we will get. Further complicating this is the fact that our front-end node (where we run Inferno) is established by a load balancer, and there are potentially 5 of us running our own file servers. So not only do we need to know how to get to the front end node, but how to get to the right front-end node.

While some form of simple broadcast service discovery may have been sufficient, I decided to take the time to see what it would take to add multicast DNS and service-discovery to the Plan 9 and Inferno DNS services. Multicast DNS and Service Discovery (aka Bonjour, aka Rendezvous, aka Zeroconf) is documented in plenty of places, here's a few good starting points for more information:

Multicast DNS resolution is fairly straightforward, it involves just sending the DNS request to a multicast address (224.0.0.251) and using port 5353 instead of port 53. The first thing I did was modify /appl/cmd/ndb/dns.b to use this address and port when looking for any domain ending in .local per the zeroconf convention.

Service discovery is a bit more problematic. It involves sending a slightly different type of request. Typical DNS requests use an type A style request which retrieves an IP address for a hostname. Service discovery uses a PTR style request which returns three types of records -- the PTR record contains the instance of the service which will include a more specific name of the service location, a SRV record which will contain the port number of the service, a TXT record which contains some protocol specific information and an A response which contains the IP address.

So, for example, if I send a PTR request for _presence._tcp.local, I get four records in response:

ptr record -> crazyjim@arl137._presence._tcp.local
srv record -> port: 5298
txt record -> last=Peterson 1st=James msg=Away status=dnd
A record -> 9.3.61.137

I added a flag to ndb/dnsrequest (-z) which forces sending a PTR request to DNS. Using the Plan 9 DNS service as a model, I modified the Inferno dns to be able to parse SRV and TXT records. The one things I changed was that the TXT response can contain several key value pairs. The Plan 9 DNS service just strings these together (with no seperator mind you). So I print the number of key value pairs, and then have the key=value pairs one per line prefixed with dual tabs to make it look nice.

The final remaining problem is that the current DNS daemon only returns a single response for each request, but multicast DNS may have several responses for a single request. This involves a much larger set of changes to dns.b. DNS will now accumulate responses and return them in one big set.

I probably need to do some work to get cs.b to play nice with such information. The whole dnsquery, and even the cs and ndb front-ends to it seem decidedly anti-Plan 9 in their layout -- particularly for something with lots of rich attribute information like zeroconf. As such I'm likely going to create a synthetic file server with which to browse zeroconf data. You register which types of zeroconf entities you are interested in by creating directories in a two level hierarchy -- the file server will then use my modified DNS to query the local net and will create nodes under those directories for responses with attribute information broken out into individual files. The other major thing that needs to be done is adding multicast server support to dns.b. But I think I'll write these up in a different lab entries as this one is getting long in the tooth already.

While this is sufficient to solve my initial problems, there are several additional aspects of zeroconf which might be nice to integrate for Inferno including support for link-local IP addresses (which I guess would only be important on native), proper uniqueness handling for claiming your local names, NAT-PMP support, DNS-LLQ support, and dynamic DNS updates.

EXAMPLE

% ./dnsquery -z _presence._tcp.local
_presence._tcp.local ptr ericvh@ericvh-desktop._presence._tcp.local
ericvh@ericvh-desktop._presence._tcp.local txt 10
txtvers=1
1st=Eric
last=Van Hensbergen
port.p2pj=5298
status=away
node=libpurple
ver=2.5.2
vc=!
email=bergevan@us.ibm.com
phsh=943420112a8b192466a802bedfe547041a62ea90

ericvh@ericvh-desktop._presence._tcp.local srv 0 0 5298 ericvh-desktop.local
ericvh-desktop.local ipv6
ericvh-desktop.local ip 9.3.61.77
npe@macintosh-16._presence._tcp.local srv 0 0 5298 macintosh-16.local
npe@macintosh-16._presence._tcp.local txt 13
ext=
phsh=f308675309a23fa653c269c95f57eb7eb84efc44
last=
AIM=
nick=
1st=Noah
port.p2pj=5298
txtvers=1
version=1
node=
jid=
email=
status=avail

macintosh-16.local ipv6
macintosh-16.local ip 9.3.61.73
_presence._tcp.local ptr npe@macintosh-16._presence._tcp.local

DISCUSSION

dns.b seems entirely too big, and I just made it bigger. It would seem better served if it were split up into a bunch of component modules. It seems like the marshalling and unmarshalling of DNS messages is a legitimate module, the cache is a module, local database/config access is another, and then a proper module interface for performing DNS queries and/or servicing DNS requests. The file and network servers could then be provided relatively cleanly. All in all it would clean the code up signifigantly and make the whole thing a lot more readable/extensible. Things like the registry or even cs could easily be implemented as plug-in modules versus discreet file servers (although also allowing them to use file services is desirable in certain scenarios so this should definitely be parameterized).

Its funny, looking back at Virgild, it was essentially a broadcast form of multicast name resolution, just with its own more simple protocol instead of DNS.

FILES

lab90/appl/cmd/ndb/dns.b
lab90/appl/cmd/ndb/dnsquery.b

lab 88 - degrees of freedom

noreply@blogger.com (caerwyn) — Fri, 02 May 2008 02:28:00 +0000

NAME

lab 88 - degrees of freedom

NOTES

The Vitanuova downloads page has historical snapshots of Inferno source from 1996 to 2003 containing all three editions before Inferno went open source. I was curious to see how well Inferno has sustained a standard set of interfaces over the last ten years so I downloaded all of them and poked around.

The biggest overall change came with 4th edition, when many parts of the system were upgraded, including the Styx protocol, several builtin modules, dis format, the limbo language and VM. Also, significantly, Inferno adopted open source licenses granting developers the freedom to modify any part of the system: something that might impact sustainability for good or ill.

While every edition prior to 4th has had some interfaces changed, these changes did not break backwards compatibility. A 3rd edition emu can run dis code from the 1st edition archive.

The difference between 3rd and 4th was large enough that limbo code needed to be ported, or at the very least recompiled, to run on the new emu.

The Sys interface is evolving still with additions made since 4th edition was released. These types of changes, adding a function or a new constant, do not break backward compatibility, but in a network of emus where there is diversity of versions, link typechecks do fail when a module is expecting an interface newer than the one available. Where is the standard interface here? This problem seems to violate some of the core ideas of Inferno, and Inferno doesn't provide an easy way of working around compatibility issues with builtin modules.

Inferno's core idea is to provide standard interfaces that free content and service providers from concern of the details of diverse hardware, software, and networks over which their content is delivered. (/sys/doc/bltj.ms)

The BLTJ paper describing Inferno listed the several dimensions of portability and versatility provided by the OS,

Portability across processors: it currently runs on Intel, Sparc, MIPS, ARM, HP-PA, and PowerPC architectures and is readily portable to others.
Portability across environments: it runs as a stand-alone operating system on small terminals, and also as a user application under Windows NT, Windows 95, Unix (Irix, Solaris, FreeBSD, Linux, AIX, HP/UX) and Plan 9. In all of these environments, Inferno applications see an identical interface.
Distributed design: the identical environment is established at the user's terminal and at the server, and each may import the resources (for example, the attached I/O devices or networks) of the other. Aided by the communications facilities of the run-time system, applications may be split easily (and even dynamically) between client and server.
Minimal hardware requirements: it runs useful applications stand-alone on machines with as little as 1 MB of memory, and does not require memory-mapping hardware.
Portable applications: Inferno applications are written in the type-safe language Limbo, whose binary representation is identical over all platforms.
Dynamic adaptability: applications may, depending on the hardware or other resources available, load different program modules to perform a specific function. For example, a video player application might use any of several different decoder modules.

Now that we have a decade of Inferno history, how many of the above degrees of freedom still hold when the whole time span is considered as one network of interconnected emus?

Don't standard interfaces also imply standard across time? A network of emus can not be expected to upgrade all at the same time. A standard is also a constraint against change in an interface. One degree of freedom is expressly limited; keep the abstraction constant.

The dilemma faced is whether to freeze an interface to provide long term compatibility based on a standard, but risk the possibility of being held back from adopting new ideas and becoming irrelevant, or to keep changing interfaces to solve new problems but pay the cost of compatibility problems.

In general it seems filesystems, namespaces and textual interfaces all lend well to creating a sustainable software environment. However, the more complex limbo language and module interfaces have shown themselves to be not so well preserved. By comparison, maybe unfairly because of the different goals of the creators, see how the works of Knuth are intended to withstand time. He specifically structures his software so that incompatibilities do not creep in, such as leaving no undefined gaps in font tables so that no one is tempted to fill them (TeX Book), or defining instructions to fill all 256 possible slots in MMIX, and making his source readable but not permitting edits except through his CWEB change file system. Knuth's software is the only kind I know of that take seriously the problem of long term compatibility.

It would be nice to run 1st edition dis code in a current emu if for no other reason than to prove the sustainability of infernos standard interfaces, but a major barrier to that is the need to bind in old Sys and Draw modules. I assume that compatibility to older interfaces should be provided through limbo modules so that the emu doesn't bear the extra weight and complexity of carrying multiple builtin implementations. There is no way to override where a builtin module is loaded from, though this might be a nice feature. For example, if /dev/dis/draw.dis represented the builtin draw module, I might bind limbo implementation over it, so that a load Draw "$Draw", would take the compatibility simulation over the builtin. (This might also work nicely going the other way so that we could bind builtin modules over limbo modules for optimization.)

This is not possible for Sys however, because we can't simulate variadic args in limbo (e.g., sys->print). A solution to this would be nice! But an alternative to providing more mechanism is simply to freeze the various interfaces.

lab 87 - mux for nintendo ds

noreply@blogger.com (caerwyn) — Mon, 07 Apr 2008 00:47:00 +0000

NAME

lab 87 - mux for nintendo ds

NOTES

In an earlier post I talked about updating mux to 4th edition Inferno in the hope of one day running it on a Nintendo DS.

Well, Inferno is now booting on the DS so I got to try it for real.

I started with getting the mux window manager working in standard inferno. Then I changed the resolution down to 256x192 and tried to get everything to fit. The files in this lab include the version of mux I ended up putting in the nds file running on the DS.

Things to try if you download it. Rocker moves up and down selection. 'A' key enters, 'B' key backs out back up to the higher level. 'Start' key returns to the top level menu.

Try Today's Newspaper, and The Thisburgh has the only working graphic. Under news, click through to actually read an article. Under games, try connect4. Audio control would look cool if any of the graphics actually came in. The Financial Reports gives a ticker. It scrolls slowly only because of the sleep interval in the code is incorrect.

If you want to try this version of mux using hosted inferno just remember you need to compile prefab into your emu. Include prefab in the mod and lib sections of your emu config file, also uncomment prefab in the /libinterp/mkfile.

Mux uses irsim for key controls. I changed my local inferno-ds code to have the DS keys output the same characters as used by irsim.

The files in this lab include the movies and tvlist apps and their data. The data didn't fit on the 4MB .nds file. But they will fit when we get the GBA ROM or dldi interface working.

I think mux is a good path to follow for DS development. It's small, starts quickly, uses the keys effectively since it was designed for remote controls, the programs are easy to understand, and they hit most of the applications I'd like to start with, small games, news reader, email reader, simple database browser (movies, tvlist), and audio.

FILES

code for lab 87

lab 86 - srv

noreply@blogger.com (caerwyn) — Sun, 06 Apr 2008 18:35:00 +0000

NAME

lab 86 - srv

NOTES

This from a post on 9fans, and also on Tip O' the Day

% dc >[0=1] | echo 0 > /srv/desk

Plan 9's srv(3) acts as a bulletin board for open file descriptors, other namespaces see all the files in srv, and so can read and write to /srv/desk.

Inferno has srv(3) which is a file2chan registry, but is also visible to all namespaces on the host. (see also srv9(3))

The current implementation of sh-file2chan(1) does not allow the above. The closest I got was,

% load file2chan
% calc >[0=1] | {file2chan /chan/desk {rblock; putrdata &} {fetchwdata > /fd/0}} &

% stream -b 1 /chan/desk
% echo 1+1 > /chan/desk

I tried implementing a command equivalent to srv(4) on Plan 9. It takes a command block or network address and post it in the srv registry.

% srv {calc >[1=0]}  /chan/desk

It using an existing '#s' instance if there is one, else binds a new one. Now we can open a console to /chan/desk from another window

% {cat & cat  >[1=0] < /dev/cons} <> /chan/desk

and other windows can write to /chan/desk, the output will be seen in the console.

Questions.

Why isn't Plan 9 srv(3) in Inferno?
File2chan(2) seems under used. Is that because of the shell interface sh-file2chan?
Is there another interface that would make file2chan more usable?
Mount(1) supports mounting from a file, as Plan 9's does. But the inferno srv(3) device must do extra copies of the read and write buffers to implement the interface. Is the file interface of Plan9 srv more elegant than the extra file2chan syscall in Inferno?
Aren't there benefits to using channels in Inferno that make file2chan preferable?

Files in this lab are for the inferno srv(4) implementation.

FILES

lab 86 code

lab 85 - stowage

noreply@blogger.com (caerwyn) — Tue, 25 Mar 2008 03:02:00 +0000

NAME

lab 85 - stowage

NOTES

In an earlier post I defined a venti-lite based on two shell scripts, getclump and putclump, that stored files in a content addressed repository, which in that instance was just an append-only gzip tar archive with an index.

After learning a little about the git SCM, this lab re-writes those scripts to use a repository layout more like git's. The key thing to know about the git repository is that it uses sha1sum(1) content addressing and that it stores the objects as regular files in a filesystem using the hash as the directory and filename,

  objects/hh/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh

In the objects directory is 256 directories named for every 2 character prefix of the sha1hash of the object. The filename is the remaining 38 characters of the hash.

Putclump calculates the hash, slices it to make the prefix and filename, tests if the file already exists, and if not writes the compressed data to the new file. Here is the important part of putclump,

 (sha f) := `{sha1sum $file}
 (a b) := ${slice 0 2 $sha} ${slice 2 40 $sha}
 
 if {ftest -e $hold/objects/$a/$b} {} {
  mkdir -p $hold/objects/$a
  gzip < $file > $hold/objects/$a/$b
 }

Getclump just needs to look up the file given a hash

 sha := $1
 (a b) := ${slice 0 2 $sha} ${slice 2 40 $sha}
 files := `{ls $hold/objects/$a/$b^* >[2] /dev/null}
 if {~ $#files 1} {gunzip < $files }

Because the git repository uses a regular file system to store objects, it makes it considerably easier to work with than the compacted file system like tar.gz, or an application specific binary format like venti. This is because instead of having to create new tools to read and write binary formats, we can re-use existing tools, like sh(1), tarfs(4), updatelog(8), and applylog(8).

For example, I wrote a script, stow, that takes a tarball and stores it in my repository, called the hold. The hold should be created first with the following directories,

 /n/hold/logs
 /n/hold/objects
 /n/hold/stowage

Then give stow the name of a .tar or .tgz file. Files not found in the hold and that were added are printed to stdout.

 % stow acme-0.11.tgz
 ...
 %

Stow uses updatelog(8) to create a stowage manifest file for the tarball I added. This manifest is saved under /n/stowage. The manifest records the pathname, perms, and sha1 hash of every file in the tarball.

Now that I've stowed all my tarballs I need a way of getting things out.

I built a holdfs, derived from tarfs(4), to read the stowage manifest and present files from the hold. By default the file system is mounted on /mnt/arch.

 % holdfs /n/hold/stowage/acme-0.11

The hold with its stowage is be a step up from a directory tarpit of tarballs. I can accumulate a version history based on tar.gz releases like that for acme-sac and inferno. The vitanuova downloads site contains inferno history going back to 1997. My downloads page contains snapshots of inferno from 2002 to 2006 and acme-sac after that.

My intended application for this was that I could encourage forks of a project and merge back many individuals releases into a single repository and still do useful comparisons.

Using a filled hold I should be able to do analysis of a file history based on the stowage manifests. Contained in this lab are a few experimental scripts to build out more of an SCM. For example, the script hold/diff attempts to use updatelog to compare a manifest with the current tree. And hold/difflog uses a modified applylog(8) to compare two manifests.

The nautical references suggest a distributed and loosely coupled network like that of shipping, and is also influenced by git's design. The unit of transfer is a tarball. It is stowed into the ships hold, along with the manifest. A file system interprets the manifests and gives an interface for searching the hold. There is also keep a ships log of what was stowed and when. I can extract patches, files, tarballs, or the complete stowage in my hold to share with someone else.

This is a simple system:

     28 putclump.sh
     16 getclump.sh
     54 stow.sh
    669 holdfs.b
    767 total

But then most of what was needed already existed in inferno.

FILES

lab 85 code

lab 84 - gridfs pattern (mapreduce)

noreply@blogger.com (caerwyn) — Sun, 23 Mar 2008 20:29:00 +0000

NAME

lab 84 - gridfs pattern (mapreduce)

NOTES

I've mentioned mapreduce in previous posts. It makes a good example application for thinking about grid computing. This lab is also about mapreduce although the point here is to illustrate an inferno pattern for grid computing. I'll call it here the gridfs pattern.

Say you have a grid of compute nodes and you want to distribute and coordinate work among them. For the gridfs pattern you construct a synthetic file system that will get exported to all the nodes. The file system is the master process and all clients to the file system are workers.

Both cpu(1) and rcmd(1) use the rstyxd(8) protocol that exports the local namespace when running remote jobs. To implement the gridfs pattern we bind our master fs into our local namespace so it gets exported when we run multiple workers across our compute grid.

A very simple example of this pattern is explained in the Simple Grid Tutorial Part 2. I export a named pipe with a provider process writing one line at a time to the pipe; then multiple worker processes running across the grid consume lines from the pipe as fast as they can do the work.

The mapreduce source files I've included in this lab are a concrete (rough and experimental) example of this pattern taken to the next level. The namespace it exports is the following,

  mapreduce/clone
  mapreduce/n
  mapreduce/n/ctl
  mapreduce/n/status

Each worker opens the clone file and gets a unique connection to the master process, represented by a numbered directory. The open clone file becomes an open file descriptor to the ctl file of the new connection. The worker reads messages from ctl describing new work, and it can write back messages about work completed. The master process will keep the status file up to date with the progress of the worker, analogous to prog(3).

An advantage of this approach over the simpler named pipe is that the master process knows exactly when the worker has closed the connection and knows how much work they have completed based on the messages written to the ctl file. It also provides a better interface to the user; The ps(1) command can easily be adapted to read the status files from the mapreduce namespace.

To try out some examples using mapreduce I need to provide a mapper and reducer function. I wrote a module interface for a mapper,

Mapper : module {
    map: fn(key, value: string, emit: chan of (string, string));
};

This takes a key and value and maps it to an intermediate key and value, which it emits on a channel; it may emit many intermediate key value pairs for a single input key value pair. Here's an implementation for a mapper that takes a string input, tokenizes it, and outputs the token and '1', which will be added later for a wordcount.

# the map function may not get the whole file in one go. maybe
# just a segment, or a line.
map(nil, value: string, emit: chan of (string, string))
{
 if(sys == nil)
  sys = load Sys Sys->PATH;
 if(str == nil)
  str = load String String->PATH;
 (nil, f) := sys->tokenize(value, 
               "[]{}()!@#$%^&*?><\":;.,|\\-_~`'+=/ \t\n\r");
 for ( ; f != nil; f = tl f) {
  ss := str->tolower(hd f);
  emit <-= (ss, "1");
 }
}

There is also an interface for a reducer,

Reducer : module {
    reduce: fn(key: string, input: chan of string, emit: chan of string);
};

This takes all the intermediate values for a key and emits a value. Here's the adder, used by the wordcount.

reduce(nil: string, v: chan of string, emit: chan of string)
{
 value := 0;
 while((s :=<- v) != nil)
  value += int s;
 emit <-= string value;
}

The mapper and reducer interfaces are known by a worker process that loads them on demand. An intermediate process that combines values of the same keys and sorts them is also implemented in the worker process (See the Google MapReduce paper for a good explanation.) This implementation of mapreduce knows only how to walk directory hierarchies and print the file names to all the worker processes. Here's an example of a mapreduce command line that counts words in all files below /lib/legal.

  % mkdir /mnt/mapreduce
  % mapreduce -M4 -R3 wordcount adder /lib/legal
  % ls /mnt/mapreduce
  /mnt/mapreduce/clone

Mapreduce should launch and manage all its own processes. However, for the code checked into this lab, to illustrate what is going on, I have it launching nothing. It just mounts the file system on /mnt/mapreduce. The arguments '-M4 -R3' say to expect 4 Mapper processes and 3 Reducer processes. As workers connect it will configure it to be a mapper or reducer depending on whether work remains. Therefore, after running the above command and doing a cat(1) on /mnt/mapreduce/clone we should see the config line then the pathnames for the first worker.

  % cat /mnt/mapreduce/clone
  worker -m -R 3 -d wordcount -i 1
  /lib/legal/GPL 0 17982
  ...

The pathnames are divided up among the workers as fast as they can process them. So in this implementation mapreduce functions almost the same as the named pipe in the simple grid tutorial. The cat of the first clone file will return all pathnames!

Mapreduce however is still expecting more workers. Cat the clone file three more times to see the input to the next 3 workers. The next cat after that you should see the config and input to the reducer. For example from a remote node,

  % rcmd ${nextcpu} cat /n/client/mnt/mapreduce/clone

Doing a listing on the /mnt/mapreduce path should show you the current workers connected (if any). After all reducers have disconnected, the mapreduce filesystem will report it's done and exit.

Lets run it again for real using the mapreduce worker processes.

  % mapreduce -m /mnt/mapreduce -M4 -R3 wordcount adder /lib
  % for i in 1 2 3 4 {mapreduce/worker /mnt/mapreduce/clone&}
  % for i in 1 2 3 {mapreduce/worker /mnt/mapreduce/clone&}

You should see the result files in /tmp/out.*

For the GSoC 2008 I suggested a project where the student implement a splitjoin file system. Create a coarse grained splitjoin service as defined loosely here (PDF) (see slides 18 on for fine grained task parallelism). This suggested implementation is really another concrete example of the gridfs pattern. It would allow control over how messages are passed round robin to all the workers. It would permit different configurations of how many to push to each node, how many to join from each node, how many commands to duplicate. E.g.,

filter | splitjoin -d10 -m5 -n3 {cmd} | filter

creates 10 duplicates of the cmd, take input from a pipeline and distributes m=5 records at a time round robin to each node and join the output n=3 records at a time from each task back out to the pipeline.

Splitjoin would take care of launching the task, and monitoring the task for completion. (Ideally, it would interact with the registry to decide where to launch services.)

Because Plan 9/Inferno is not participating this year in GSoC I will probably have a crack at this.

FILES

lab 84 code

lab 83 - lcmd local cpu

noreply@blogger.com (caerwyn) — Wed, 12 Mar 2008 02:12:00 +0000

NAME

lab 83 - lcmd local cpu

NOTES

While thinking of the Simple Grid Tutorial Part 1 and Part 2, I wondered whether I could implement the equivalent of rcmd(1) but for a local emu launched using os(1). For example,

 lcmd math/linbench 100

would launch a new emu, export the local fs to it through a pipe rather than a network socket, and run the command in that namespace. The idea seemed simple, no apparent obstacles, but it actually took me a couple of evenings to get it to work. So I'm posting it more because of the effort rather than its value.

First lets look at what rcmd does, ignoring the networking. Given its arguments it builds a string, calculates its length + 1, and writes the length then the string to a file descriptor, then exports the local namespace to the same file descriptor. Well that part is easy to do in sh(1). Here it is as a braced block assuming all work is done on file descriptor 1.

fn lcmd {
 load expr string
 args := $*
 s := sh -c ${quote $"args}
 echo ${expr ${len $"s} 1 + } 
 echo $s
 export / /fd/1
}

We can test that,

 lcmd {ls } | auxi/rstyxd

Now, instead of running rstyxd in the current VM, I want to run another instance and run in it that. This is where it gets complicated. You might think this might work,

 
 lcmd {ls} | os emu auxi/rstyxd

It doesn't because os treats stdin as read only, stdout as write only. Because export(1) needs to read and write on one file descriptor, and so does rstyxd(8), we need to setup extra pipes, both on the local end and the remote end.

Another problem presents itself in emu. At startup rstyxd will see /dev/cons as stdin. But I'd need to bypass the keyboard handling and get the direct stdin from the pipe. We see the answer to that in /dev,

% ls -l /dev/host*
--rw-r--r-- c 0 caerwyn caerwyn 0 Oct 30 22:43 /dev/hostowner
---w--w--w- c 0 caerwyn caerwyn 0 Oct 30 22:43 /dev/hoststderr
--r--r--r-- c 0 caerwyn caerwyn 0 Oct 30 22:43 /dev/hoststdin
---w--w--w- c 0 caerwyn caerwyn 0 Oct 30 22:43 /dev/hoststdout

This looks good but when I tried them they were not fully implemented in the current emu. The details are not interesting. I fixed that in the acme-sac tree and committed it.

Finally, we can build our full lcmd

fn lcmd {
 load std expr string
 pctl forkns
 args := $*
 s := sh -c ${quote $"args}
 bind '#|' /tmp/lpipe
 
 {
  echo ${expr ${len $"s} 1 + }  >/fd/0;  
  echo $s >/fd/0; 
  export / /fd/0
 } <>/tmp/lpipe/data    &
 
 os -d 'd:/acme-sac' d:/acme-sac/sys/Nt/386/bin/icell.exe -c1  sh -c '
  bind  ''#|'' /tmp/pipes; 
  cat /tmp/pipes/data > /dev/hoststdout& 
  cat /dev/hoststdin > /tmp/pipes/data& 
  auxi/rstyxd <>/tmp/pipes/data1 >[2] /dev/null;  
  echo halt > /dev/sysctl' < /tmp/lpipe/data1 >/tmp/lpipe/data1
}

Heh!

I'm using icell.exe built using the cell config from acme-sac. This is a really small emu configuration. The directories /tmp/pipes, /tmp/lpipe are assumed to exist.

From this definition we can replace rcmd with lcmd in the commands for rsplit and lk in the Grid Tutorial Part 2 and get emu tools for multicores without the setup required for the grid.

lab 82, again

noreply@blogger.com (hugo) — Tue, 12 Feb 2008 14:08:00 +0000

NAME

lab 82, again - txt2iaf

NOTES

After thinking the matter a little, I finally wrote the txt2iaf app. This allows to use any text file, with one or two columns of data, to be converted to an iaf file that can be played using auplay(1). Now, my sound analysis goes something like this:

1. Record something using a microphone. For now, I only record stuff using some app in Windows (Sound Recorder) or Linux (arecord(1)). This is because Inferno has some issues when trying to record something from /dev/audio: an annoying tick every so often that wrecks my sound analysis intentions. Maybe, I can help to fix this problem, probably related with buffering.

2. Convert the wav file obtained to an iaf file, using wav2iaf(1).

3. Get the data from the iaf file to a text format, using iaf2txt(?).

4. Read data from the text file using any data analysis package.

5. Do whatever you want to with the data.

6. If you wish or need to, output the data in a text file.

7. Using txt2iaf(?), create an iaf file using the data in the text file. Enjoy your new song using auplay(1) ;-)

8. If you want to, convert your iaf file to a wav file using iaf2wav(?).

I also fixed some ugly code in iaf2txt(?) and iaf2wav(?) (really ugly, indeed). Please, update.

Cheers

FILES

lab 82 code

A couple of IAF utilities

noreply@blogger.com (hugo) — Wed, 06 Feb 2008 21:53:00 +0000

NAME

lab 82 - iaf2txt and iaf2wav

NOTES

Currently, a couple of my friends and I are playing a little with human voices. For this purpose I wrote two applications to convert iaf files to plain text and to the wav audio format. Both apps support the standard iaf files as described in audio(6), except for the encoding: only PCM is supported for now.

Why in the world would one need a text file converted from an iaf? Well... text files are easier to handle with data analysis software like the R programming language. I know MATLAB supports working with wav files directly, but there are mainly two reasons I needed an iaf to txt converter:

1. I do no use MATLAB.

2. When I wrote the iaf2txt app there was not an iaf to wav converter.

Maybe R can handle wav files directly, but I do not know.

I am not really sure if I need a text to iaf converter, but I am thinking about this issue. So far I do not need one.

FILES

lab 82 code