Saturday, January 03, 2009

NAME

lab 92 - vxinferno

NOTES

In this lab I create a new Inferno builtin module that calls the vx32 library and get a minimal system working that runs native x86 code, with system calls redirected to inferno's system calls and therefore making the inferno namespace visible to the sandboxed code.

Vx32 is a new user-level sandboxing library by Bryan Ford and Russ Cox. From the vx32 paper,

"Vx32 is a multipurpose user-level sandbox that enables any application to load and safely execute one or more guest plug-ins, confining each guest to a system call API controlled by the host application and to a restricted memory region within the host’s address space."

Inferno, being a virtual operating system, provides its own system call API to limbo applications. The same system calls are available as a C API for use by native libraries that appear as builtin modules or devices within the inferno environment. This C API is a natural fit for building a Vx32 sandbox allowing native code of all kinds to run within inferno, which has considerable flexibility over the namespace made available to the native code. This would allow inferno to be extended in new ways beyond limbo, builtins, or external processes that export styx protocol. It also extends the reach of native code that when run in this hosted environment its use of files take on even greater significance.

Please read the vx32 paper, download the code and play with it. I haven't included the vx32 code in the lab. Instead this lab is more tutorial in creating a new builtin module for inferno. This labs code, linked to in the steps below, is all the code necessary to make vx32 appear as a builtin. I've done enough to show some simple examples working, but I haven't defined the full system call interface.

So here are the steps in creating a new builtin module linkage.

module interface

Create the limbo module interface, e.g. /module/vxrun.m. I created the interface to closely resemble the vxrun application in the vx32 distribution. The module contains one function to load and run a native ELF executable.

Edit /module/runt.m to include new include the new module interface. This file includes all builtin modules and is used later to generate a runtime C struct.

incorporate library code

Copy library and header files into inferno-os tree. I copied vx32.h to /include/vx32.h. I created a new libvx32 folder at the root of the tree and created a dummy mkfile. I didn't copy all the source into the tree, I cheated and just copied libvx32.a to /Linux/386/lib. But the emu build will expect the folder and mkfile to exist. So this is a placeholder for now.

add builtin to libinterp

Implement the builtin linkage /libinterp/vxrun.c This is the bulk of the work, where we call the vx32 API and map the system calls defined in the codelet C library that comes with the vx32 distribution, libvxc, to inferno's API defined in /include/kernel.h.

The template can be generated by /dis/limbo from the vxrun.m interface

% limbo -T Vxrun /module/vxrun.m
#include <lib9.h>
#include <isa.h>
#include <interp.h>
#include "Vxrunmod.h"


void
Vxrunmodinit(void)
{
 builtinmod("$Vxrun", Vxrunmodtab);
}

void
Vxrun_run(void *fp)
{
 F_Vxrun_run *f = fp;
}

Which is actually wrong, it should be,

Vxrunmodinit(void)
{
 builtinmod("$Vxrun", Vxrunmodtab, Vxrunmodlen);
}

The hard work is the filling in the Vxrun_run() function. A lot of this code was taken from vx32/src/vxrun/vxrun.c.

The main work of running the native code and redirecting syscalls is in the following loop,

 for (;;) {
  int rc = vxproc_run(p);
  if (rc < 0){
   acquire();
   *f->ret = -4;
   return;
  }
  if (rc == VXTRAP_SYSCALL) {
   if(dosyscall(p, f->ret))
    continue;
   else
    break;
  }
 }

In dosyscall we switch on the syscall number and call the Inferno method, e.g.,

static int 
dosyscall(vxproc *proc, int* fret)
{
...
 switch (NUM) {
 case VXSYSREAD:
  fd = ARG1;
  addr = ARG2;
  len = ARG3;
  if (!vxmem_checkperm(proc->mem, addr, len, VXPERM_WRITE, NULL))
   print("bad arguments to read");
  ret = kread(fd, (char*)m->base + addr, len);
  break;
... other syscalls
 }
 RET = ret;
 return 1;
}

To get this to build we need to edit the /libinterp/mkfile. Add vxrun.$O to the list of OFILES, add vxrun.m to the list of MODULES, and add the following rules to ensure the module header, vxrunmod.h, is generated.

vxrunmod.h:D: $MODULES
 rm -f $target && limbo -t Vxrun -I../module ../module/runt.m > $target
 
vxrun.$O: vxrunmod.h

We can now compile libinterp.

edit emu config

The final step is to edit /emu/Linux/emu configuration file and add the dependencies on the vxrun module and the vx32 library. We can now build a new emu that has the vx32 vxrun as a builtin module.

test

We need a limbo command to call the module. I included vxinferno.b in the lab code. But it does nothing more than load the module and call it passing in any command line arguments.

init(nil:ref Draw->Context, args:list of string)
{
 vxrun := load Vxrun Vxrun->PATH;
 vxrun->run(tl args);
}

I used the vx32-gcc to compile the native code. I included one example, cat.c, that would test the system calls, open, read, write, from the inferno namespace. Note that the name of the executable to call from inside Inferno is the host pathname, because vx32 itself is not using the Inferno system calls. This could be fixed by either changing the elf loader, or by using the library call to load the ELF from memory.

$ cd ~/vx32/src/vxrun
$ vxrungcc cat.c 
$ emu -s -r ~/inferno-os
; vxinferno /home/caerwyn/vx32/src/vxrun/_a.out /dev/drivers
#/ root
#c cons
#e env
#M mnt
...

conclusion

This lab confirmed vx32 as a builtin to inferno would work. Now it needs to be implemented in full. I'd most like to see vx32 ported to windows. There is an effort to port vx32 to windows, but it seems to have stalled.

I think this is really cool!

FILES

inferno-lab/92

Tuesday, December 23, 2008

NAME

lab 91 - using freetype

NOTES

While fiddling with Charon's fonts and wondering what work would be involved to replace the whole set I decided to take a quick look at the freetype module. This lab documents some of my progress.

A recent post to the acme-sac mail-list pointed me to the DejaVu fonts. They are derived from Bitstream Vera Fonts but with more characters. It includes various styles: Sans, Serif, Italic, Oblique, Bold, and Mono, making it a good choice for Charon. At first I considered converting the whole set over to Inferno format.

There is a program to convert TrueType fonts to the inferno format. But the program is designed to run on Plan 9 and I don't have a ready Plan 9 environment anymore. So the effort of setting up an environment, compiling and fixing problems I know exist in the conversion tool, creating font files for all the styles, and in a variety of sizes, and I was ready to look for an easier solution.

The Freetype library is compiled into the inferno-os emulator and exports a builtin limbo interface. There are no programs in inferno-os that use Freetype. And there is no documentation describing the limbo interface. There have, however, been a few posts to the inferno-list describing its use. Also, the Freetype documentation from its source website is good. The tutorial basically describes what needs to be done within inferno to use the freetype module.

The first example from inferno-list, testfreetype.b, shows the use of the library for rotation and scaling.

% cd lab 91
% limbo testfreetype.b
% testfreetype fonts/DejaVuSans.ttf 'Hello World'

In this screenshot, the Inferno logo image is the background, and the word 'freetype' is scaled and rotated above it, with some transparency.

108112312208-Inferno

The example dbft2.b is a simplification of the above that demonstrates writing a string to a window. I took the dbft2.b code an tried to adapt it to the frame module, a port a libframe from Plan 9, and used by Acme in inferno-os,

First of my own demos is an application called term that accepts keyboard input and uses frame to display the entered text inside a window. This uses inferno fonts. Using frame requires some setup code,

 framem = load Framem Framem->PATH;
 
 # setup a window client
 ...   
 win := wmclient->window(ctxt, "Term", Wmclient->Appl);
 ...

 font = Font.open(display, "/fonts/lucidasans/unicode.8.font");
 textcols = array[NCOL] of ref Draw->Image;
 textcols[BACK] = display.black;
 textcols[HIGH] = display.color(Draw->Darkyellow);
 textcols[BORD] = display.color(Draw->Yellowgreen);
 textcols[TEXT] = display.color(Draw->Medgreen);
 textcols[HTEXT] = display.black;
 framem->init(ctxt);
 frame = framem->newframe();
 win.image.draw(win.image.r, textcols[BACK], nil, ZP);
 framem->frclear(frame, 0);
 framem->frinit(frame, win.image.r,  font, win.image, textcols);

Its not documented in Inferno, but see the Plan 9 manual page.

On input from the keyboard we append it to a buffer and pass the buffer to frame:

 c := <-w.ctxt.kbd =>
  buf[len buf] = c;
  framem->frinsert(frame, buf[len buf - 1:], 1, frame.p0);

Once I got that baseline working, the next program is frame2.b and term2.b that uses freetype. In the init function I load the freetype module and load a new face.

 freetype = load Freetype Freetype->PATH;
 face = freetype->newface("./fonts/DejaVuSerif-BoldItalic.ttf", 0);
 face.setcharsize(20<<6, 72, 72);
 glyphsimg = ctxt.display.newimage(Rect((0,0), (20,20)), Draw->GREY8, 0, Draw->Black);

I rewrote the three functions used by frame to display strings, stringx, charwidth, and strwidth. Stringx does the work of loading the glyph and drawing it.

stringx(d : ref Image, p : Point, f : ref Font, s : string, c : ref Image)
{
 origin := Point(p.x<<6, (p.y+face.ascent)<<6);
 for (i := 0; i < len s; i++)
 {
  g := face.loadglyph(s[i]);
  if (g == nil){
   sys->print("No glyph for char [%c]\n", s[i]);
   continue;
  }
  drawpt := Point((origin.x>>6)+g.left, (origin.y>>6)-g.top);
  r := Rect((0,0), (g.width, g.height));
  r = r.addpt(drawpt);
  glyphsimg.writepixels(Rect((0,0), (g.width, g.height)), g.bitmap);
  d.draw(r, c, glyphsimg, (0,0));
  origin.x += g.advance.x;
 }
}

In this screenshot, the term application is running inside inferno-os, with some typed text. (Note, there is a error in my render of lowercase 'f', the top of the 'f' has been chopped off.)

108112314543-Inferno

Acme uses frame so we can quickly experiment with using a TTF file for the Acme font. The above three functions are in another module in the acme source called graph.b. I've included a graph2.b file in this lab that can be bound over the existing graph.dis and will load a hardcoded TTF file for the font. Below is a screenshot of acme running the DejaVuSans.ttf font.

108112392028-Inferno

Now back to my original aim: changing the charon fonts. I created a module that defined a new Font ADT to replace the one defined in the draw module. I changed the interface slightly, to include the freetype face and added the stringx function.

All the code for drawing text in charon is in layout.b. In this labs code I've included a replacement that uses the freetype Font adt. I changed all calls to Image.text() to call a local function that itself calls Font.stringx(). Layout.b locally defines a Fontinfo that specifies all the font files and sizes. I modified it to point to all the DejuVu fonts with sizes as appropriate. To run this yourself inside inferno-os you'll need to extract the DejuVu ttf files directly under /fonts, then,

% cd /appl/charon
% bind -bc '/n/local/inferno-lab/91' .
% limbo layout.b
% limbo ftfont.b
% bind layout.dis /dis/charon/layout.dis
% charon&

And here is an screenshot of the results.

1081123203852-Inferno

I haven't looked at converting Tk.

FILES

lab/91

Thursday, December 04, 2008

NAME

lab 89 - electroquongton

NOTES

The code for this lab was something I was playing with to display on a Nintendo DS. It uses the mux window manager and the prefab module builtin. Because of the dependency on the builtin, which isn't usually part of the inferno-os standard emu build, I've included a muxemu.exe for Windows in this labs code. To launch the code run the following,

% muxemu -r . -g256x384 /dis/mux/mux.dis

and on the screen you should see this,

10872522390-Inferno

Move up and down using keys 'i' and 'm'. Enter a selection using the 'Enter' key and return to the main menu by pressing the spacebar.

I'm not experienced with the draw(2) API so I started with screens for board games.

10861214276-Inferno

I was experimenting using transparency effects. The look I was going for was Electroplankton for Nintendo DS.

I tried to get shapes with edges that blended out into a graded background and some simple animations of shapes pulsing.

108725231623-Inferno

The one serious application I was trying to write was a QUONG keyboard for convenient touch input.

10872420332-Inferno

This application responds to the mouse to enter letters. The top half of the windows is an implementation of Plan 9's libframe. It was part of Inferno's acme implementation and I extracted it from acme's dependencies. If you take anything away from this lab it would probably be this one library, maybe to build an inferno based 9term.

A big disappointment is that these didn't actually work on the inferno-ds due to what I guess is a bug in the handling of graphics with masks. They did seem to work on the DS emulators, but slowly. Another difficulty is writing them so that they fit in the DS memory, the techniques of which I'm completely ignorant because I'm used to the great spaciousness of modern desktops.

FILES

inferno-lab/89

Monday, November 10, 2008

NAME

lab 90 - Multicast DNS and Zeroconf (client portion)

NOTES

While using synthetic file systems to publish services works great, you still need to know where your server is. This information can be provided from a shared ndb/local or even DHCP, but there are plenty of scenarios I have run into where I don't control the DHCP server and distributing ndb/local is tedious. In working with Blue Gene the problem becomes a bit worse in that we don't know apriori which portion of the machine (and therefore which IP address) we will get. Further complicating this is the fact that our front-end node (where we run Inferno) is established by a load balancer, and there are potentially 5 of us running our own file servers. So not only do we need to know how to get to the front end node, but how to get to the right front-end node.

While some form of simple broadcast service discovery may have been sufficient, I decided to take the time to see what it would take to add multicast DNS and service-discovery to the Plan 9 and Inferno DNS services. Multicast DNS and Service Discovery (aka Bonjour, aka Rendezvous, aka Zeroconf) is documented in plenty of places, here's a few good starting points for more information:

Multicast DNS resolution is fairly straightforward, it involves just sending the DNS request to a multicast address (224.0.0.251) and using port 5353 instead of port 53. The first thing I did was modify /appl/cmd/ndb/dns.b to use this address and port when looking for any domain ending in .local per the zeroconf convention.

Service discovery is a bit more problematic. It involves sending a slightly different type of request. Typical DNS requests use an type A style request which retrieves an IP address for a hostname. Service discovery uses a PTR style request which returns three types of records -- the PTR record contains the instance of the service which will include a more specific name of the service location, a SRV record which will contain the port number of the service, a TXT record which contains some protocol specific information and an A response which contains the IP address.

So, for example, if I send a PTR request for _presence._tcp.local, I get four records in response:

  1. ptr record -> crazyjim@arl137._presence._tcp.local
  2. srv record -> port: 5298
  3. txt record -> last=Peterson 1st=James msg=Away status=dnd
  4. A record -> 9.3.61.137

I added a flag to ndb/dnsrequest (-z) which forces sending a PTR request to DNS. Using the Plan 9 DNS service as a model, I modified the Inferno dns to be able to parse SRV and TXT records. The one things I changed was that the TXT response can contain several key value pairs. The Plan 9 DNS service just strings these together (with no seperator mind you). So I print the number of key value pairs, and then have the key=value pairs one per line prefixed with dual tabs to make it look nice.

The final remaining problem is that the current DNS daemon only returns a single response for each request, but multicast DNS may have several responses for a single request. This involves a much larger set of changes to dns.b. DNS will now accumulate responses and return them in one big set.

I probably need to do some work to get cs.b to play nice with such information. The whole dnsquery, and even the cs and ndb front-ends to it seem decidedly anti-Plan 9 in their layout -- particularly for something with lots of rich attribute information like zeroconf. As such I'm likely going to create a synthetic file server with which to browse zeroconf data. You register which types of zeroconf entities you are interested in by creating directories in a two level hierarchy -- the file server will then use my modified DNS to query the local net and will create nodes under those directories for responses with attribute information broken out into individual files. The other major thing that needs to be done is adding multicast server support to dns.b. But I think I'll write these up in a different lab entries as this one is getting long in the tooth already.

While this is sufficient to solve my initial problems, there are several additional aspects of zeroconf which might be nice to integrate for Inferno including support for link-local IP addresses (which I guess would only be important on native), proper uniqueness handling for claiming your local names, NAT-PMP support, DNS-LLQ support, and dynamic DNS updates.

EXAMPLE

% ./dnsquery -z _presence._tcp.local
_presence._tcp.local ptr ericvh@ericvh-desktop._presence._tcp.local
ericvh@ericvh-desktop._presence._tcp.local txt 10
txtvers=1
1st=Eric
last=Van Hensbergen
port.p2pj=5298
status=away
node=libpurple
ver=2.5.2
vc=!
email=bergevan@us.ibm.com
phsh=943420112a8b192466a802bedfe547041a62ea90

ericvh@ericvh-desktop._presence._tcp.local srv 0 0 5298 ericvh-desktop.local
ericvh-desktop.local ipv6
ericvh-desktop.local ip 9.3.61.77
npe@macintosh-16._presence._tcp.local srv 0 0 5298 macintosh-16.local
npe@macintosh-16._presence._tcp.local txt 13
ext=
phsh=f308675309a23fa653c269c95f57eb7eb84efc44
last=
AIM=
nick=
1st=Noah
port.p2pj=5298
txtvers=1
version=1
node=
jid=
email=
status=avail

macintosh-16.local ipv6
macintosh-16.local ip 9.3.61.73
_presence._tcp.local ptr npe@macintosh-16._presence._tcp.local

DISCUSSION

dns.b seems entirely too big, and I just made it bigger. It would seem better served if it were split up into a bunch of component modules. It seems like the marshalling and unmarshalling of DNS messages is a legitimate module, the cache is a module, local database/config access is another, and then a proper module interface for performing DNS queries and/or servicing DNS requests. The file and network servers could then be provided relatively cleanly. All in all it would clean the code up signifigantly and make the whole thing a lot more readable/extensible. Things like the registry or even cs could easily be implemented as plug-in modules versus discreet file servers (although also allowing them to use file services is desirable in certain scenarios so this should definitely be parameterized).

Its funny, looking back at Virgild, it was essentially a broadcast form of multicast name resolution, just with its own more simple protocol instead of DNS.

FILES

  • lab90/appl/cmd/ndb/dns.b
  • lab90/appl/cmd/ndb/dnsquery.b

Labels: ,

Thursday, May 01, 2008

NAME

lab 88 - degrees of freedom

NOTES

The Vitanuova downloads page has historical snapshots of Inferno source from 1996 to 2003 containing all three editions before Inferno went open source. I was curious to see how well Inferno has sustained a standard set of interfaces over the last ten years so I downloaded all of them and poked around.

The biggest overall change came with 4th edition, when many parts of the system were upgraded, including the Styx protocol, several builtin modules, dis format, the limbo language and VM. Also, significantly, Inferno adopted open source licenses granting developers the freedom to modify any part of the system: something that might impact sustainability for good or ill.

While every edition prior to 4th has had some interfaces changed, these changes did not break backwards compatibility. A 3rd edition emu can run dis code from the 1st edition archive.

The difference between 3rd and 4th was large enough that limbo code needed to be ported, or at the very least recompiled, to run on the new emu.

The Sys interface is evolving still with additions made since 4th edition was released. These types of changes, adding a function or a new constant, do not break backward compatibility, but in a network of emus where there is diversity of versions, link typechecks do fail when a module is expecting an interface newer than the one available. Where is the standard interface here? This problem seems to violate some of the core ideas of Inferno, and Inferno doesn't provide an easy way of working around compatibility issues with builtin modules.

Inferno's core idea is to provide standard interfaces that free content and service providers from concern of the details of diverse hardware, software, and networks over which their content is delivered. (/sys/doc/bltj.ms)

The BLTJ paper describing Inferno listed the several dimensions of portability and versatility provided by the OS,

  • Portability across processors: it currently runs on Intel, Sparc, MIPS, ARM, HP-PA, and PowerPC architectures and is readily portable to others.
  • Portability across environments: it runs as a stand-alone operating system on small terminals, and also as a user application under Windows NT, Windows 95, Unix (Irix, Solaris, FreeBSD, Linux, AIX, HP/UX) and Plan 9. In all of these environments, Inferno applications see an identical interface.
  • Distributed design: the identical environment is established at the user's terminal and at the server, and each may import the resources (for example, the attached I/O devices or networks) of the other. Aided by the communications facilities of the run-time system, applications may be split easily (and even dynamically) between client and server.
  • Minimal hardware requirements: it runs useful applications stand-alone on machines with as little as 1 MB of memory, and does not require memory-mapping hardware.
  • Portable applications: Inferno applications are written in the type-safe language Limbo, whose binary representation is identical over all platforms.
  • Dynamic adaptability: applications may, depending on the hardware or other resources available, load different program modules to perform a specific function. For example, a video player application might use any of several different decoder modules.

Now that we have a decade of Inferno history, how many of the above degrees of freedom still hold when the whole time span is considered as one network of interconnected emus?

Don't standard interfaces also imply standard across time? A network of emus can not be expected to upgrade all at the same time. A standard is also a constraint against change in an interface. One degree of freedom is expressly limited; keep the abstraction constant.

The dilemma faced is whether to freeze an interface to provide long term compatibility based on a standard, but risk the possibility of being held back from adopting new ideas and becoming irrelevant, or to keep changing interfaces to solve new problems but pay the cost of compatibility problems.

In general it seems filesystems, namespaces and textual interfaces all lend well to creating a sustainable software environment. However, the more complex limbo language and module interfaces have shown themselves to be not so well preserved. By comparison, maybe unfairly because of the different goals of the creators, see how the works of Knuth are intended to withstand time. He specifically structures his software so that incompatibilities do not creep in, such as leaving no undefined gaps in font tables so that no one is tempted to fill them (TeX Book), or defining instructions to fill all 256 possible slots in MMIX, and making his source readable but not permitting edits except through his CWEB change file system. Knuth's software is the only kind I know of that take seriously the problem of long term compatibility.

It would be nice to run 1st edition dis code in a current emu if for no other reason than to prove the sustainability of infernos standard interfaces, but a major barrier to that is the need to bind in old Sys and Draw modules. I assume that compatibility to older interfaces should be provided through limbo modules so that the emu doesn't bear the extra weight and complexity of carrying multiple builtin implementations. There is no way to override where a builtin module is loaded from, though this might be a nice feature. For example, if /dev/dis/draw.dis represented the builtin draw module, I might bind limbo implementation over it, so that a load Draw "$Draw", would take the compatibility simulation over the builtin. (This might also work nicely going the other way so that we could bind builtin modules over limbo modules for optimization.)

This is not possible for Sys however, because we can't simulate variadic args in limbo (e.g., sys->print). A solution to this would be nice! But an alternative to providing more mechanism is simply to freeze the various interfaces.

Sunday, April 06, 2008

NAME

lab 87 - mux for nintendo ds

NOTES

In an earlier post I talked about updating mux to 4th edition Inferno in the hope of one day running it on a Nintendo DS.

Well, Inferno is now booting on the DS so I got to try it for real.

I started with getting the mux window manager working in standard inferno. Then I changed the resolution down to 256x192 and tried to get everything to fit. The files in this lab include the version of mux I ended up putting in the nds file running on the DS.

Things to try if you download it. Rocker moves up and down selection. 'A' key enters, 'B' key backs out back up to the higher level. 'Start' key returns to the top level menu.

Try Today's Newspaper, and The Thisburgh has the only working graphic. Under news, click through to actually read an article. Under games, try connect4. Audio control would look cool if any of the graphics actually came in. The Financial Reports gives a ticker. It scrolls slowly only because of the sleep interval in the code is incorrect.

If you want to try this version of mux using hosted inferno just remember you need to compile prefab into your emu. Include prefab in the mod and lib sections of your emu config file, also uncomment prefab in the /libinterp/mkfile.

Mux uses irsim for key controls. I changed my local inferno-ds code to have the DS keys output the same characters as used by irsim.

The files in this lab include the movies and tvlist apps and their data. The data didn't fit on the 4MB .nds file. But they will fit when we get the GBA ROM or dldi interface working.

I think mux is a good path to follow for DS development. It's small, starts quickly, uses the keys effectively since it was designed for remote controls, the programs are easy to understand, and they hit most of the applications I'd like to start with, small games, news reader, email reader, simple database browser (movies, tvlist), and audio.

FILES

code for lab 87

NAME

lab 86 - srv

NOTES

This from a post on 9fans, and also on Tip O' the Day

% dc >[0=1] | echo 0 > /srv/desk

Plan 9's srv(3) acts as a bulletin board for open file descriptors, other namespaces see all the files in srv, and so can read and write to /srv/desk.

Inferno has srv(3) which is a file2chan registry, but is also visible to all namespaces on the host. (see also srv9(3))

The current implementation of sh-file2chan(1) does not allow the above. The closest I got was,

% load file2chan
% calc >[0=1] | {file2chan /chan/desk {rblock; putrdata &} {fetchwdata > /fd/0}} &

% stream -b 1 /chan/desk
% echo 1+1 > /chan/desk

I tried implementing a command equivalent to srv(4) on Plan 9. It takes a command block or network address and post it in the srv registry.

% srv {calc >[1=0]}  /chan/desk

It using an existing '#s' instance if there is one, else binds a new one. Now we can open a console to /chan/desk from another window

% {cat & cat  >[1=0] < /dev/cons} <> /chan/desk

and other windows can write to /chan/desk, the output will be seen in the console.

Questions.

  1. Why isn't Plan 9 srv(3) in Inferno?
  2. File2chan(2) seems under used. Is that because of the shell interface sh-file2chan?
  3. Is there another interface that would make file2chan more usable?
  4. Mount(1) supports mounting from a file, as Plan 9's does. But the inferno srv(3) device must do extra copies of the read and write buffers to implement the interface. Is the file interface of Plan9 srv more elegant than the extra file2chan syscall in Inferno?
  5. Aren't there benefits to using channels in Inferno that make file2chan preferable?

Files in this lab are for the inferno srv(4) implementation.

FILES

lab 86 code

Monday, March 24, 2008

NAME

lab 85 - stowage

NOTES

In an earlier post I defined a venti-lite based on two shell scripts, getclump and putclump, that stored files in a content addressed repository, which in that instance was just an append-only gzip tar archive with an index.

After learning a little about the git SCM, this lab re-writes those scripts to use a repository layout more like git's. The key thing to know about the git repository is that it uses sha1sum(1) content addressing and that it stores the objects as regular files in a filesystem using the hash as the directory and filename,

  objects/hh/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh

In the objects directory is 256 directories named for every 2 character prefix of the sha1hash of the object. The filename is the remaining 38 characters of the hash.

Putclump calculates the hash, slices it to make the prefix and filename, tests if the file already exists, and if not writes the compressed data to the new file. Here is the important part of putclump,

 (sha f) := `{sha1sum $file}
 (a b) := ${slice 0 2 $sha} ${slice 2 40 $sha}
 
 if {ftest -e $hold/objects/$a/$b} {} {
  mkdir -p $hold/objects/$a
  gzip < $file > $hold/objects/$a/$b
 }

Getclump just needs to look up the file given a hash

 sha := $1
 (a b) := ${slice 0 2 $sha} ${slice 2 40 $sha}
 files := `{ls $hold/objects/$a/$b^* >[2] /dev/null}
 if {~ $#files 1} {gunzip < $files } 

Because the git repository uses a regular file system to store objects, it makes it considerably easier to work with than the compacted file system like tar.gz, or an application specific binary format like venti. This is because instead of having to create new tools to read and write binary formats, we can re-use existing tools, like sh(1), tarfs(4), updatelog(8), and applylog(8).

For example, I wrote a script, stow, that takes a tarball and stores it in my repository, called the hold. The hold should be created first with the following directories,

 /n/hold/logs
 /n/hold/objects
 /n/hold/stowage

Then give stow the name of a .tar or .tgz file. Files not found in the hold and that were added are printed to stdout.

 % stow acme-0.11.tgz
 ...
 %

Stow uses updatelog(8) to create a stowage manifest file for the tarball I added. This manifest is saved under /n/stowage. The manifest records the pathname, perms, and sha1 hash of every file in the tarball.

Now that I've stowed all my tarballs I need a way of getting things out.

I built a holdfs, derived from tarfs(4), to read the stowage manifest and present files from the hold. By default the file system is mounted on /mnt/arch.

 % holdfs /n/hold/stowage/acme-0.11

The hold with its stowage is be a step up from a directory tarpit of tarballs. I can accumulate a version history based on tar.gz releases like that for acme-sac and inferno. The vitanuova downloads site contains inferno history going back to 1997. My downloads page contains snapshots of inferno from 2002 to 2006 and acme-sac after that.

My intended application for this was that I could encourage forks of a project and merge back many individuals releases into a single repository and still do useful comparisons.

Using a filled hold I should be able to do analysis of a file history based on the stowage manifests. Contained in this lab are a few experimental scripts to build out more of an SCM. For example, the script hold/diff attempts to use updatelog to compare a manifest with the current tree. And hold/difflog uses a modified applylog(8) to compare two manifests.

The nautical references suggest a distributed and loosely coupled network like that of shipping, and is also influenced by git's design. The unit of transfer is a tarball. It is stowed into the ships hold, along with the manifest. A file system interprets the manifests and gives an interface for searching the hold. There is also keep a ships log of what was stowed and when. I can extract patches, files, tarballs, or the complete stowage in my hold to share with someone else.

This is a simple system:

     28 putclump.sh
     16 getclump.sh
     54 stow.sh
    669 holdfs.b
    767 total

But then most of what was needed already existed in inferno.

FILES

lab 85 code