Note: If you are reading this document because it has been included in
the docs directory of a precompiled binary package, then skip down a
couple of paragraphs to TEST and start there - you don't need to read
the BUILD and QUICK INSTALL instructions.



BUILD instructions

It should be sufficient to type "make config all" in this directory.
But check first

  0) that you have the kernel sources installed 

  1) that the kernel source directory LINUXDIR in the Makefile is correct

  2) that you (a) set SMP=1 in the Makefile if your target kernel is SMP;
     and (b) that there isn't an old config.cache hanging around in the nbd
     subdirectory when you run the make config.

In addition, you must have [open]ssl headers and libs installed if you
plan to compile with USING_SSL=1. If you don't understand what SSL
means, just ignore this paragraph.

The make will build enbd.o, enbd-server, enbd-client in /tmp.  Change BUILD
in the Makefile to put the build directory somewhere else.



QUICK INSTALL instructions

Place enbd-server and enbd-client in /usr/local/sbin and put their manual
pages in /usr/local/man/man8.  "make install" will do this part of the
job automatically (see INSTALL below).

You will need to place the module enbd.o yourself.  A possibility is to
place it in the misc directory of your /lib/modules/x.y.z/ hierarchy.
To use it you must not have kernel nbd already compiled into your
kernel, nor already loaded as a module.

You will also need the /dev/nda nda1 nda2 ... /dev/ndb ndb1 ndb2 ...
entries. Make them with the included MAKEDEV script.



TEST instructions

If you want to try a test, use "make test". But first ...

  3) you will need sudo and ssh installed on both SERVER and CLIENT -
     I can't generically guarrantee remote root access any other way.
     Go get 'em!

  4) edit the Makefile and replace SERVER and CLIENT with the name of your
     build machine and a willing target machine respectively.  "localhost"
     will do for both.

  5) the /dev/ndxN files must exist on the client for the test to
     work, though in the more general context you can set up alternative
     naming schemes, and indeed if you have devfs support in the kernel
     you will get an alternative (compatible) scheme anyway.
     
I've provided a script called MAKEDEV to make the standard non-devfs
device naming scheme.  With any luck it'll be made on demand when you
run the test.  If not, on the client, do sh MAKEDEV using the supplied
MAKEDEV.  Be careful ...  there is already a script called MAKEDEV in
/dev.  You should end up with

        /dev/nda /dev/nda1 /dev/nda2 /dev/nda3 ..
        /dev/ndb /dev/ndb1 /dev/ndb2 /dev/ndb3 ..
        ..

(spot the difference: the kernel nbd uses /dev/nb{1,2,..} )

The target must be running a kernel into which the enbd.o module will load
(so you will have to figure out what that means and make it so - running
a kernel built from the build machines sources should be enough). The
target can be the same as the build machine and setting both SERVER and
CLIENT to "localhost" is a safe bet.

  6) run "make test".

The test should set up a small file in /tmp and serve it to the client
machine as the clients enbd device. That'll be /dev/nda. You can make a
file system on it (use -o sync when you mount it) and play. Look at
/proc/nbdinfo.

To do this test by hand. try some variant on:

    (make a pair of files /tmp/core{0,1} of 4MB or so each)
    server: /tmp/enbd-server 12345 /tmp/core?
    client: insmod /tmp/enbd.o
    client: /tmp/enbd-client <server>:12345 -n 2

To stop the test, you can try running "make stop". And possibly
also "make rescue" in case of difficulties. I don't guarantee a
rescue in all circumstances, but it'll try, and you can elaborate
the Makefile to suit your circumstances. To stop the test by hand,
"killall enbd-server enbd-client" and "rmmod enbd".



DON'T PANIC!

If you get into trouble at any point, try "echo 0 > /proc/nbdinfo".
That kills all kernel requests pending on the device, and also holds the
device off for 5 seconds afterwards.  Replace "0" with "1" for an even
brusquer effect, but be careful with that - it also clears the kernels
reference count so that you can rmmod it even if some process is still
using it.  That will probably cause a minor kernel oops.



COMMON GOTCHAS

Note that when running *localhost-to-localhost* the client is subject to
a VFS to VFS deadlock in the kernel when you write a file larger than
ram to the device, so be careful there too.  You should mount the file
system -o sync to help avoid it.

If you care to know, the device can't do anything about the
localhost-to-localhost write deadlock, as it happens before the enbd
driver ever gets anything to do.  VFS fills up with blocks not sent to
the device, and then tries to flush them to the device, which activates
the client, which writes to the server, and then VFS needs yet more
blocks as the server tries to write to disk; it's a generic kernel
problem.  You could live dangerously and try the "-a" flag on the
client, which makes the driver asynchronously ack the kernel before
receiving an ack from the server for each request, which will help.
Running across the net avoids the deadlock completely.

If you do get into deadlock to localhost, remember that "echo -n 0 >
/proc/nbdinfo" should error out the device for five seconds, and thereby
release enough VFS blocks to get you out of jail.

Note also that the device holds an internal signature, generated
by the server at first contact. This makes it hard for you if
you kill the server and  try reconnecting, because the new server will
generate a new random signature, and it won't match the one already
in the device. Result - puzzlement, unless you read this. Use an
explicit "-i foobar" option at the server end, so that it uses the
same signature each time. It's not aimed at frustrating you, honest!
It's an antispoofing device.



INSTALL instructions

  7) edit the makefile and check the PREFIX variable

  8) run "make install"

Beware: "make install" now insidiously and perniciously adds two services
enbd-{s,c}statd to /etc/services, and a line to /etc/inetd to start them
on demand.  Comment these lines if you don't like them.  You'll lose
some of the functionality that allows connections to survive across
reboots, and that's "all".  Commenting them out is sufficient to stop
subsequent installs from reestablishing them.

At this point I should tell you what the "make install" puts where. To
the best of my knowledge, the list is:

   /usr/local/sbin/enbd-{server,client,statd,cstatd,test}
   /usr/local/man/man8/enbd-{server,client,statd,cstatd,test}.8
   /usr/local/man/man5/enbd.conf.5
   /usr/local/doc/enbd/{INSTALL,README}

You'll need to look after the enbd.o module yourself.

SETUP and configuration instructions.

The best way to set up is to look in the nbd/etc subdir of the
distributed archive.  There you will find a /etc/enbd.conf and an
/etc/init.d/enbd file.  Write the details of the server or client (or
both) that should run on the local host in the enbd.conf file, as
indicated there and on its manpage, and then simply run /etc/init.d/enbd
start.  Do the same on the remote machine.

The init.d/enbd script understands "start" and "stop" and other commands
too.  You can say "start server", and "start server foo", if foo is the
label of one of the server lines in /etc/enbd.conf.  You should link the
script into your sysv init system, which probably means creating links
to it below /etc/rc*.d/ (this is distribution dependent).


FURTHER READING

Matthew Sackman has written a very nice howto on setting ENBD up under
heartbeat. See http://welquite.org/redundant/. I've taken his scripts,
modified them for my tastes, and included them in the ./nbd/etc/ha.d/
subdirectory of the distributed archive. Please read (the comments
inside are quite helpful) and improve them, and pass the improved
versions back.

I've also added a chapter on failover to the online documentation
at http://www.it.uc3m.es/ptb/enbd/ .


Peter T. Breuer (ptb@it.uc3m.es)
