| 1 |
BACKGROUND |
| 2 |
Frederik Eaton ([email protected]) |
| 3 |
2003/3/26 |
| 4 |
|
| 5 |
ORIGINS |
| 6 |
|
| 7 |
AVFS originated as a shared library which was intended to intercept |
| 8 |
shared library calls from an executable to glibc, overriding a basic |
| 9 |
set of the glibc filesystem-related functions to give transparent |
| 10 |
access to a number of "virtual" filesystem modules. Most of the |
| 11 |
modules are intended to provide access to files stored in various |
| 12 |
types of archives, but there are several which are used to access |
| 13 |
files remotely, via common protocols such as ssh or ftp, and several |
| 14 |
others which don't fall into either category. |
| 15 |
|
| 16 |
MC-VFS |
| 17 |
|
| 18 |
The design of avfs is based on that of mc-vfs, a library which was |
| 19 |
created to enable Midnight Commander to use virtual filesystem modules |
| 20 |
like the ones described above. MC-VFS consists of replacements for the |
| 21 |
basic file operations (namely open, close, read, write, opendir, |
| 22 |
readdir, closedir, telldir, seekdir, stat, lstat, fstat, chmod, chown, |
| 23 |
utime, readlink, symlink, link, unlink, rename, chdir, lseek, |
| 24 |
mknod). The functions provided by mc-vfs are designed to work as |
| 25 |
drop-in replacements for the corresponding glibc functions, making it |
| 26 |
easier to modify an existing application to use them. |
| 27 |
|
| 28 |
This design goes much farther than facilitating better graphical file |
| 29 |
browsers or more convenient utilities - by linking a file server such |
| 30 |
as NFS to mc-vfs, one can export an mc-vfs virtual filesystem through |
| 31 |
the kernel as a standard kernel filesystem, making it available to |
| 32 |
every user on the system. |
| 33 |
|
| 34 |
USERVFS |
| 35 |
|
| 36 |
Pavel Machek's uservfs (originally podfuk) was to my knowledge the |
| 37 |
first project to take advantage of this idea. Early versions of his |
| 38 |
program consisted of patches to NFS, but because of problems with |
| 39 |
kernel deadlock and the way NFS allocates filehandles (you'll have to |
| 40 |
ask him, I don't understand), he switched to Coda. Neither NFS nor |
| 41 |
Coda is ideal as the kernel side of a userland virtual file system - |
| 42 |
NFS because rpc.nfsd is single threaded and has a deadlock problem |
| 43 |
when used with virtual filesystems, Coda because, among other things, |
| 44 |
its protocol makes random access to files impossible. (Another |
| 45 |
project, Jeremy Fitzhardinge's userfs, solved these problems by |
| 46 |
providing its own kernel filesystem, but the kernel code for that was |
| 47 |
never brought up to date with newer kernels, and instead of using an |
| 48 |
API similar to mc-vfs, it had an interface closer to the kernel VFS |
| 49 |
interface which made it difficult to write new modules for it.) |
| 50 |
|
| 51 |
AVFS-PRELOAD |
| 52 |
|
| 53 |
The original version of AVFS, avfs-preload, took a different approach |
| 54 |
to serving userland filesystems. It was modeled after a program |
| 55 |
called zlibc, which is supposed to transparently compress and |
| 56 |
uncompress your files for you. Referred to as "the LD_PRELOAD hack", |
| 57 |
the method used by zlibc, and later avfs-preload, circumvented the |
| 58 |
kernel completely, avoiding the shortcomings of existing kernel |
| 59 |
filesystems and allowing users without root access to install and use |
| 60 |
the programs themselves. |
| 61 |
|
| 62 |
To use the original version of avfs, the user would simply set the |
| 63 |
LD_PRELOAD environment variable to point to the avfs shared |
| 64 |
library. Then, when he started an executable from that shell, the |
| 65 |
linker would load libavfs.so before glibc. Libavfs.so overrode a few |
| 66 |
key glibc functions with avfs' own virtual replacements, modifying the |
| 67 |
executable's view of the root filesystem to include avfs' virtual |
| 68 |
modules. |
| 69 |
|
| 70 |
Since nearly all programs which make system calls do so through glibc, |
| 71 |
the preload method was compatible with almost everything and worked |
| 72 |
quite well. Although the preload method is no longer preferred for a |
| 73 |
number of reasons, it is still possible to compile avfs to use it. |
| 74 |
|
| 75 |
PROBLEMS WITH THE PRELOAD METHOD |
| 76 |
|
| 77 |
Why did the preload method fall out of favor? There are two (??) main |
| 78 |
problems with it. First, for some reason newer versions of glibc |
| 79 |
(>2.1??) made certain key functions private so they couldn't be |
| 80 |
overridden by programs like avfs. Since these functions were used |
| 81 |
internally by many other public glibc functions, fixing avfs to work |
| 82 |
with the newer glibc would have meant overriding an impractical number |
| 83 |
of additional glibc calls. (Note: Solaris libc still works with |
| 84 |
avfs-preload, it is only Linux/glibc which has this problem) The |
| 85 |
second reason is that, for all the cleverness behind the original |
| 86 |
LD_PRELOAD hack, there are still some things that are impossible |
| 87 |
without a real kernel filesystem - the obvious one is making your |
| 88 |
filesystem visible from within the kernel, but there are other issues, |
| 89 |
such as keeping virtual file descriptors open over exec(). (because |
| 90 |
avfs-preload can't do this, it was never able to handle shell |
| 91 |
redirections such as "gzip < virtual_file"; you had to do "cat |
| 92 |
virtual_file | gzip" instead) XXX this paragraph is out of date and |
| 93 |
needs to be updated. |
| 94 |
|
| 95 |
AVFSCODA |
| 96 |
|
| 97 |
The current replacement for the preload method is similar to uservfs - |
| 98 |
called avfscoda, it exports a real kernel filesystem using Coda, |
| 99 |
reading commands from the kernel Coda module through the /dev/cfs0 |
| 100 |
character device. To use avfscoda you must start avfscoda, mount |
| 101 |
/dev/cfs0 on /overlay, and install a special kernel module redir.o |
| 102 |
which redirects failed lookups in your root filesystem to /overlay (so |
| 103 |
you can pretend that the virtual files you see are actually on your |
| 104 |
root filesystem together with the other files). Redir has been tested |
| 105 |
on 2.2.x and 2.4.x kernels, and is unlikely to become incompatible |
| 106 |
with future versions since it works by patching the system call table, |
| 107 |
an interface which tends to be fairly static. |
| 108 |
|
| 109 |
OTHER VFS LIBRARIES |
| 110 |
|
| 111 |
Because the code for libavfs - the modules and vfs infrastructure - |
| 112 |
comprise the bulk of the project, we can say that avfs is first and |
| 113 |
foremost a virtual filesystem library. So we should take the |
| 114 |
opportunity to mention some of its cousin vfs libraries here: |
| 115 |
|
| 116 |
mc-vfs |
| 117 |
|
| 118 |
Also see above. mc-vfs is a predecessor to avfs, and is currently used |
| 119 |
in mc and gmc. (?? is it being actively developed?) (?? which avfs |
| 120 |
modules other than extfs are inherited from mc-vfs?) |
| 121 |
|
| 122 |
gnome-vfs |
| 123 |
|
| 124 |
- foundation for nautilus (Gnome's new "second generation" file |
| 125 |
manager), but intended to be used by other gnome apps as well |
| 126 |
- extensible via shared library modules (like avfs) |
| 127 |
- unlike avfs, treats 'stackable' modules like 'tar' separately from |
| 128 |
root modules like 'ftp'; protocols such as ftp, http, smb are |
| 129 |
specified in URI's, while stackable tar, gzip are invoked with the |
| 130 |
familiar '#'. |
| 131 |
|
| 132 |
Example: "http://server/path/to/file.tar#tar/path/to/file.gz#gzip" in |
| 133 |
gnome-vfs would be |
| 134 |
"/#http:server|path|to|file.tar#utar/path/to/file.gz#ugz" in avfs |
| 135 |
|
| 136 |
- unlike avfs, gnome-vfs' API is not intended to be interchangeable |
| 137 |
with glibc's: |
| 138 |
- gnome-vfs refers to files with a pointer to GnomeVFSHandle, rather |
| 139 |
than using integer file descriptors; |
| 140 |
- gnome-vfs has extra functions implementing asynchronous I/O (by |
| 141 |
spawning helper threads or processes) |
| 142 |
|
| 143 |
- last I checked, gnome-vfs had very little documentation |
| 144 |
|
| 145 |
kio |
| 146 |
|
| 147 |
Something like a KDE version of gnome-vfs?? Was unable to find any |
| 148 |
documentation of it. |