BACKGROUND Frederik Eaton (frederik@ugcs.caltech.edu) 2003/3/26 ORIGINS AVFS originated as a shared library which was intended to intercept shared library calls from an executable to glibc, overriding a basic set of the glibc filesystem-related functions to give transparent access to a number of "virtual" filesystem modules. Most of the modules are intended to provide access to files stored in various types of archives, but there are several which are used to access files remotely, via common protocols such as ssh or ftp, and several others which don't fall into either category. MC-VFS The design of avfs is based on that of mc-vfs, a library which was created to enable Midnight Commander to use virtual filesystem modules like the ones described above. MC-VFS consists of replacements for the basic file operations (namely open, close, read, write, opendir, readdir, closedir, telldir, seekdir, stat, lstat, fstat, chmod, chown, utime, readlink, symlink, link, unlink, rename, chdir, lseek, mknod). The functions provided by mc-vfs are designed to work as drop-in replacements for the corresponding glibc functions, making it easier to modify an existing application to use them. This design goes much farther than facilitating better graphical file browsers or more convenient utilities - by linking a file server such as NFS to mc-vfs, one can export an mc-vfs virtual filesystem through the kernel as a standard kernel filesystem, making it available to every user on the system. USERVFS Pavel Machek's uservfs (originally podfuk) was to my knowledge the first project to take advantage of this idea. Early versions of his program consisted of patches to NFS, but because of problems with kernel deadlock and the way NFS allocates filehandles (you'll have to ask him, I don't understand), he switched to Coda. Neither NFS nor Coda is ideal as the kernel side of a userland virtual file system - NFS because rpc.nfsd is single threaded and has a deadlock problem when used with virtual filesystems, Coda because, among other things, its protocol makes random access to files impossible. (Another project, Jeremy Fitzhardinge's userfs, solved these problems by providing its own kernel filesystem, but the kernel code for that was never brought up to date with newer kernels, and instead of using an API similar to mc-vfs, it had an interface closer to the kernel VFS interface which made it difficult to write new modules for it.) AVFS-PRELOAD The original version of AVFS, avfs-preload, took a different approach to serving userland filesystems. It was modeled after a program called zlibc, which is supposed to transparently compress and uncompress your files for you. Referred to as "the LD_PRELOAD hack", the method used by zlibc, and later avfs-preload, circumvented the kernel completely, avoiding the shortcomings of existing kernel filesystems and allowing users without root access to install and use the programs themselves. To use the original version of avfs, the user would simply set the LD_PRELOAD environment variable to point to the avfs shared library. Then, when he started an executable from that shell, the linker would load libavfs.so before glibc. Libavfs.so overrode a few key glibc functions with avfs' own virtual replacements, modifying the executable's view of the root filesystem to include avfs' virtual modules. Since nearly all programs which make system calls do so through glibc, the preload method was compatible with almost everything and worked quite well. Although the preload method is no longer preferred for a number of reasons, it is still possible to compile avfs to use it. PROBLEMS WITH THE PRELOAD METHOD Why did the preload method fall out of favor? There are two (??) main problems with it. First, for some reason newer versions of glibc (>2.1??) made certain key functions private so they couldn't be overridden by programs like avfs. Since these functions were used internally by many other public glibc functions, fixing avfs to work with the newer glibc would have meant overriding an impractical number of additional glibc calls. (Note: Solaris libc still works with avfs-preload, it is only Linux/glibc which has this problem) The second reason is that, for all the cleverness behind the original LD_PRELOAD hack, there are still some things that are impossible without a real kernel filesystem - the obvious one is making your filesystem visible from within the kernel, but there are other issues, such as keeping virtual file descriptors open over exec(). (because avfs-preload can't do this, it was never able to handle shell redirections such as "gzip < virtual_file"; you had to do "cat virtual_file | gzip" instead) XXX this paragraph is out of date and needs to be updated. AVFSCODA The current replacement for the preload method is similar to uservfs - called avfscoda, it exports a real kernel filesystem using Coda, reading commands from the kernel Coda module through the /dev/cfs0 character device. To use avfscoda you must start avfscoda, mount /dev/cfs0 on /overlay, and install a special kernel module redir.o which redirects failed lookups in your root filesystem to /overlay (so you can pretend that the virtual files you see are actually on your root filesystem together with the other files). Redir has been tested on 2.2.x and 2.4.x kernels, and is unlikely to become incompatible with future versions since it works by patching the system call table, an interface which tends to be fairly static. OTHER VFS LIBRARIES Because the code for libavfs - the modules and vfs infrastructure - comprise the bulk of the project, we can say that avfs is first and foremost a virtual filesystem library. So we should take the opportunity to mention some of its cousin vfs libraries here: mc-vfs Also see above. mc-vfs is a predecessor to avfs, and is currently used in mc and gmc. (?? is it being actively developed?) (?? which avfs modules other than extfs are inherited from mc-vfs?) gnome-vfs - foundation for nautilus (Gnome's new "second generation" file manager), but intended to be used by other gnome apps as well - extensible via shared library modules (like avfs) - unlike avfs, treats 'stackable' modules like 'tar' separately from root modules like 'ftp'; protocols such as ftp, http, smb are specified in URI's, while stackable tar, gzip are invoked with the familiar '#'. Example: "http://server/path/to/file.tar#tar/path/to/file.gz#gzip" in gnome-vfs would be "/#http:server|path|to|file.tar#utar/path/to/file.gz#ugz" in avfs - unlike avfs, gnome-vfs' API is not intended to be interchangeable with glibc's: - gnome-vfs refers to files with a pointer to GnomeVFSHandle, rather than using integer file descriptors; - gnome-vfs has extra functions implementing asynchronous I/O (by spawning helper threads or processes) - last I checked, gnome-vfs had very little documentation kio Something like a KDE version of gnome-vfs?? Was unable to find any documentation of it.