Debian Codegraph @

doc/api-overview

raw
1 AVFS API Overview
2 Frederik Eaton ([email protected])
3 2003/3/26
4
5 Writing a module
6
7 There are several different ways to write a new filesystem module for
8 avfs. If you only need basic functionality and you don't care about
9 efficiency, you might want to look at extfs. extfs allows you to
10 implement simple filesystem modules as short, standalone scripts. It
11 is documented in extfs/README.extfs. If extfs isn't good enough, then
12 you can use the low-level C API, or one of the helper libraries built
13 on top of it. There are currently four such libraries: 'filter', for
14 modules which act on single files (see ubzip2.c); 'archive', for
15 archive-style modules (see uar.c); 'remote' for things like ssh, rsh,
16 and ftp (see rsh.c); and 'state' which exports a procfs-like "get/set"
17 filesystem (see ftp_init_ctl in ftp.c). An example of a module built
18 directly on the low-level API is 'volatile', in modules/volatile.c.
19
20 The final source of information about each of the different interfaces
21 is always the avfs source code. This document tries to give a
22 high-level overview which should help make the code more
23 understandable.
24
25 The low-level interface
26
27 Data structures
28
29 Let's start with ventry and vfile. These are somewhat analogous to the
30 Linux kernel's 'inode' and 'file' structures, respectively.
31
32 struct ventry {
33 void *data;
34 struct avmount *mnt;
35 };
36
37 struct vfile {
38 void *data;
39 struct avmount *mnt;
40 int flags;
41 int ptr;
42 avmutex lock;
43 };
44
45 A ventry just represents a path in the file system. You can convert
46 from a path to a ventry using av_get_ventry (internal.h). In order to
47 read or write to the corresponding file, one must obtain a vfile by
48 "opening" the ventry with av_open (oper.h). This calls the 'open'
49 method of the appropriate filesystem module. This method, along with
50 many others, is stored as a function pointer in an object of type
51 'struct avfs'. 'struct avfs' is the data structure we use to define a
52 module, something like the Linux kernel's super_operations,
53 inode_operations, and file_operations put together:
54
55 /* from src/avfs.h */
56 struct avfs {
57 /* private */
58 struct vmodule *module;
59 avmutex lock;
60 avino_t inoctr;
61
62 /* read-only: */
63 char *name;
64 struct ext_info *exts;
65 void *data;
66 int version;
67 int flags;
68 avdev_t dev;
69
70 void (*destroy) (struct avfs *avfs);
71
72 int (*lookup) (ventry *ve, const char *name, void **retp);
73 void (*putent) (ventry *ve);
74 int (*copyent) (ventry *ve, void **retp);
75 int (*getpath) (ventry *ve, char **retp);
76
77 int (*access) (ventry *ve, int amode);
78 int (*readlink)(ventry *ve, char **bufp);
79 int (*symlink) (const char *path, ventry *newve);
80 int (*unlink) (ventry *ve);
81 int (*rmdir) (ventry *ve);
82 int (*mknod) (ventry *ve, avmode_t mode, avdev_t dev);
83 int (*mkdir) (ventry *ve, avmode_t mode);
84 int (*rename) (ventry *ve, ventry *newve);
85 int (*link) (ventry *ve, ventry *newve);
86
87 int (*open) (ventry *ve, int flags, avmode_t mode, void **retp);
88 int (*close) (vfile *vf);
89 avssize_t (*read) (vfile *vf, char *buf, avsize_t nbyte);
90 avssize_t (*write) (vfile *vf, const char *buf, avsize_t nbyte);
91 int (*readdir) (vfile *vf, struct avdirent *buf);
92 int (*getattr) (vfile *vf, struct avstat *buf, int attrmask);
93 int (*setattr) (vfile *vf, struct avstat *buf, int attrmask);
94 int (*truncate)(vfile *vf, avoff_t length);
95 avoff_t (*lseek) (vfile *vf, avoff_t offset, int whence);
96 };
97
98 Notice that each of the three structures has a "void *data" member,
99 which is used to store filesystem-specific information. Functions in
100 'struct avfs' with a 'void **retp' parameter use 'retp' to return one
101 of these pointers - for lookup(), copyent(), and getpath(), *retp will
102 be used as the 'data' member of a new ventry; for open() it is put
103 into a vfile.data.
104
105 Most operations listed above have a corresponding wrapper function
106 prefixed with "av_" in 'oper.h' (e.g. av_open for open, av_lseek for
107 lseek); when accessing files from other modules, the module writer
108 should always use these functions rather than trying to look up the
109 method and call it directly. Exceptions are the entry management
110 operations putent, copyent, getpath which can be accessed through
111 av_copy_ventry, av_free_ventry, and av_generate_path respectively (in
112 avfs.h), and the lookup operation which is called recursively by
113 av_get_ventry (in internal.h).
114
115 The Object System
116
117 Avfs employs a simple system for garbage collection through reference
118 counting.
119
120 void *av_new_obj(avsize_t nbyte, void (*destr)(void *));
121 void av_ref_obj(void *obj);
122 void av_unref_obj(void *obj);
123 #define AV_NEW_OBJ(ptr, destr) \
124 ptr = av_new_obj(sizeof(*(ptr)), (void (*)(void *)) destr)
125
126 av_new_obj creates a new object with the specified size and
127 destructor, and with a reference count of 1. The returned pointer
128 addresses a free region at least nbytes long. The reference count and
129 destructor are stored immediately before this region in memory.
130 av_ref_obj increments the reference counter associated with the given
131 block; av_unref_obj decrements the reference counter and calls the
132 destructor function when the count reaches 0. Examples of how these
133 functions are used can be found throughout the code. The common task
134 of allocating memory for a fixed-size data structure can be
135 abbreviated by calling the AV_NEW_OBJ macro with the pointer to be
136 assigned to and a destructor function; because of this, the function
137 av_new_obj is almost never called directly.
138
139 Of the data structures which have been introduced so far, only struct
140 vfile and struct avfs are allocated as reference-counted objects;
141 ventry and avmount are allocated and freed using av_* versions of
142 malloc and free (see avfs.h).
143
144 Because the avfs core is used in long-running processes such as
145 avfscoda and avfs_server, care must be taken to avoid leaking memory
146 in a module by forgetting to decrement a reference or free a pointer.
147
148 Utility libraries
149
150 Avfs uses a few internal utility libraries throughout the code. One
151 example is a facility for organizing information into a named
152 hierarchy, called
153
154 Namespace
155
156 Namespace is pretty straightforward. I'll skip over the details since
157 they belong in a separate document (which doesn't exist as of this
158 moment, but reading the code is easy). The two important data types
159 are 'struct entry' and 'struct namespace', both defined opaquely in
160 "namespace.h". An empty namespace is created with
161
162 struct namespace *av_namespace_new();
163
164 'struct entry' has a 'void *data' member which can be accessed with
165
166 void av_namespace_set(struct entry *ent, void *data);
167 void *av_namespace_get(struct entry *ent);
168
169 To look up an entry, use
170
171 struct entry *av_namespace_resolve(struct namespace *ns, const char *path);
172
173 or
174
175 struct entry *av_namespace_lookup(struct namespace *ns, struct entry *parent,
176 const char *name);
177
178 The first function looks up an absolute path, starting at the root of
179 the namespace. The path components are separated with '/'. The second
180 function looks up an child entry in the 'parent' entry. If 'name' is
181 null, then it returns the parent of 'parent'. When a path or name
182 doesn't exist in the namespace, both functions create a new entry and
183 return that. As with most functions returning objects, these return an
184 entry with an incremented reference count - you are responsible for
185 decrementing the reference count when you are done with the returned
186 entry. Nothing else inside namespace holds references to entries
187 except an entry's own children, so when you unreference a childless
188 entry it disappears.
189
190 Cache
191
192 This is another semi-important general-use facility in avfs which
193 requires some explanation. It manages a cache of files in /tmp. Saying
194 "manages" is a bit of a stretch, though, because all the
195 responsibility for creating and deleting the files is placed on the
196 user. The only thing that 'cache' does is maintain a list of objects.
197 These are of the kind described above, typically created with
198 AV_NEW_OBJ and carrying a virtual destructor located just before the
199 object memory area. With each object is associated a disk usage
200 number. Two things must hold for objects used in 'cache': they must be
201 unreferenced with av_unref_obj after insertion into the cache (so that
202 'cache' holds the only reference to them), and the virtual destructor
203 must delete some file on the same filesystem as /tmp, which is the
204 same size as reported by the cache object. So even though the 'cache'
205 module itself never worries about creating or deleting files, each
206 object in its list represents a disk file, and the module can regulate
207 disk usage indirectly by unreferencing (and causing the destruction
208 of) these objects.
209
210 The 'cache' module keeps track of the disk usage of its own objects in
211 a global variable, and checks the amount of free space on the
212 filesystem containing /tmp whenever the size of a cache object is
213 changed or when other avfs modules call av_cache_diskfull(). Whenever
214 the amount of free space drops below a certain level, or the space
215 used by the cache objects grows above a certain level, 'cache' starts
216 deleting objects from its list in least-recently-used order. (These
217 levels are configurable through "#avfsstat/cache")
218
219 Here is the API (from cache.h):
220
221 struct cacheobj *av_cacheobj_new(void *obj, const char *name);
222 void *av_cacheobj_get(struct cacheobj *cobj);
223 void av_cacheobj_setsize(struct cacheobj *cobj, avoff_t diskusage);
224 void av_cache_checkspace();
225 void av_cache_diskfull();
226
227 Note that the 'cacheobj' object returned by av_cacheobj_new is a
228 standard avfs object; as per the convention, it is returned with a 1
229 reference count and should only be unreferenced when it is no longer
230 needed. This will cause the stored "void* obj" to be deleted.
231
232 See remote.c for an example usage - remnode holds a reference to a
233 cacheobj (remnode.file); in rem_get_file, av_cacheobj_get is called to
234 retrieve a remfile object. At this point the returned object will be
235 null if it has been purged, in which case it must be recreated and
236 reinserted into the cache.
237
238 Higher-level filesystem libraries
239
240 Here I'll touch briefly on four of the higher-level libraries, which
241 can simplify implementation of some of the more common types of
242 filesystems.
243
244 State
245
246 This is the simplest library. It is similar to procfs in linux. To use
247 it, you create a namespace with av_state_new and populate that
248 namespace with 'statefile' objects, one for each virtual file you want
249 to export. Each 'statefile' object has a pair of 'get' and 'set'
250 functions which are called when data is read and written,
251 respectively, from the corresponding virtual file.
252
253 int av_state_new(struct vmodule *module, const char *name,
254 struct namespace **resp, struct avfs **avfsp);
255
256 struct statefile {
257 void *data;
258
259 int (*get) (struct entry *ent, const char *param, char **resp);
260 int (*set) (struct entry *ent, const char *param, const char *val);
261 };
262
263 The 'ent' argument is the entry for the file in your original
264 namespace. XXX what is param? 'val' is the value to set; 'resp' holds
265 the value retrieved by get().
266
267 There is also a shortcut for creating files in /#avfsstat (from
268 internal.h):
269
270 void av_avfsstat_register(const char *path, struct statefile *func);
271
272 Filter
273
274 'filter' lets you create modules which provide access to some
275 translation of the data in a file. It is currently only used for
276 decompression and compression in the ugzip, gz, ubzip2, bz2, and uz
277 modules, but it could be used for simple decryption/encryption and
278 other things. Create a module with the following function:
279
280 int av_init_filt(struct vmodule *module, int version, const char *name,
281 const char *prog[], const char *revprog[],
282 struct ext_info *exts, struct avfs **resp);
283
284 'name' is the name of the module. The translation is carried out by an
285 external command, which is specified in the 'prog' and 'revprog'
286 arguments. 'revprog' is needed to allow writing to the virtual file,
287 and should perform the reverse operation of 'prog'. See gz.c for an
288 example invocation. (XXX: say more about filter implementation.
289 filtprog, serialfile)
290
291 Archive
292
293 Used in extfs, uar, urar, utar, and uzip. Provides access to 'archive'
294 formats; the result is a virtual directory hierarchy.
295
296 I'll just give an overview here because I don't understand much of the
297 structure of the code. For details, see e.g. uzip.c.
298
299 To instantiate an archive module, use av_archive_init. This fills in
300 most of a 'struct avfs', including all the methods. The avfs.data
301 member points to a 'struct archparams', which you are supposed to
302 initialize yourself. The methods of this structure define your module.
303 They are called by the archive library's own 'struct avfs' methods.
304
305 struct archparams {
306 void *data;
307 int flags;
308 int (*parse) (void *data, ventry *ent, struct archive *arch);
309 int (*open) (ventry *ve, struct archfile *fil);
310 int (*close) (struct archfile *fil);
311 avssize_t (*read) (vfile *vf, char *buf, avsize_t nbyte);
312 void (*release) (struct archive *arch, struct archnode *nod);
313 };
314
315 The open, close, read, and release methods are optional. Only the
316 parse method is required.
317
318 'parse' is called when an archive is first used and creates a list of
319 all of the file entries in the archive. These are stored internally in
320 a 'namespace' object in 'struct archive'. New entries are added by
321 obtaining a 'struct entry' with av_arch_create() or av_arch_resolve()
322 (archive.h, archutil.c) and then associating with it a node of type 'struct
323 archnode' with av_arch_new_node(). Among other things, the 'archnode'
324 data structure holds basic stat flags and permissions for the file,
325 the byte offset of the file data within the archive, and a 'void*
326 data' member which can be used to point to a module-specific data
327 structure. This should be filled in by 'parse'. 'void *data' is the
328 struct archparams.data member (for your own use); 'ventry *ent' is the
329 location of the archive to be parsed; and 'arch' is the (opaque) main
330 archive data structure, which is needed by av_arch_create et al.
331
332 'read' reads data from a file. The vfile.data member is an archfile:
333
334 struct archfile {
335 vfile *basefile;
336 struct archive *arch;
337 struct archnode *nod;
338 struct entry *ent; /* Only for readdir */
339 struct entry *curr;
340 int currn;
341 void *data;
342 };
343
344 If the file is stored contiguously starting at its offset in the
345 archive, then av_arch_read(), the default value for archparams.read,
346 will suffice. The archive library can figure out what to do since the
347 file offset has already been stored in the archnode by 'parse'. But if
348 the file is compressed (see uzip.c) or sparse (see utar.c), then it
349 will need special handling in your module, and you must define a
350 custom 'read' method.
351
352 'open' and 'close' can be defined if you need to allocate resources or
353 special data structures in preparation for reading a file. 'open' is
354 called with an archfile after all its fields but 'data' have been
355 filled in, at the very end of the open sequence; 'close' is called at
356 the beginning of the close sequence before anything has been
357 deallocated. Examples can be found in uzip.c.
358
359 'release' is called when an archive is no longer being used. It should
360 be defined if the 'parse' method allocated resources which need to be
361 freed.
362
363 Remote
364
365 The remote library is used by the ssh (rsh.c), rsh, ftp, dav, and
366 floppy modules. It differs from 'archive' in that the directory tree
367 is read one directory at a time (rather than all at once), and files
368 are read completely if at all (rather than allowing random access). A
369 remote module is defined with the following structure:
370
371 struct remote {
372 void *data;
373 char *name;
374 int flags;
375
376 int (*list) (struct remote *rem, struct remdirlist *dl);
377 int (*get) (struct remote *rem, struct remgetparam *gp);
378 int (*wait) (struct remote *rem, void *data, avoff_t end);
379 void (*destroy) (struct remote *rem);
380 };
381
382 This should be passed to av_remote_init, which will give you a 'struct
383 avfs*':
384
385 int av_remote_init(struct vmodule *module, struct remote *rem,
386 struct avfs **resp);
387
388 As always, remote.data is for your own use. 'name' is the name of your
389 module. XXX define 'flags'. The rest of the fields are methods:
390
391 list: obtain a directory listing. It should call av_remote_add to add
392 all the entries of dl->hostpath.path on dl->hostpath.host to the
393 remdirlist object.
394
395 get: start transferring a file. Your implementation should start
396 copying a remote file to a temporary file (possibly with
397 av_get_tmpfile) and return the name in gp->localname. You should point
398 gp->data to an object which will let you find out how much has been
399 transferred (so you can implement remote.wait). In addition, this
400 object will be unreferenced when the file is no longer needed, so its
401 destructor should delete the temporary file.
402
403 wait: block until a certain amount of data has been read. The
404 'void *data' parameter is from gp->data in remote.get. This function
405 should return 1 if the data is ready, 0 if EOF was reached, and -error
406 if there was an error.
407
408 destroy: do cleanup. Should call av_free(rem), possibly after freeing
409 module-specific data pointed to by rem.data.
410
411 THE END