| 1 |
.. _imap-developer-api-cyrusdb2: |
| 2 |
|
| 3 |
.. Note: This document was converted from the original by Nic Bernstein |
| 4 |
(Onlight). Any formatting mistakes are my fault and not the |
| 5 |
original author's. Converted via the pandoc tool from HTML. |
| 6 |
|
| 7 |
cyrusdb API |
| 8 |
=========== |
| 9 |
|
| 10 |
Intro |
| 11 |
----- |
| 12 |
|
| 13 |
The ``cyrusdb`` API is a common interface to a key-value store, used |
| 14 |
throughout the Cyrus code. It allows a choice of different backends for |
| 15 |
different access patterns, while ensuring a consistent interface. |
| 16 |
|
| 17 |
This document will describe the interface, and how to use the cyrusdb |
| 18 |
interface from within parts of Cyrus code, as well as how to implement |
| 19 |
your own backend |
| 20 |
|
| 21 |
If you pass incorrect values to these APIs, you will get an assertion |
| 22 |
failure in most cases. That's generally considered safer than silently |
| 23 |
breaking things. Exceptions are noted below. |
| 24 |
|
| 25 |
Code Layout |
| 26 |
----------- |
| 27 |
|
| 28 |
The implementation of each interface is in ``lib/cyrusdb_NAME.c``, for |
| 29 |
example lib/cyrusdb\_flat.c. General functions are in ``lib/cyrusdb.c`` |
| 30 |
and the interface in ``lib/cyrusdb.h``. |
| 31 |
|
| 32 |
Configuration |
| 33 |
------------- |
| 34 |
|
| 35 |
The name of the backend for each of the main internal databases can be |
| 36 |
configured in imapd.conf, for example: ``annotation_db: skiplist``. This |
| 37 |
is then read in imap/global.h and imap/global.c during startup, so that |
| 38 |
the global variable ``config_annotation_db`` gets set to the configured |
| 39 |
backend name: |
| 40 |
|
| 41 |
:: |
| 42 |
|
| 43 |
config_annotation_db = config_getstring(IMAPOPT_ANNOTATION_DB); |
| 44 |
|
| 45 |
(Beware the misleading naming here: ``config_annotation_db`` is a string |
| 46 |
describing the *backend* used by the database, not, say, its location on |
| 47 |
disk.) |
| 48 |
|
| 49 |
Internally, the main module for each database sets up struct of pointers |
| 50 |
to the cyrusdb functions it implements, which is registered in |
| 51 |
``lib/cyrusdb.c`` |
| 52 |
|
| 53 |
``lib/cyrusdb.c`` provides backend-agnostic wrapper functions for |
| 54 |
interacting with cyrusdb databases. |
| 55 |
|
| 56 |
A full example |
| 57 |
-------------- |
| 58 |
|
| 59 |
:: |
| 60 |
|
| 61 |
struct db *db = NULL; |
| 62 |
struct txn *tid = NULL; |
| 63 |
const char *filename = NULL; |
| 64 |
int flags = CYRUSDB_CREATE; |
| 65 |
|
| 66 |
cyrus_init(alt_config, "toolname", 0); |
| 67 |
|
| 68 |
filename = config_getstring(IMAPOPT_ANNOTATION_DB_PATH); |
| 69 |
|
| 70 |
r = cyrusdb_open(config_annotation_db, filename, flags, &db); |
| 71 |
|
| 72 |
r = cyrusdb_fetch(db, key, keylen, &data, &datalen, &tid); |
| 73 |
|
| 74 |
r = cyrusdb_commit(db, tid); |
| 75 |
|
| 76 |
r = cyrusdb_close(db); |
| 77 |
|
| 78 |
cyrus_done(); |
| 79 |
|
| 80 |
Note that you always open a database first, and close it at the end. You |
| 81 |
must always call cyrus\_init() and cyrus\_done() to properly initialize |
| 82 |
and clean up the ``cyrusdb`` environments. |
| 83 |
|
| 84 |
This example also uses a transaction, meaning that the database is |
| 85 |
locked in exclusive mode between the 'fetch' (the first use of the |
| 86 |
transaction) and the commit. |
| 87 |
|
| 88 |
About Transactions |
| 89 |
------------------ |
| 90 |
|
| 91 |
The cyrusdb interface works in two modes - transactional and |
| 92 |
non-transactional. The value of the 'tid' parameter decides which mode |
| 93 |
is used. There are three possible values: |
| 94 |
|
| 95 |
- NULL - non-transactional. Will create a temporary lock for the |
| 96 |
duration of the current action - either a write lock for "store" or a |
| 97 |
read lock for "fetch". If you call "foreach", the lock will be |
| 98 |
dropped between each record fetched |
| 99 |
- Pointer to NULL - transactional, transaction not yet started. Will |
| 100 |
always take a write lock on the database, and update the pointer to |
| 101 |
point to the new transaction. |
| 102 |
- Pointer to a valid transaction. Will keep using this transaction |
| 103 |
|
| 104 |
If you are currently in a transaction, you MUST pass the same |
| 105 |
transaction to every database call. It is not possible to mix or nest |
| 106 |
transactions. There is one exception in the skiplist backend: *If you |
| 107 |
pass NULL to a fetch or foreach while the database is in a transaction, |
| 108 |
it will silently do the read in the current transaction rather than |
| 109 |
returning an error* |
| 110 |
|
| 111 |
API Reference |
| 112 |
------------- |
| 113 |
|
| 114 |
All functions follow the normal C API of returning '0' on success, and |
| 115 |
an error code on failure |
| 116 |
|
| 117 |
cyrusdb\_init(void) |
| 118 |
~~~~~~~~~~~~~~~~~~~ |
| 119 |
|
| 120 |
Is called once per process. Don't call this yourself, use |
| 121 |
``cyrus_init()``. No other calls will be made until this is called. |
| 122 |
|
| 123 |
cyrusdb\_done(void) |
| 124 |
~~~~~~~~~~~~~~~~~~~ |
| 125 |
|
| 126 |
The opposite of ``cyrusdb_init()`` - called once per process to do any |
| 127 |
cleaning up after all database usage is finished. Don't call this |
| 128 |
yourself, use ``cyrus_done()``. |
| 129 |
|
| 130 |
cyrusdb\_sync(const char \*backend) |
| 131 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 132 |
|
| 133 |
Perform a checkpoint of the database environment. Used by berkeley |
| 134 |
backend. Is called by ``ctl_cyrusdb -c`` on a regular basis |
| 135 |
|
| 136 |
cyrusdb\_open(const char \*backend, const char \*fname, int flags, struct db \*\*retdb) |
| 137 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 138 |
|
| 139 |
Opens the database with the specified 'file name' (or other descriptor, |
| 140 |
for example the sql backend is not a filename), and if successful |
| 141 |
returns an opaque database structure |
| 142 |
|
| 143 |
Flags: |
| 144 |
|
| 145 |
- CYRUSDB\_CREATE - create the database if it doesn't exist |
| 146 |
|
| 147 |
Errors: |
| 148 |
|
| 149 |
- CYRUSDB\_IOERROR - if there is any error reading the file, or any |
| 150 |
corruption detected while loading the file |
| 151 |
|
| 152 |
cyrusdb\_close(struct db \*db) |
| 153 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 154 |
|
| 155 |
Close the named database. Will release any locks if they are still held, |
| 156 |
but it's bad practice to close without committing or aborting, so the |
| 157 |
backend should log an error |
| 158 |
|
| 159 |
Errors: |
| 160 |
|
| 161 |
- CYRUSDB\_IOERROR - if there are any errors during close |
| 162 |
|
| 163 |
cyrusdb\_fetch(struct db \*db, const char \*key, size\_t keylen, const char \*\*data, size\_t \*datalen, struct txn \*\*tidptr) |
| 164 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 165 |
|
| 166 |
cyrusdb\_fetchlock(struct db \*db, const char \*key, size\_t keylen, const char \*\*data, size\_t \*datalen, struct txn \*\*tidptr) |
| 167 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 168 |
|
| 169 |
Fetch the value for the exact key given by key and keylen. If data is |
| 170 |
not NULL, set datalen and return a valid pointer to the start of the |
| 171 |
value. |
| 172 |
|
| 173 |
Fetchlock is identical to fetch, but gives a hint to the database that |
| 174 |
the record is likely to be modified soon. |
| 175 |
|
| 176 |
NOTE: it is possible to store a key with a zero length data record, in |
| 177 |
which case \*datalen will be set to zero, and \*data will be set to a |
| 178 |
non-NULL value |
| 179 |
|
| 180 |
It is an error to call fetch with a NULL key or a zero keylen |
| 181 |
|
| 182 |
It is an error to call fetch with a NULL datalen and a non-NULL data, |
| 183 |
however it is acceptable to call with a NULL data and a non-NULL datalen |
| 184 |
if you are only interested in the length |
| 185 |
|
| 186 |
Errors: |
| 187 |
|
| 188 |
- CYRUSDB\_IOERROR - if any error occurs reading from the database |
| 189 |
- CYRUSDB\_LOCKED - if tidptr is incorrect |
| 190 |
- CYRUSDB\_NOTFOUND - if there is no record that matches the key |
| 191 |
|
| 192 |
cyrusdb\_foreach(struct db \*db, const char \*prefix, size\_t prefixlen, foreach\_p \*goodp, foreach\_p \*procp, void \*rock, struct txn \*\*tidptr) |
| 193 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 194 |
|
| 195 |
cyrusdb\_forone(struct db \*db, const char \*key, size\_t keylen, foreach\_p \*goodp, foreach\_p \*procp, void \*rock, struct txn \*\*tidptr) |
| 196 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 197 |
|
| 198 |
``cyrusdb_foreach()`` iterates over all records matching the given |
| 199 |
prefix, in database order (which may be MBOXLIST sort, depending on the |
| 200 |
parameters given to open |
| 201 |
|
| 202 |
It is legal to give a NULL pointer as prefix if prefixlen is zero, in |
| 203 |
which case it will return all records in the database. It is an error to |
| 204 |
give a non-zero prefixlen with a NULL prefix. |
| 205 |
|
| 206 |
``cyrusdb_forone()`` "iterates" over the single record matched by the |
| 207 |
given key. If you've already built callbacks for processing each record |
| 208 |
from a foreach, this lets you use the same interface to process a single |
| 209 |
record. |
| 210 |
|
| 211 |
``goodp`` - this function is only used for deciding if the record needs |
| 212 |
to be further processed. It can be used for basic filtering, and returns |
| 213 |
true (non-zero) to process, or zero to skip and move straight to the |
| 214 |
next record. Because goodp can't make any database changes, it doesn't |
| 215 |
break the lock, so it's faster to use goodp to filter records if you |
| 216 |
don't need to process all of them. NULL is a legal value for goodp, and |
| 217 |
means that all records will be processed. |
| 218 |
|
| 219 |
``procp`` - procp is the main callback function. If you use foreach in |
| 220 |
non-transactional mode, the database is unlocked before calling procp, |
| 221 |
and locked again afterwards. You are allowed to add, delete or modify |
| 222 |
values in the same database from within procp. If procp returns |
| 223 |
non-zero, the foreach loop breaks at this point, and the return value of |
| 224 |
the foreach becomes the return value of procp. If procp returns zero, |
| 225 |
the foreach loop will continue at the NEXT record by sort order, |
| 226 |
regardless of whether the current record has changed or been removed. |
| 227 |
procp MUST NOT be NULL. |
| 228 |
|
| 229 |
Errors: |
| 230 |
|
| 231 |
- procp\_result - whatever your callback returns |
| 232 |
- CYRUSDB\_IOERROR - if any error occurs while reading |
| 233 |
- CYRUSDB\_LOCKED - if tidptr is incorrect |
| 234 |
|
| 235 |
cyrusdb\_create(struct db \*db, const char \*key, size\_t keylen, const char \*data, size\_t datalen, struct txn \*\*tidptr) |
| 236 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 237 |
|
| 238 |
cyrusdb\_store(struct db \*db, const char \*key, size\_t keylen, const char \*data, size\_t datalen, struct txn \*\*tidptr) |
| 239 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 240 |
|
| 241 |
Create a new record or replace an existing one. The only difference |
| 242 |
between these two is that ``cyrusdb_create`` will return an error if the |
| 243 |
record already exists, while ``cyrusdb_store`` will replace it |
| 244 |
|
| 245 |
If tidptr is NULL, create/store will take a write lock for the duration |
| 246 |
of the action. |
| 247 |
|
| 248 |
Any failure during create/store will abort the current transaction as |
| 249 |
well as returning an error |
| 250 |
|
| 251 |
It is legal to pass NULL for the data field ONLY if datalen is zero. It |
| 252 |
is not legal to pass NULL for key or zero for keylen |
| 253 |
|
| 254 |
Errors: |
| 255 |
|
| 256 |
- CYRUSDB\_IOERROR - any error to write to the database |
| 257 |
- CYRUSDB\_LOCKED - if tidptr is incorrect |
| 258 |
- CYRUSDB\_EXISTS - if ``cyrusdb_create`` is called on an existing key |
| 259 |
- CYRUSDB\_AGAIN - if a deadlock is created. The current transaction |
| 260 |
has been aborted, but a retry may succeed |
| 261 |
|
| 262 |
cyrusdb\_delete(struct db \*db, const char \*key, size\_t keylen, struct txn \*\*tidptr, int force) |
| 263 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 264 |
|
| 265 |
Delete the given record from the database. If force is true, then |
| 266 |
succeed even if the record doesn't currently exist. |
| 267 |
|
| 268 |
It is not legal to pass NULL for key or zero for keylen |
| 269 |
|
| 270 |
Errors: |
| 271 |
|
| 272 |
- CYRUSDB\_IOERROR - any error to write to the database |
| 273 |
- CYRUSDB\_LOCKED - if tidptr is incorrect |
| 274 |
- CYRUSDB\_NOTFOUND - if force is not set and the key doesn't exist |
| 275 |
- CYRUSDB\_AGAIN - if a deadlock is created. The current transaction |
| 276 |
has been aborted, but a retry may succeed |
| 277 |
|
| 278 |
cyrusdb\_commit(struct db \*db, struct txn \*tid) |
| 279 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 280 |
|
| 281 |
Commit the current transaction. tid will not be valid after this call, |
| 282 |
regardless of success |
| 283 |
|
| 284 |
If the commit fails, it will attempt to abort the transaction |
| 285 |
|
| 286 |
Errors: |
| 287 |
|
| 288 |
- CYRUSDB\_IOERROR - any error to write to the database |
| 289 |
- CYRUSDB\_LOCKED - if tidptr is incorrect |
| 290 |
- CYRUSDB\_AGAIN - if a deadlock is created. The current transaction |
| 291 |
has been aborted, but a retry may succeed |
| 292 |
|
| 293 |
cyrusdb\_abort(struct db \*db, struct txn \*tid) |
| 294 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 295 |
|
| 296 |
Abort the current transaction. tid will not be valid after this call, |
| 297 |
regardless of success |
| 298 |
|
| 299 |
Attempt to roll back all changes made in the current transaction. |
| 300 |
|
| 301 |
Errors: |
| 302 |
|
| 303 |
- CYRUSDB\_IOERROR - any error to write to the database |
| 304 |
- CYRUSDB\_LOCKED - if tidptr is incorrect |
| 305 |
|
| 306 |
cyrusdb\_dump(struct db \*db, int detail) |
| 307 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 308 |
|
| 309 |
Optional function to dump the internal structure of the database to |
| 310 |
stdout for debugging purposes. Don't use. |
| 311 |
|
| 312 |
cyrusdb\_consistent(struct db \*db) |
| 313 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 314 |
|
| 315 |
Check if the DB is internally consistent. Looks pretty bogus, and isn't |
| 316 |
used anywhere. Don't use. |