| 1 |
<!DOCTYPE html> |
| 2 |
<html> |
| 3 |
<head> |
| 4 |
<title>XMLS: Common Lisp XML Parser</title> |
| 5 |
</head> |
| 6 |
<body> |
| 7 |
<h1>XMLS</h1> |
| 8 |
<center>Manual For Version 3</center> |
| 9 |
|
| 10 |
<h2>Summary</h2> |
| 11 |
<p> |
| 12 |
Xmls is a small, simple, non-validating xml parser for Common Lisp. It's |
| 13 |
designed to be a self-contained, easily embedded parser that recognizes a useful |
| 14 |
subset of the XML spec. It provides a simple mapping from xml to lisp |
| 15 |
structures or s-expressions and back. |
| 16 |
</p> |
| 17 |
<p>Since XMLS was first released it has gained some additional |
| 18 |
complications/features. In particular:</p> |
| 19 |
<ul> |
| 20 |
<li><strong>Now XMLS by default parses XML documents into lisp structures, |
| 21 |
rather than s-expressions.</strong> This makes accessing the structures |
| 22 |
simpler and more reliable. See <a href="#Compatibility" >section on backward compatibility</a>.</li> |
| 23 |
<li>We have added clearly named accessors to further improve |
| 24 |
extraction of information from parsed XML.</li> |
| 25 |
<li>Thanks to Max Rottenkolber, we now have the affiliated library, |
| 26 |
<a href="#octets"><code>xmls/octets</code></a> that will open streams for the XMLS parser, |
| 27 |
processing any content-type declarations in the process.</li> |
| 28 |
</ul> |
| 29 |
|
| 30 |
|
| 31 |
<h2>Features</h2> |
| 32 |
<ul> |
| 33 |
<li> |
| 34 |
Free (BSD license). |
| 35 |
</li> |
| 36 |
<li> |
| 37 |
Understands enough of the xml spec to parse many common documents, including |
| 38 |
those occurring in common internet protocols like xml-rpc, webdav, and BEEP. |
| 39 |
Parses 85 out of the 98 valid documents in the oasis parser compliance suite. |
| 40 |
</li> |
| 41 |
<li> |
| 42 |
Small and easily embedded. The entire parser is contained in one |
| 43 |
file and it's currently less than 600 lines of code. Xmls is written in |
| 44 |
pure lisp and requires no external parsing tools or foreign libraries. |
| 45 |
</li> |
| 46 |
<li> |
| 47 |
Supports xml namespaces. |
| 48 |
</li> |
| 49 |
<li> |
| 50 |
Threadsafe. |
| 51 |
</li> |
| 52 |
<li> |
| 53 |
Serializes s-expr list structures back to xml as well as parsing xml. |
| 54 |
</li> |
| 55 |
</ul> |
| 56 |
|
| 57 |
<h2>Limitations</h2> |
| 58 |
<ul> |
| 59 |
<li> |
| 60 |
Parses entire document into memory and consequently can't handle large |
| 61 |
documents. |
| 62 |
</li> |
| 63 |
<li> |
| 64 |
No detailed error reporting. |
| 65 |
</li> |
| 66 |
<li> |
| 67 |
Hand-built LR parser, meaning the parser structure is a little hard |
| 68 |
to understand, and can be hard to modify. Use of CL-YACC or similar |
| 69 |
might be a preferable route for a rewrite. |
| 70 |
</li> |
| 71 |
</ul> |
| 72 |
|
| 73 |
<h2>XML Representation</h2> |
| 74 |
|
| 75 |
<p> |
| 76 |
Parsed xml is represented as a nested lisp structure, unlike in the |
| 77 |
original version, where it was a lisp list. The s-expression |
| 78 |
representation is still maintained, and there are <a href="#translators">functions to |
| 79 |
translate to and from this notation</a>. |
| 80 |
</p> |
| 81 |
|
| 82 |
<h3>XML representation as lisp structures</h3> |
| 83 |
|
| 84 |
<p>In the structure representation, a node, corresponding to an XML |
| 85 |
element, is defined as follows: |
| 86 |
</p> |
| 87 |
|
| 88 |
<pre>(defstruct (node (:constructor %make-node)) |
| 89 |
name |
| 90 |
ns |
| 91 |
attrs |
| 92 |
children)</pre> |
| 93 |
|
| 94 |
<p> |
| 95 |
Xmls also includes a helper function, make-node for creating xml nodes |
| 96 |
of this form: |
| 97 |
</p> |
| 98 |
|
| 99 |
<pre> |
| 100 |
(make-node &key name ns attrs children) |
| 101 |
</pre> |
| 102 |
|
| 103 |
<p> |
| 104 |
Xmls provides the corresponding accessor functions node-name, node-ns |
| 105 |
node-attrs, and node-children. |
| 106 |
</p> |
| 107 |
|
| 108 |
<h3>XML representation as s-expressions</h3> |
| 109 |
|
| 110 |
<p>In the s-expression representation, a node is represented as follows:</p> |
| 111 |
|
| 112 |
<pre> |
| 113 |
(name (attributes) children*) |
| 114 |
</pre> |
| 115 |
|
| 116 |
<p> |
| 117 |
A name is either a simple string, if the element does not belong to a namespace, |
| 118 |
or a list of (name namespace-url) if the element does belong to a namespace. |
| 119 |
</p> |
| 120 |
|
| 121 |
<p> |
| 122 |
Attributes are stored as (name value) lists. |
| 123 |
</p> |
| 124 |
|
| 125 |
<p> |
| 126 |
Children are stored as a list of either element nodes or text nodes. |
| 127 |
</p> |
| 128 |
|
| 129 |
<p> |
| 130 |
For example, the following xml document: |
| 131 |
</p> |
| 132 |
|
| 133 |
<pre> |
| 134 |
<?xml version="1.0"?> |
| 135 |
<!-- test document --> |
| 136 |
<book title='The Cyberiad'> |
| 137 |
<!-- comment in here --> |
| 138 |
<author xmlns='http://authors'>Stanislaw Lem</author> |
| 139 |
<info:subject xmlns:info='http://bookinfo' rank='1'>&quot;Cybernetic Fables&quot;</info:subject> |
| 140 |
</book> |
| 141 |
</pre> |
| 142 |
|
| 143 |
Would parse as: |
| 144 |
|
| 145 |
<pre> |
| 146 |
("book" (("title" "The Cyberiad")) |
| 147 |
(("author" . "http://authors") NIL "Stanislaw Lem") |
| 148 |
(("subject" . "http://bookinfo") (("rank" "1")) "\"Cybernetic Fables\"")) |
| 149 |
</pre> |
| 150 |
|
| 151 |
<a name="Compatibility"><h3>Backward Compatibility</h3></a> |
| 152 |
|
| 153 |
<p>To detect whether in this version of XMLS the return value of <code>PARSE</code> will be a |
| 154 |
list or a structure, check for the feature <code>:XMLS-NODES-ARE-STRUCTS</code>. |
| 155 |
|
| 156 |
<p>For old code that wants XML parsed into lists, instead of |
| 157 |
structures, you may replace calls to <code>(parse str)</code> with |
| 158 |
<code>(node->nodelist (parse str))</code>.</p> |
| 159 |
|
| 160 |
<p>For greater convenience, we offer <code>PARSE-TO-LIST</code>, which |
| 161 |
performs the same function.</p> |
| 162 |
|
| 163 |
<h2>Usage</h2> |
| 164 |
|
| 165 |
<p> |
| 166 |
The interface is straightforward. The two main functions are |
| 167 |
<code>PARSE</code> and <code>TOXML</code>. |
| 168 |
</p> |
| 169 |
|
| 170 |
<pre> |
| 171 |
(parse source &key (compress-whitespace t) (quash-errors t) |
| 172 |
</pre> |
| 173 |
|
| 174 |
<p> |
| 175 |
Parse accepts either a string or an input stream and attempts to parse the xml |
| 176 |
document contained therein. It will return the s-expr parse tree if it's |
| 177 |
successful or nil if parsing fails.</p> |
| 178 |
|
| 179 |
<p> |
| 180 |
If <code>COMPRESS-WHITESPACE</code> is non-<code>NIL</code>, content nodes will be trimmed of whitespace and |
| 181 |
empty whitespace strings between nodes will be discarded. |
| 182 |
</p> |
| 183 |
|
| 184 |
|
| 185 |
<pre> |
| 186 |
(parse-to-list source (&rest args)) |
| 187 |
</pre> |
| 188 |
|
| 189 |
<p>Functions as <code>PARSE</code>, but returns a list representation |
| 190 |
of the XML document, instead of a structure.</p> |
| 191 |
|
| 192 |
<pre> |
| 193 |
(write-prologue xml-decl doctype stream) |
| 194 |
</pre> |
| 195 |
|
| 196 |
<p> |
| 197 |
write-prologue writes the leading |
| 198 |
<code><?xml ... ?></code> and <code><!DOCTYPE ... ></code> |
| 199 |
elements to <code>stream</code>. |
| 200 |
<code>xml-decl</code> is an alist of attribute name value pairs. |
| 201 |
Valid xml-decl attributes per the xml spec are "version", "encoding", |
| 202 |
and "standalone", though write-prologue does not verify this. |
| 203 |
<code>doctype</code> is a string containing the document type definition. |
| 204 |
</p> |
| 205 |
|
| 206 |
<pre> |
| 207 |
(write-prolog xml-decl doctype stream) |
| 208 |
</pre> |
| 209 |
<p> |
| 210 |
U.S. spelling alternative to <code>write-prologue</code>. |
| 211 |
</p> |
| 212 |
|
| 213 |
<pre> |
| 214 |
(write-xml xml stream &key (indent nil)) |
| 215 |
</pre> |
| 216 |
|
| 217 |
<p> |
| 218 |
write-xml accepts a lisp list in the format described above and writes the |
| 219 |
equivalent xml string to stream. Currently, if nodes use namespaces xmls will not |
| 220 |
assign namespaces prefixes but will explicitly assign the namespace to each node. This |
| 221 |
will be changed in a later release. |
| 222 |
|
| 223 |
Xmls will indent the generated xml output if indent is non-nil. |
| 224 |
</p> |
| 225 |
|
| 226 |
<pre> |
| 227 |
(toxml node &key (indent nil)) |
| 228 |
</pre> |
| 229 |
|
| 230 |
<p> |
| 231 |
<code>TOXML</code> is a convenience wrapper around write-xml that returns the in a newly |
| 232 |
allocated string. |
| 233 |
</p> |
| 234 |
|
| 235 |
<a name="translators"><h3>Translating to and from s-expressions</h3></a> |
| 236 |
|
| 237 |
<p>XMLS provides two exported functions to translate between the CL |
| 238 |
structure representation of the XML tree and the s-expression |
| 239 |
representation:</p> |
| 240 |
<dl> |
| 241 |
<dt><code>node->nodelist</code></dt> |
| 242 |
<dd>Translate the structure representation into s-expressions.</dd> |
| 243 |
|
| 244 |
<dt><code>nodelist->nodes</code></dt> |
| 245 |
<dd>Translate the s-expression representation of an XMLS parse tree |
| 246 |
into lisp structures.</dd> |
| 247 |
</dl> |
| 248 |
|
| 249 |
|
| 250 |
<h3>Helper functions</h3> |
| 251 |
|
| 252 |
<p>These are intended to allow programmers to avoid direct manipulation of the |
| 253 |
XMLS element representation. If you use these, your code should be easier to |
| 254 |
read and you will avoid problems if there is a change in internal |
| 255 |
representation (such changes would be hard to even find, much less correct, if |
| 256 |
using the lists directly).</p> |
| 257 |
|
| 258 |
<dl> |
| 259 |
<dt><code> |
| 260 |
make-xmlrep (tag &key attribs children) |
| 261 |
</code> |
| 262 |
</dt> |
| 263 |
<dd>Constructor function.</dd> |
| 264 |
|
| 265 |
<dt> |
| 266 |
<code>xmlrep-add-child! (xmlrep child)</code> |
| 267 |
</dt> |
| 268 |
<dd>Add a new child node to the XMLREP node.</dd> |
| 269 |
|
| 270 |
<dt> |
| 271 |
<code>xmlrep-tag (xmlrep)</code> |
| 272 |
</dt> |
| 273 |
<dd>Extract the tag from XMLREP.</dd> |
| 274 |
|
| 275 |
<dt> |
| 276 |
<code>xmlrep-tagmatch (tag treenode)</code> |
| 277 |
</dt> |
| 278 |
<dd>Returns true if TAG is the tag of TREENODE. Match is |
| 279 |
case <em>insensitive</em> (quite possibly this is the Wrong Thing). </dd> |
| 280 |
|
| 281 |
<dt> |
| 282 |
<code>xmlrep-attribs (xmlrep)</code> |
| 283 |
</dt> |
| 284 |
<dd>Extract the attributes from an XMLREP node.</dd> |
| 285 |
|
| 286 |
<dt> |
| 287 |
<code>xmlrep-children (xmlrep)</code> |
| 288 |
</dt> |
| 289 |
<dd>Extract the children from an XMLREP node.</dd> |
| 290 |
|
| 291 |
<dt> |
| 292 |
<code>xmlrep-find-child-tags (tag treenode)</code> |
| 293 |
</dt> |
| 294 |
<dd> |
| 295 |
Return all of the (direct) children of TREENODE whose tags are TAG. |
| 296 |
Matching done by <a href="#xmlrep-tagmatch"><code>xmlrep-tagmatch</code></a>. |
| 297 |
</dd> |
| 298 |
|
| 299 |
<dt> |
| 300 |
<code>xmlrep-find-child-tag (tag treenode &optional (if-unfound :error))</code> |
| 301 |
</dt> |
| 302 |
<dd> |
| 303 |
Find a <em>single</em> child of TREENODE with TAG. Returns an error |
| 304 |
if there is more or less than one such child. |
| 305 |
</dd> |
| 306 |
|
| 307 |
<dt> |
| 308 |
<code>xmlrep-string-child (treenode &optional (if-unfound :error))</code> |
| 309 |
</dt> |
| 310 |
<dd> |
| 311 |
Returns the <em>single</em> string-valued child of TREENODE. |
| 312 |
If there is more than one child, or if a single child is not |
| 313 |
a simple value, returns IF-UNFOUND, which defaults to :ERROR. |
| 314 |
</dd> |
| 315 |
|
| 316 |
<dt> |
| 317 |
<code>xmlrep-integer-child (treenode)</code> |
| 318 |
</dt> |
| 319 |
<dd> |
| 320 |
Find the <em>single</em> child of TREENODE whose value is a string that |
| 321 |
can be parsed into an integer. Returns an |
| 322 |
error if there is more than one child, or if a single child is not |
| 323 |
appropriately valued. |
| 324 |
</dd> |
| 325 |
|
| 326 |
<dt> |
| 327 |
<code>xmlrep-attrib-value (attrib treenode &optional (if-undefined :error))</code> |
| 328 |
</dt> |
| 329 |
<dd> |
| 330 |
Find the value of ATTRIB, a string, in TREENODE. |
| 331 |
if there is no ATTRIB, will return the value of IF-UNDEFINED, |
| 332 |
which defaults to :ERROR. |
| 333 |
</dd> |
| 334 |
|
| 335 |
<dt> |
| 336 |
<code>xmlrep-boolean-attrib-value (attrib treenode &optional (if-undefined :error))</code> |
| 337 |
</dt> |
| 338 |
<dd> |
| 339 |
Find the value of ATTRIB, a string, in TREENODE. |
| 340 |
The value should be either "true" or "false". The |
| 341 |
function will return T or NIL, accordingly. If there is no ATTRIB, |
| 342 |
will return the value of IF-UNDEFINED, which defaults to :ERROR. |
| 343 |
</dd> |
| 344 |
</dl> |
| 345 |
|
| 346 |
<a name="octets"><h2>XMLS/Octets</h2></a> |
| 347 |
<p> |
| 348 |
XMLS itself simply processes strings or streams. This means that it |
| 349 |
does not provide native support for handling character encodings, as |
| 350 |
declared in the XML headers. The system <code>xmls/octets</code>, |
| 351 |
which depends on <code>xmls</code> provides that support with the |
| 352 |
exported function <code>make-xml-stream</code>, which takes an |
| 353 |
octet-stream as argument, processes its header, choosing the |
| 354 |
appropriate character encoding, and then returns a stream suitable for |
| 355 |
passing to <code>xmls:parse</code>.</p> |
| 356 |
|
| 357 |
<p>Probably <code>make-xml-stream</code> should be made generic, and support |
| 358 |
arguments of other types (e.g., strings interpreted as filenames, |
| 359 |
pathnames, etc.).</p> |
| 360 |
|
| 361 |
<h2>Installation</h2> |
| 362 |
|
| 363 |
<p> |
| 364 |
xmls can be installed as an asdf system. An asdf |
| 365 |
system definition is provided with the distribution. |
| 366 |
</p> |
| 367 |
<p>Previous versions of XMLS were single files, and could be installed simply by |
| 368 |
loading the file xmls.lisp. This option is no longer supported.</p> |
| 369 |
|
| 370 |
<h2>Contact Information</h2> |
| 371 |
|
| 372 |
<p> |
| 373 |
Please contact Robert Goldman, rpgoldman AT sift.net with any |
| 374 |
questions or bug reports. |
| 375 |
</p> |
| 376 |
</body> |
| 377 |
</html> |