Debian Codegraph @

LspCpp/third_party/rapidjson/doc/sax.md

raw
1 # SAX
2
3 The term "SAX" originated from [Simple API for XML](http://en.wikipedia.org/wiki/Simple_API_for_XML). We borrowed this term for JSON parsing and generation.
4
5 In RapidJSON, `Reader` (typedef of `GenericReader<...>`) is the SAX-style parser for JSON, and `Writer` (typedef of `GenericWriter<...>`) is the SAX-style generator for JSON.
6
7 [TOC]
8
9 # Reader {#Reader}
10
11 `Reader` parses a JSON from a stream. While it reads characters from the stream, it analyzes the characters according to the syntax of JSON, and publishes events to a handler.
12
13 For example, here is a JSON.
14
15 ~~~~~~~~~~js
16 {
17 "hello": "world",
18 "t": true ,
19 "f": false,
20 "n": null,
21 "i": 123,
22 "pi": 3.1416,
23 "a": [1, 2, 3, 4]
24 }
25 ~~~~~~~~~~
26
27 When a `Reader` parses this JSON, it publishes the following events to the handler sequentially:
28
29 ~~~~~~~~~~
30 StartObject()
31 Key("hello", 5, true)
32 String("world", 5, true)
33 Key("t", 1, true)
34 Bool(true)
35 Key("f", 1, true)
36 Bool(false)
37 Key("n", 1, true)
38 Null()
39 Key("i")
40 Uint(123)
41 Key("pi")
42 Double(3.1416)
43 Key("a")
44 StartArray()
45 Uint(1)
46 Uint(2)
47 Uint(3)
48 Uint(4)
49 EndArray(4)
50 EndObject(7)
51 ~~~~~~~~~~
52
53 These events can be easily matched with the JSON, but some event parameters need further explanation. Let's see the `simplereader` example which produces exactly the same output as above:
54
55 ~~~~~~~~~~cpp
56 #include "rapidjson/reader.h"
57 #include <iostream>
58
59 using namespace rapidjson;
60 using namespace std;
61
62 struct MyHandler : public BaseReaderHandler<UTF8<>, MyHandler> {
63 bool Null() { cout << "Null()" << endl; return true; }
64 bool Bool(bool b) { cout << "Bool(" << boolalpha << b << ")" << endl; return true; }
65 bool Int(int i) { cout << "Int(" << i << ")" << endl; return true; }
66 bool Uint(unsigned u) { cout << "Uint(" << u << ")" << endl; return true; }
67 bool Int64(int64_t i) { cout << "Int64(" << i << ")" << endl; return true; }
68 bool Uint64(uint64_t u) { cout << "Uint64(" << u << ")" << endl; return true; }
69 bool Double(double d) { cout << "Double(" << d << ")" << endl; return true; }
70 bool String(const char* str, SizeType length, bool copy) {
71 cout << "String(" << str << ", " << length << ", " << boolalpha << copy << ")" << endl;
72 return true;
73 }
74 bool StartObject() { cout << "StartObject()" << endl; return true; }
75 bool Key(const char* str, SizeType length, bool copy) {
76 cout << "Key(" << str << ", " << length << ", " << boolalpha << copy << ")" << endl;
77 return true;
78 }
79 bool EndObject(SizeType memberCount) { cout << "EndObject(" << memberCount << ")" << endl; return true; }
80 bool StartArray() { cout << "StartArray()" << endl; return true; }
81 bool EndArray(SizeType elementCount) { cout << "EndArray(" << elementCount << ")" << endl; return true; }
82 };
83
84 void main() {
85 const char json[] = " { \"hello\" : \"world\", \"t\" : true , \"f\" : false, \"n\": null, \"i\":123, \"pi\": 3.1416, \"a\":[1, 2, 3, 4] } ";
86
87 MyHandler handler;
88 Reader reader;
89 StringStream ss(json);
90 reader.Parse(ss, handler);
91 }
92 ~~~~~~~~~~
93
94 Note that RapidJSON uses templates to statically bind the `Reader` type and the handler type, instead of using classes with virtual functions. This paradigm can improve performance by inlining functions.
95
96 ## Handler {#Handler}
97
98 As shown in the previous example, the user needs to implement a handler which consumes the events (via function calls) from the `Reader`. The handler must contain the following member functions.
99
100 ~~~~~~~~~~cpp
101 class Handler {
102 bool Null();
103 bool Bool(bool b);
104 bool Int(int i);
105 bool Uint(unsigned i);
106 bool Int64(int64_t i);
107 bool Uint64(uint64_t i);
108 bool Double(double d);
109 bool RawNumber(const Ch* str, SizeType length, bool copy);
110 bool String(const Ch* str, SizeType length, bool copy);
111 bool StartObject();
112 bool Key(const Ch* str, SizeType length, bool copy);
113 bool EndObject(SizeType memberCount);
114 bool StartArray();
115 bool EndArray(SizeType elementCount);
116 };
117 ~~~~~~~~~~
118
119 `Null()` is called when the `Reader` encounters a JSON null value.
120
121 `Bool(bool)` is called when the `Reader` encounters a JSON true or false value.
122
123 When the `Reader` encounters a JSON number, it chooses a suitable C++ type mapping. And then it calls *one* function out of `Int(int)`, `Uint(unsigned)`, `Int64(int64_t)`, `Uint64(uint64_t)` and `Double(double)`. If `kParseNumbersAsStrings` is enabled, `Reader` will always calls `RawNumber()` instead.
124
125 `String(const char* str, SizeType length, bool copy)` is called when the `Reader` encounters a string. The first parameter is pointer to the string. The second parameter is the length of the string (excluding the null terminator). Note that RapidJSON supports null character `\0` inside a string. If such situation happens, `strlen(str) < length`. The last `copy` indicates whether the handler needs to make a copy of the string. For normal parsing, `copy = true`. Only when *insitu* parsing is used, `copy = false`. And be aware that the character type depends on the target encoding, which will be explained later.
126
127 When the `Reader` encounters the beginning of an object, it calls `StartObject()`. An object in JSON is a set of name-value pairs. If the object contains members it first calls `Key()` for the name of member, and then calls functions depending on the type of the value. These calls of name-value pairs repeat until calling `EndObject(SizeType memberCount)`. Note that the `memberCount` parameter is just an aid for the handler; users who do not need this parameter may ignore it.
128
129 Arrays are similar to objects, but simpler. At the beginning of an array, the `Reader` calls `BeginArray()`. If there is elements, it calls functions according to the types of element. Similarly, in the last call `EndArray(SizeType elementCount)`, the parameter `elementCount` is just an aid for the handler.
130
131 Every handler function returns a `bool`. Normally it should return `true`. If the handler encounters an error, it can return `false` to notify the event publisher to stop further processing.
132
133 For example, when we parse a JSON with `Reader` and the handler detects that the JSON does not conform to the required schema, the handler can return `false` and let the `Reader` stop further parsing. This will place the `Reader` in an error state, with error code `kParseErrorTermination`.
134
135 ## GenericReader {#GenericReader}
136
137 As mentioned before, `Reader` is a typedef of a template class `GenericReader`:
138
139 ~~~~~~~~~~cpp
140 namespace rapidjson {
141
142 template <typename SourceEncoding, typename TargetEncoding, typename Allocator = MemoryPoolAllocator<> >
143 class GenericReader {
144 // ...
145 };
146
147 typedef GenericReader<UTF8<>, UTF8<> > Reader;
148
149 } // namespace rapidjson
150 ~~~~~~~~~~
151
152 The `Reader` uses UTF-8 as both source and target encoding. The source encoding means the encoding in the JSON stream. The target encoding means the encoding of the `str` parameter in `String()` calls. For example, to parse a UTF-8 stream and output UTF-16 string events, you can define a reader by:
153
154 ~~~~~~~~~~cpp
155 GenericReader<UTF8<>, UTF16<> > reader;
156 ~~~~~~~~~~
157
158 Note that, the default character type of `UTF16` is `wchar_t`. So this `reader` needs to call `String(const wchar_t*, SizeType, bool)` of the handler.
159
160 The third template parameter `Allocator` is the allocator type for internal data structure (actually a stack).
161
162 ## Parsing {#SaxParsing}
163
164 The main function of `Reader` is used to parse JSON.
165
166 ~~~~~~~~~~cpp
167 template <unsigned parseFlags, typename InputStream, typename Handler>
168 bool Parse(InputStream& is, Handler& handler);
169
170 // with parseFlags = kDefaultParseFlags
171 template <typename InputStream, typename Handler>
172 bool Parse(InputStream& is, Handler& handler);
173 ~~~~~~~~~~
174
175 If an error occurs during parsing, it will return `false`. User can also call `bool HasParseError()`, `ParseErrorCode GetParseErrorCode()` and `size_t GetErrorOffset()` to obtain the error states. In fact, `Document` uses these `Reader` functions to obtain parse errors. Please refer to [DOM](doc/dom.md) for details about parse errors.
176
177 ## Token-by-Token Parsing {#TokenByTokenParsing}
178
179 Some users may wish to parse a JSON input stream a single token at a time, instead of immediately parsing an entire document without stopping. To parse JSON this way, instead of calling `Parse`, you can use the `IterativeParse` set of functions:
180
181 ~~~~~~~~~~cpp
182 void IterativeParseInit();
183
184 template <unsigned parseFlags, typename InputStream, typename Handler>
185 bool IterativeParseNext(InputStream& is, Handler& handler);
186
187 bool IterativeParseComplete();
188 ~~~~~~~~~~
189
190 Here is an example of iteratively parsing JSON, token by token:
191
192 ~~~~~~~~~~cpp
193 reader.IterativeParseInit();
194 while (!reader.IterativeParseComplete()) {
195 reader.IterativeParseNext<kParseDefaultFlags>(is, handler);
196 // Your handler has been called once.
197 }
198 ~~~~~~~~~~
199
200 # Writer {#Writer}
201
202 `Reader` converts (parses) JSON into events. `Writer` does exactly the opposite. It converts events into JSON.
203
204 `Writer` is very easy to use. If your application only need to converts some data into JSON, it may be a good choice to use `Writer` directly, instead of building a `Document` and then stringifying it with a `Writer`.
205
206 In `simplewriter` example, we do exactly the reverse of `simplereader`.
207
208 ~~~~~~~~~~cpp
209 #include "rapidjson/writer.h"
210 #include "rapidjson/stringbuffer.h"
211 #include <iostream>
212
213 using namespace rapidjson;
214 using namespace std;
215
216 void main() {
217 StringBuffer s;
218 Writer<StringBuffer> writer(s);
219
220 writer.StartObject();
221 writer.Key("hello");
222 writer.String("world");
223 writer.Key("t");
224 writer.Bool(true);
225 writer.Key("f");
226 writer.Bool(false);
227 writer.Key("n");
228 writer.Null();
229 writer.Key("i");
230 writer.Uint(123);
231 writer.Key("pi");
232 writer.Double(3.1416);
233 writer.Key("a");
234 writer.StartArray();
235 for (unsigned i = 0; i < 4; i++)
236 writer.Uint(i);
237 writer.EndArray();
238 writer.EndObject();
239
240 cout << s.GetString() << endl;
241 }
242 ~~~~~~~~~~
243
244 ~~~~~~~~~~
245 {"hello":"world","t":true,"f":false,"n":null,"i":123,"pi":3.1416,"a":[0,1,2,3]}
246 ~~~~~~~~~~
247
248 There are two `String()` and `Key()` overloads. One is the same as defined in handler concept with 3 parameters. It can handle string with null characters. Another one is the simpler version used in the above example.
249
250 Note that, the example code does not pass any parameters in `EndArray()` and `EndObject()`. An `SizeType` can be passed but it will be simply ignored by `Writer`.
251
252 You may doubt that, why not just using `sprintf()` or `std::stringstream` to build a JSON?
253
254 There are various reasons:
255 1. `Writer` must output a well-formed JSON. If there is incorrect event sequence (e.g. `Int()` just after `StartObject()`), it generates assertion fail in debug mode.
256 2. `Writer::String()` can handle string escaping (e.g. converting code point `U+000A` to `\n`) and Unicode transcoding.
257 3. `Writer` handles number output consistently.
258 4. `Writer` implements the event handler concept. It can be used to handle events from `Reader`, `Document` or other event publisher.
259 5. `Writer` can be optimized for different platforms.
260
261 Anyway, using `Writer` API is even simpler than generating a JSON by ad hoc methods.
262
263 ## Template {#WriterTemplate}
264
265 `Writer` has a minor design difference to `Reader`. `Writer` is a template class, not a typedef. There is no `GenericWriter`. The following is the declaration.
266
267 ~~~~~~~~~~cpp
268 namespace rapidjson {
269
270 template<typename OutputStream, typename SourceEncoding = UTF8<>, typename TargetEncoding = UTF8<>, typename Allocator = CrtAllocator<>, unsigned writeFlags = kWriteDefaultFlags>
271 class Writer {
272 public:
273 Writer(OutputStream& os, Allocator* allocator = 0, size_t levelDepth = kDefaultLevelDepth)
274 // ...
275 };
276
277 } // namespace rapidjson
278 ~~~~~~~~~~
279
280 The `OutputStream` template parameter is the type of output stream. It cannot be deduced and must be specified by user.
281
282 The `SourceEncoding` template parameter specifies the encoding to be used in `String(const Ch*, ...)`.
283
284 The `TargetEncoding` template parameter specifies the encoding in the output stream.
285
286 The `Allocator` is the type of allocator, which is used for allocating internal data structure (a stack).
287
288 The `writeFlags` are combination of the following bit-flags:
289
290 Parse flags | Meaning
291 ------------------------------|-----------------------------------
292 `kWriteNoFlags` | No flag is set.
293 `kWriteDefaultFlags` | Default write flags. It is equal to macro `RAPIDJSON_WRITE_DEFAULT_FLAGS`, which is defined as `kWriteNoFlags`.
294 `kWriteValidateEncodingFlag` | Validate encoding of JSON strings.
295 `kWriteNanAndInfFlag` | Allow writing of `Infinity`, `-Infinity` and `NaN`.
296
297 Besides, the constructor of `Writer` has a `levelDepth` parameter. This parameter affects the initial memory allocated for storing information per hierarchy level.
298
299 ## PrettyWriter {#PrettyWriter}
300
301 While the output of `Writer` is the most condensed JSON without white-spaces, suitable for network transfer or storage, it is not easily readable by human.
302
303 Therefore, RapidJSON provides a `PrettyWriter`, which adds indentation and line feeds in the output.
304
305 The usage of `PrettyWriter` is exactly the same as `Writer`, expect that `PrettyWriter` provides a `SetIndent(Ch indentChar, unsigned indentCharCount)` function. The default is 4 spaces.
306
307 ## Completeness and Reset {#CompletenessReset}
308
309 A `Writer` can only output a single JSON, which can be any JSON type at the root. Once the singular event for root (e.g. `String()`), or the last matching `EndObject()` or `EndArray()` event, is handled, the output JSON is well-formed and complete. User can detect this state by calling `Writer::IsComplete()`.
310
311 When a JSON is complete, the `Writer` cannot accept any new events. Otherwise the output will be invalid (i.e. having more than one root). To reuse the `Writer` object, user can call `Writer::Reset(OutputStream& os)` to reset all internal states of the `Writer` with a new output stream.
312
313 # Techniques {#SaxTechniques}
314
315 ## Parsing JSON to Custom Data Structure {#CustomDataStructure}
316
317 `Document`'s parsing capability is completely based on `Reader`. Actually `Document` is a handler which receives events from a reader to build a DOM during parsing.
318
319 User may uses `Reader` to build other data structures directly. This eliminates building of DOM, thus reducing memory and improving performance.
320
321 In the following `messagereader` example, `ParseMessages()` parses a JSON which should be an object with key-string pairs.
322
323 ~~~~~~~~~~cpp
324 #include "rapidjson/reader.h"
325 #include "rapidjson/error/en.h"
326 #include <iostream>
327 #include <string>
328 #include <map>
329
330 using namespace std;
331 using namespace rapidjson;
332
333 typedef map<string, string> MessageMap;
334
335 struct MessageHandler
336 : public BaseReaderHandler<UTF8<>, MessageHandler> {
337 MessageHandler() : state_(kExpectObjectStart) {
338 }
339
340 bool StartObject() {
341 switch (state_) {
342 case kExpectObjectStart:
343 state_ = kExpectNameOrObjectEnd;
344 return true;
345 default:
346 return false;
347 }
348 }
349
350 bool String(const char* str, SizeType length, bool) {
351 switch (state_) {
352 case kExpectNameOrObjectEnd:
353 name_ = string(str, length);
354 state_ = kExpectValue;
355 return true;
356 case kExpectValue:
357 messages_.insert(MessageMap::value_type(name_, string(str, length)));
358 state_ = kExpectNameOrObjectEnd;
359 return true;
360 default:
361 return false;
362 }
363 }
364
365 bool EndObject(SizeType) { return state_ == kExpectNameOrObjectEnd; }
366
367 bool Default() { return false; } // All other events are invalid.
368
369 MessageMap messages_;
370 enum State {
371 kExpectObjectStart,
372 kExpectNameOrObjectEnd,
373 kExpectValue,
374 }state_;
375 std::string name_;
376 };
377
378 void ParseMessages(const char* json, MessageMap& messages) {
379 Reader reader;
380 MessageHandler handler;
381 StringStream ss(json);
382 if (reader.Parse(ss, handler))
383 messages.swap(handler.messages_); // Only change it if success.
384 else {
385 ParseErrorCode e = reader.GetParseErrorCode();
386 size_t o = reader.GetErrorOffset();
387 cout << "Error: " << GetParseError_En(e) << endl;;
388 cout << " at offset " << o << " near '" << string(json).substr(o, 10) << "...'" << endl;
389 }
390 }
391
392 int main() {
393 MessageMap messages;
394
395 const char* json1 = "{ \"greeting\" : \"Hello!\", \"farewell\" : \"bye-bye!\" }";
396 cout << json1 << endl;
397 ParseMessages(json1, messages);
398
399 for (MessageMap::const_iterator itr = messages.begin(); itr != messages.end(); ++itr)
400 cout << itr->first << ": " << itr->second << endl;
401
402 cout << endl << "Parse a JSON with invalid schema." << endl;
403 const char* json2 = "{ \"greeting\" : \"Hello!\", \"farewell\" : \"bye-bye!\", \"foo\" : {} }";
404 cout << json2 << endl;
405 ParseMessages(json2, messages);
406
407 return 0;
408 }
409 ~~~~~~~~~~
410
411 ~~~~~~~~~~
412 { "greeting" : "Hello!", "farewell" : "bye-bye!" }
413 farewell: bye-bye!
414 greeting: Hello!
415
416 Parse a JSON with invalid schema.
417 { "greeting" : "Hello!", "farewell" : "bye-bye!", "foo" : {} }
418 Error: Terminate parsing due to Handler error.
419 at offset 59 near '} }...'
420 ~~~~~~~~~~
421
422 The first JSON (`json1`) was successfully parsed into `MessageMap`. Since `MessageMap` is a `std::map`, the printing order are sorted by the key. This order is different from the JSON's order.
423
424 In the second JSON (`json2`), `foo`'s value is an empty object. As it is an object, `MessageHandler::StartObject()` will be called. However, at that moment `state_ = kExpectValue`, so that function returns `false` and cause the parsing process be terminated. The error code is `kParseErrorTermination`.
425
426 ## Filtering of JSON {#Filtering}
427
428 As mentioned earlier, `Writer` can handle the events published by `Reader`. `condense` example simply set a `Writer` as handler of a `Reader`, so it can remove all white-spaces in JSON. `pretty` example uses the same relationship, but replacing `Writer` by `PrettyWriter`. So `pretty` can be used to reformat a JSON with indentation and line feed.
429
430 Actually, we can add intermediate layer(s) to filter the contents of JSON via these SAX-style API. For example, `capitalize` example capitalize all strings in a JSON.
431
432 ~~~~~~~~~~cpp
433 #include "rapidjson/reader.h"
434 #include "rapidjson/writer.h"
435 #include "rapidjson/filereadstream.h"
436 #include "rapidjson/filewritestream.h"
437 #include "rapidjson/error/en.h"
438 #include <vector>
439 #include <cctype>
440
441 using namespace rapidjson;
442
443 template<typename OutputHandler>
444 struct CapitalizeFilter {
445 CapitalizeFilter(OutputHandler& out) : out_(out), buffer_() {
446 }
447
448 bool Null() { return out_.Null(); }
449 bool Bool(bool b) { return out_.Bool(b); }
450 bool Int(int i) { return out_.Int(i); }
451 bool Uint(unsigned u) { return out_.Uint(u); }
452 bool Int64(int64_t i) { return out_.Int64(i); }
453 bool Uint64(uint64_t u) { return out_.Uint64(u); }
454 bool Double(double d) { return out_.Double(d); }
455 bool RawNumber(const char* str, SizeType length, bool copy) { return out_.RawNumber(str, length, copy); }
456 bool String(const char* str, SizeType length, bool) {
457 buffer_.clear();
458 for (SizeType i = 0; i < length; i++)
459 buffer_.push_back(std::toupper(str[i]));
460 return out_.String(&buffer_.front(), length, true); // true = output handler need to copy the string
461 }
462 bool StartObject() { return out_.StartObject(); }
463 bool Key(const char* str, SizeType length, bool copy) { return String(str, length, copy); }
464 bool EndObject(SizeType memberCount) { return out_.EndObject(memberCount); }
465 bool StartArray() { return out_.StartArray(); }
466 bool EndArray(SizeType elementCount) { return out_.EndArray(elementCount); }
467
468 OutputHandler& out_;
469 std::vector<char> buffer_;
470 };
471
472 int main(int, char*[]) {
473 // Prepare JSON reader and input stream.
474 Reader reader;
475 char readBuffer[65536];
476 FileReadStream is(stdin, readBuffer, sizeof(readBuffer));
477
478 // Prepare JSON writer and output stream.
479 char writeBuffer[65536];
480 FileWriteStream os(stdout, writeBuffer, sizeof(writeBuffer));
481 Writer<FileWriteStream> writer(os);
482
483 // JSON reader parse from the input stream and let writer generate the output.
484 CapitalizeFilter<Writer<FileWriteStream> > filter(writer);
485 if (!reader.Parse(is, filter)) {
486 fprintf(stderr, "\nError(%u): %s\n", (unsigned)reader.GetErrorOffset(), GetParseError_En(reader.GetParseErrorCode()));
487 return 1;
488 }
489
490 return 0;
491 }
492 ~~~~~~~~~~
493
494 Note that, it is incorrect to simply capitalize the JSON as a string. For example:
495 ~~~~~~~~~~
496 ["Hello\nWorld"]
497 ~~~~~~~~~~
498
499 Simply capitalizing the whole JSON would contain incorrect escape character:
500 ~~~~~~~~~~
501 ["HELLO\NWORLD"]
502 ~~~~~~~~~~
503
504 The correct result by `capitalize`:
505 ~~~~~~~~~~
506 ["HELLO\nWORLD"]
507 ~~~~~~~~~~
508
509 More complicated filters can be developed. However, since SAX-style API can only provide information about a single event at a time, user may need to book-keeping the contextual information (e.g. the path from root value, storage of other related values). Some processing may be easier to be implemented in DOM than SAX.