| 1 |
# JSONStream |
| 2 |
|
| 3 |
streaming JSON.parse and stringify |
| 4 |
|
| 5 |
 |
| 6 |
|
| 7 |
## install |
| 8 |
```npm install JSONStream``` |
| 9 |
|
| 10 |
## example |
| 11 |
|
| 12 |
``` js |
| 13 |
|
| 14 |
var request = require('request') |
| 15 |
, JSONStream = require('JSONStream') |
| 16 |
, es = require('event-stream') |
| 17 |
|
| 18 |
request({url: 'http://isaacs.couchone.com/registry/_all_docs'}) |
| 19 |
.pipe(JSONStream.parse('rows.*')) |
| 20 |
.pipe(es.mapSync(function (data) { |
| 21 |
console.error(data) |
| 22 |
return data |
| 23 |
})) |
| 24 |
``` |
| 25 |
|
| 26 |
## JSONStream.parse(path) |
| 27 |
|
| 28 |
parse stream of values that match a path |
| 29 |
|
| 30 |
``` js |
| 31 |
JSONStream.parse('rows.*.doc') |
| 32 |
``` |
| 33 |
|
| 34 |
The `..` operator is the recursive descent operator from [JSONPath](http://goessner.net/articles/JsonPath/), which will match a child at any depth (see examples below). |
| 35 |
|
| 36 |
If your keys have keys that include `.` or `*` etc, use an array instead. |
| 37 |
`['row', true, /^doc/]`. |
| 38 |
|
| 39 |
If you use an array, `RegExp`s, booleans, and/or functions. The `..` operator is also available in array representation, using `{recurse: true}`. |
| 40 |
any object that matches the path will be emitted as 'data' (and `pipe`d down stream) |
| 41 |
|
| 42 |
If `path` is empty or null, no 'data' events are emitted. |
| 43 |
|
| 44 |
If you want to have keys emitted, you can prefix your `*` operator with `$`: `obj.$*` - in this case the data passed to the stream is an object with a `key` holding the key and a `value` property holding the data. |
| 45 |
|
| 46 |
### Examples |
| 47 |
|
| 48 |
query a couchdb view: |
| 49 |
|
| 50 |
``` bash |
| 51 |
curl -sS localhost:5984/tests/_all_docs&include_docs=true |
| 52 |
``` |
| 53 |
you will get something like this: |
| 54 |
|
| 55 |
``` js |
| 56 |
{"total_rows":129,"offset":0,"rows":[ |
| 57 |
{ "id":"change1_0.6995461115147918" |
| 58 |
, "key":"change1_0.6995461115147918" |
| 59 |
, "value":{"rev":"1-e240bae28c7bb3667f02760f6398d508"} |
| 60 |
, "doc":{ |
| 61 |
"_id": "change1_0.6995461115147918" |
| 62 |
, "_rev": "1-e240bae28c7bb3667f02760f6398d508","hello":1} |
| 63 |
}, |
| 64 |
{ "id":"change2_0.6995461115147918" |
| 65 |
, "key":"change2_0.6995461115147918" |
| 66 |
, "value":{"rev":"1-13677d36b98c0c075145bb8975105153"} |
| 67 |
, "doc":{ |
| 68 |
"_id":"change2_0.6995461115147918" |
| 69 |
, "_rev":"1-13677d36b98c0c075145bb8975105153" |
| 70 |
, "hello":2 |
| 71 |
} |
| 72 |
}, |
| 73 |
]} |
| 74 |
|
| 75 |
``` |
| 76 |
|
| 77 |
we are probably most interested in the `rows.*.doc` |
| 78 |
|
| 79 |
create a `Stream` that parses the documents from the feed like this: |
| 80 |
|
| 81 |
``` js |
| 82 |
var stream = JSONStream.parse(['rows', true, 'doc']) //rows, ANYTHING, doc |
| 83 |
|
| 84 |
stream.on('data', function(data) { |
| 85 |
console.log('received:', data); |
| 86 |
}); |
| 87 |
//emits anything from _before_ the first match |
| 88 |
stream.on('header', function (data) { |
| 89 |
console.log('header:', data) // => {"total_rows":129,"offset":0} |
| 90 |
}) |
| 91 |
|
| 92 |
``` |
| 93 |
awesome! |
| 94 |
|
| 95 |
In case you wanted the contents the doc emitted: |
| 96 |
|
| 97 |
``` js |
| 98 |
var stream = JSONStream.parse(['rows', true, 'doc', {emitKey: true}]) //rows, ANYTHING, doc, items in docs with keys |
| 99 |
|
| 100 |
stream.on('data', function(data) { |
| 101 |
console.log('key:', data.key); |
| 102 |
console.log('value:', data.value); |
| 103 |
}); |
| 104 |
|
| 105 |
``` |
| 106 |
|
| 107 |
You can also emit the path: |
| 108 |
|
| 109 |
``` js |
| 110 |
var stream = JSONStream.parse(['rows', true, 'doc', {emitPath: true}]) //rows, ANYTHING, doc, items in docs with keys |
| 111 |
|
| 112 |
stream.on('data', function(data) { |
| 113 |
console.log('path:', data.path); |
| 114 |
console.log('value:', data.value); |
| 115 |
}); |
| 116 |
|
| 117 |
``` |
| 118 |
|
| 119 |
### recursive patterns (..) |
| 120 |
|
| 121 |
`JSONStream.parse('docs..value')` |
| 122 |
(or `JSONStream.parse(['docs', {recurse: true}, 'value'])` using an array) |
| 123 |
will emit every `value` object that is a child, grand-child, etc. of the |
| 124 |
`docs` object. In this example, it will match exactly 5 times at various depth |
| 125 |
levels, emitting 0, 1, 2, 3 and 4 as results. |
| 126 |
|
| 127 |
```js |
| 128 |
{ |
| 129 |
"total": 5, |
| 130 |
"docs": [ |
| 131 |
{ |
| 132 |
"key": { |
| 133 |
"value": 0, |
| 134 |
"some": "property" |
| 135 |
} |
| 136 |
}, |
| 137 |
{"value": 1}, |
| 138 |
{"value": 2}, |
| 139 |
{"blbl": [{}, {"a":0, "b":1, "value":3}, 10]}, |
| 140 |
{"value": 4} |
| 141 |
] |
| 142 |
} |
| 143 |
``` |
| 144 |
|
| 145 |
## JSONStream.parse(pattern, map) |
| 146 |
|
| 147 |
provide a function that can be used to map or filter |
| 148 |
the json output. `map` is passed the value at that node of the pattern, |
| 149 |
if `map` return non-nullish (anything but `null` or `undefined`) |
| 150 |
that value will be emitted in the stream. If it returns a nullish value, |
| 151 |
nothing will be emitted. |
| 152 |
|
| 153 |
`JSONStream` also emits `'header'` and `'footer'` events, |
| 154 |
the `'header'` event contains anything in the output that was before |
| 155 |
the first match, and the `'footer'`, is anything after the last match. |
| 156 |
|
| 157 |
## JSONStream.stringify(open, sep, close) |
| 158 |
|
| 159 |
Create a writable stream. |
| 160 |
|
| 161 |
you may pass in custom `open`, `close`, and `seperator` strings. |
| 162 |
But, by default, `JSONStream.stringify()` will create an array, |
| 163 |
(with default options `open='[\n', sep='\n,\n', close='\n]\n'`) |
| 164 |
|
| 165 |
If you call `JSONStream.stringify(false)` |
| 166 |
the elements will only be seperated by a newline. |
| 167 |
|
| 168 |
If you only write one item this will be valid JSON. |
| 169 |
|
| 170 |
If you write many items, |
| 171 |
you can use a `RegExp` to split it into valid chunks. |
| 172 |
|
| 173 |
## JSONStream.stringifyObject(open, sep, close) |
| 174 |
|
| 175 |
Very much like `JSONStream.stringify`, |
| 176 |
but creates a writable stream for objects instead of arrays. |
| 177 |
|
| 178 |
Accordingly, `open='{\n', sep='\n,\n', close='\n}\n'`. |
| 179 |
|
| 180 |
When you `.write()` to the stream you must supply an array with `[ key, data ]` |
| 181 |
as the first argument. |
| 182 |
|
| 183 |
## unix tool |
| 184 |
|
| 185 |
query npm to see all the modules that browserify has ever depended on. |
| 186 |
|
| 187 |
``` bash |
| 188 |
curl https://registry.npmjs.org/browserify | JSONStream 'versions.*.dependencies' |
| 189 |
``` |
| 190 |
|
| 191 |
## numbers |
| 192 |
|
| 193 |
numbers will be emitted as numbers. |
| 194 |
huge numbers that cannot be represented in memory as javascript numbers will be emitted as strings. |
| 195 |
cf https://github.com/creationix/jsonparse/commit/044b268f01c4b8f97fb936fc85d3bcfba179e5bb for details. |
| 196 |
|
| 197 |
## Acknowlegements |
| 198 |
|
| 199 |
this module depends on https://github.com/creationix/jsonparse |
| 200 |
by Tim Caswell |
| 201 |
and also thanks to Florent Jaby for teaching me about parsing with: |
| 202 |
https://github.com/Floby/node-json-streams |
| 203 |
|
| 204 |
## license |
| 205 |
|
| 206 |
Dual-licensed under the MIT License or the Apache License, version 2.0 |
| 207 |
|