Debian Codegraph @

lualib/lua_shape/README.md

raw
1 # lua_shape
2
3 A comprehensive schema validation and transformation library for Rspamd, designed to replace tableshape with improved error reporting, documentation generation, and export capabilities.
4
5 ## Features
6
7 1. **Better Error Reporting**: Structured error trees with intersection analysis for `one_of` types
8 2. **Documentation Generation**: Extract structured documentation from schemas
9 3. **Type Constraints**: Numeric ranges, string lengths, patterns, and more
10 4. **First-class Mixins**: Field composition with origin tracking
11 5. **JSON Schema Export**: Export schemas for UCL validation
12 6. **Transform Support**: Immutable transformations with validation
13 7. **Pure Lua**: No dependencies on external modules (except optional lpeg for patterns)
14
15 ## Quick Start
16
17 ```lua
18 local T = require "lua_shape.core"
19
20 -- Define a schema
21 local config_schema = T.table({
22 host = T.string({ min_len = 1 }),
23 port = T.integer({ min = 1, max = 65535 }):with_default(8080),
24 timeout = T.number({ min = 0 }):optional(),
25 ssl = T.boolean():with_default(false)
26 })
27
28 -- Validate configuration
29 local ok, result = config_schema:check({
30 host = "localhost",
31 port = 3000
32 })
33
34 if not ok then
35 print("Validation error:")
36 print(T.format_error(result))
37 end
38
39 -- Transform with defaults applied
40 local ok, config = config_schema:transform({
41 host = "example.com"
42 })
43 -- config.port == 8080 (default applied)
44 -- config.ssl == false (default applied)
45 ```
46
47 ## Core Types
48
49 ### Scalars
50
51 - `T.string(opts)` - String with optional constraints
52 - `min_len`, `max_len` - Length constraints
53 - `pattern` - Lua pattern for validation (e.g., `"^%d+$"` for digits only)
54 - `lpeg` - Optional lpeg pattern for complex parsing
55 - `T.number(opts)` - Number with optional range constraints (min, max)
56 - Accepts both numbers and values convertible via `tonumber`
57 - `T.integer(opts)` - Integer (number with integer constraint)
58 - `T.boolean()` - Boolean value
59 - `T.callable()` - Function/callable value
60 - `T.enum(values)` - One of a fixed set of values
61 - `T.literal(value)` - Exact value match
62
63 ### Structured Types
64
65 - `T.array(item_schema, opts)` - Array with item validation
66 - Underlying table must be a dense, 1-indexed array (no sparse or string keys)
67 - `min_items`, `max_items` - Size constraints
68 - `T.table(fields, opts)` - Table/object with field schemas
69 - `open = true` - Allow additional fields not defined in schema
70 - `open = false` (default) - Reject unknown fields
71 - `extra = schema` - Schema for validating extra fields
72 - `mixins` - Array of mixin schemas for composition (applied when the schema is resolved via the registry)
73 - `T.one_of(variants)` - Sum type (match exactly one alternative)
74
75 ### Composition
76
77 - `schema:optional()` - Make schema optional
78 - `schema:with_default(value)` - Add default value (can be a function for dynamic defaults)
79 - `schema:doc(doc_table)` - Add documentation
80 - `schema:transform_with(fn)` - Apply transformation
81 - `T.transform(schema, fn)` - Transform wrapper
82 - `T.ref(id)` - Reference to registered schema
83 - `T.mixin(schema, opts)` - Mixin for table composition
84
85 ## Examples
86
87 ### Basic Types with Constraints
88
89 ```lua
90 -- String with length constraint
91 local name_schema = T.string({ min_len = 3, max_len = 50 })
92
93 -- String with Lua pattern (validates format)
94 local email_schema = T.string({ pattern = "^[%w%.]+@[%w%.]+$" })
95 local ipv4_schema = T.string({ pattern = "^%d+%.%d+%.%d+%.%d+$" })
96
97 -- Integer with range
98 local age_schema = T.integer({ min = 0, max = 150 })
99
100 -- Enum
101 local level_schema = T.enum({"debug", "info", "warning", "error"})
102 ```
103
104 ### Arrays and Tables
105
106 ```lua
107 -- Array of strings
108 local tags_schema = T.array(T.string())
109
110 -- Table with required and optional fields
111 local user_schema = T.table({
112 name = T.string(),
113 email = T.string(),
114 age = T.integer():optional(),
115 role = T.enum({"admin", "user"}):with_default("user")
116 })
117
118 -- Closed table (default): rejects unknown fields
119 local strict_config = T.table({
120 host = T.string(),
121 port = T.integer()
122 }, { open = false })
123
124 -- Open table: allows additional fields not in schema
125 local flexible_config = T.table({
126 host = T.string(),
127 port = T.integer()
128 }, { open = true })
129 -- Accepts: { host = "localhost", port = 8080, custom_field = "value" }
130 ```
131
132 ### one_of with Intersection
133
134 ```lua
135 -- Multiple config variants
136 local config_schema = T.one_of({
137 {
138 name = "file_config",
139 schema = T.table({
140 type = T.literal("file"),
141 path = T.string()
142 })
143 },
144 {
145 name = "redis_config",
146 schema = T.table({
147 type = T.literal("redis"),
148 host = T.string(),
149 port = T.integer():with_default(6379)
150 })
151 }
152 })
153
154 -- Error messages show intersection:
155 -- "all alternatives require: type (string)"
156 ```
157
158 ### Transforms
159
160 `T.transform(accepted_type, transformer)` validates input against `accepted_type`, then applies `transformer` function.
161
162 ```lua
163 -- Accept string, convert to number
164 local num_from_string = T.transform(T.string(), tonumber)
165
166 -- Accept number or string, convert both to number
167 local flexible_number = T.one_of({
168 T.number(),
169 T.transform(T.string(), tonumber)
170 })
171
172 -- Accept string, parse time interval to number
173 local timeout_schema = T.one_of({
174 T.number({ min = 0 }),
175 T.transform(T.string(), parse_time_interval) -- "5s" -> 5.0
176 })
177 ```
178
179 **Semantics:**
180 1. Input is validated against accepted type (first argument)
181 2. If valid, transformer function is called with pcall
182 3. If transformer returns `nil` or errors, validation fails
183 4. Otherwise, result is accepted without type checking
184
185 > **Note:** Transform functions run only in `schema:transform(...)` mode. In `schema:check(...)` mode, only the input type is validated.
186
187 ### Callable Defaults
188
189 Defaults can be functions that are called each time a default is needed:
190
191 ```lua
192 local function get_current_timestamp()
193 return os.time()
194 end
195
196 local event_schema = T.table({
197 name = T.string(),
198 timestamp = T.number():with_default(get_current_timestamp), -- Function called each time
199 priority = T.integer():with_default(0) -- Static default
200 })
201
202 -- Each transform gets a fresh timestamp
203 local ok, event1 = event_schema:transform({ name = "login" })
204 -- event1.timestamp will be the current time when transform was called
205 ```
206
207 ### Schema Registry
208
209 ```lua
210 local Registry = require "lua_shape.registry"
211 local reg = Registry.global()
212
213 -- Define reusable schemas
214 local redis_schema = reg:define("redis.options", T.table({
215 servers = T.array(T.string()),
216 db = T.integer({ min = 0, max = 15 }):with_default(0)
217 }))
218
219 -- Reference in other schemas
220 local app_schema = T.table({
221 cache = T.ref("redis.options")
222 })
223
224 -- Resolve references
225 local resolved = reg:resolve_schema(app_schema)
226 -- Validate/transforms should use the resolved schema so mixins/references are applied
227 local ok, cfg_or_err = resolved:transform({
228 cache = {
229 servers = {"redis:6379"}
230 }
231 })
232 ```
233
234 ### Mixins with Origin Tracking
235
236 ```lua
237 -- Base mixin
238 local redis_mixin = T.table({
239 redis_host = T.string(),
240 redis_port = T.integer():with_default(6379)
241 })
242
243 -- Use mixin in another schema
244 local plugin_schema = T.table({
245 enabled = T.boolean(),
246 plugin_option = T.string()
247 }, {
248 mixins = {
249 T.mixin(redis_mixin, { as = "redis" })
250 }
251 })
252
253 -- Documentation will show:
254 -- Direct fields: enabled, plugin_option
255 -- Mixin "redis": redis_host, redis_port
256 ```
257
258 Mixins are merged into the resulting table schema by `Registry:resolve_schema` (or `Registry:define`). Always validate against the resolved schema so that mixin fields participate in `:check` / `:transform` and emit proper documentation/JSON Schema output.
259
260 ### JSON Schema Export
261
262 ```lua
263 local jsonschema = require "lua_shape.jsonschema"
264
265 -- Export single schema
266 local json = jsonschema.from_schema(config_schema, {
267 id = "https://rspamd.com/schema/config",
268 title = "Application Config"
269 })
270
271 -- Export all schemas from registry
272 local all_schemas = jsonschema.export_registry(Registry.global())
273 ```
274
275 ### Documentation Generation
276
277 ```lua
278 local docs = require "lua_shape.docs"
279
280 -- Generate documentation IR
281 local doc_tree = docs.for_schema(config_schema)
282
283 -- Render as markdown
284 local markdown_lines = docs.render_markdown(doc_tree.schema_doc)
285 for _, line in ipairs(markdown_lines) do
286 print(line)
287 end
288 ```
289
290 ## Error Reporting
291
292 ### Structured Errors
293
294 Errors are represented as trees:
295
296 ```lua
297 {
298 kind = "table_invalid",
299 path = "config",
300 details = {
301 errors = {
302 port = {
303 kind = "constraint_violation",
304 path = "config.port",
305 details = { constraint = "max", expected = 65535, got = 99999 }
306 }
307 }
308 }
309 }
310 ```
311
312 ### Human-Readable Formatting
313
314 ```lua
315 local T = require "lua_shape.core"
316 print(T.format_error(error_tree))
317 ```
318
319 Output:
320 ```
321 table validation failed at config:
322 constraint violation at config.port: max (expected: 65535, got: 99999)
323 ```
324
325 ### one_of Intersection Errors
326
327 When all variants of a one_of fail, the error shows common requirements:
328
329 ```
330 value does not match any alternative at :
331 all alternatives require:
332 - name: string
333 - type: string
334 some alternatives also expect:
335 - path: string (in file_config variant)
336 - host: string (in redis_config variant)
337 tried alternatives:
338 - file_config: ...
339 - redis_config: ...
340 ```
341
342 ## API Reference
343
344 ### Core Module (`lua_shape.core`)
345
346 #### Type Constructors
347
348 - `T.string(opts?)` - String type
349 - opts: `min_len`, `max_len`, `pattern`, `lpeg`, `doc`
350 - `T.number(opts?)` - Number type
351 - opts: `min`, `max`, `doc`
352 - `T.integer(opts?)` - Integer type (number with integer=true)
353 - opts: `min`, `max`, `doc`
354 - `T.boolean(opts?)` - Boolean type
355 - `T.enum(values, opts?)` - Enum type
356 - `T.literal(value, opts?)` - Literal value type
357 - `T.array(item_schema, opts?)` - Array type
358 - opts: `min_items`, `max_items`, `doc`
359 - `T.table(fields, opts?)` - Table type
360 - opts: `open`, `extra`, `mixins`, `doc`
361 - `T.one_of(variants, opts?)` - Sum type
362 - `T.optional(schema, opts?)` - Optional wrapper
363 - `T.default(schema, value)` - Default value wrapper
364 - `T.transform(accepted_type, transformer, opts?)` - Transform wrapper (validates input against accepted_type, then applies transformer)
365 - `T.ref(id, opts?)` - Schema reference placeholder (must be resolved via the registry before validation)
366 - `T.mixin(schema, opts?)` - Mixin definition
367
368 #### Schema Methods
369
370 - `schema:check(value, ctx?)` - Validate value
371 - `schema:transform(value, ctx?)` - Transform and validate (tableshape-compatible `(result)` / `(nil, err)` contract)
372 - `schema:optional(opts?)` - Make optional
373 - `schema:with_default(value)` - Add default
374 - `schema:doc(doc_table)` - Add documentation
375 - `schema:transform_with(fn, opts?)` - Add transformation
376
377 ### Registry Module (`lua_shape.registry`)
378
379 - `Registry.global()` - Get/create global registry
380 - `registry:define(id, schema)` - Register schema with ID (returns the resolved version with mixins/reference chains applied)
381 - `registry:get(id)` - Get schema by ID
382 - `registry:resolve_schema(schema)` - Resolve references and mixins (recurses into nested arrays/one_of/options and caches the result)
383 - `registry:list()` - List all schema IDs
384 - `registry:export_all()` - Export all schemas
385
386 ### Core Utilities
387
388 The core module also includes utility functions:
389
390 - `T.format_error(err)` - Format error tree as human-readable string
391 - `T.deep_clone(value)` - Deep clone value for immutable transformations
392
393 ### JSON Schema Module (`lua_shape.jsonschema`)
394
395 - `jsonschema.from_schema(schema, opts?)` - Convert to JSON Schema
396 - `jsonschema.export_registry(registry, opts?)` - Export all schemas
397
398 ### Docs Module (`lua_shape.docs`)
399
400 - `docs.for_schema(schema, opts?)` - Generate documentation IR
401 - `docs.for_registry(registry, opts?)` - Generate docs for all schemas
402 - `docs.render_markdown(doc_tree, indent?)` - Render as markdown
403
404 ## Migration from tableshape
405
406 See [MIGRATION.md](MIGRATION.md) for detailed migration guide.
407
408 Quick reference:
409
410 | tableshape | lua_shape |
411 |------------|---------------|
412 | `ts.string` | `T.string()` |
413 | `ts.number` | `T.number()` |
414 | `ts.array_of(ts.string)` | `T.array(T.string())` |
415 | `ts.shape({...})` | `T.table({...})` |
416 | `field:is_optional()` | `field:optional()` or `{ schema = ..., optional = true }` |
417 | `ts.string + ts.number` | `T.one_of({ T.string(), T.number() })` |
418 | `ts.string / fn` | `T.string():transform_with(fn)` or `T.transform(T.string(), fn)` |
419 | `field:describe("...")` | `field:doc({ summary = "..." })` |
420
421 ## Files
422
423 - `core.lua` - Core type system, validation, and utilities
424 - `registry.lua` - Schema registration and reference resolution
425 - `jsonschema.lua` - JSON Schema export
426 - `docs.lua` - Documentation generation
427 - `MIGRATION.md` - Migration guide from tableshape
428 - `README.md` - This file
429
430 ## License
431
432 Apache License 2.0 - Same as Rspamd