| 1 |
# lua_shape |
| 2 |
|
| 3 |
A comprehensive schema validation and transformation library for Rspamd, designed to replace tableshape with improved error reporting, documentation generation, and export capabilities. |
| 4 |
|
| 5 |
## Features |
| 6 |
|
| 7 |
1. **Better Error Reporting**: Structured error trees with intersection analysis for `one_of` types |
| 8 |
2. **Documentation Generation**: Extract structured documentation from schemas |
| 9 |
3. **Type Constraints**: Numeric ranges, string lengths, patterns, and more |
| 10 |
4. **First-class Mixins**: Field composition with origin tracking |
| 11 |
5. **JSON Schema Export**: Export schemas for UCL validation |
| 12 |
6. **Transform Support**: Immutable transformations with validation |
| 13 |
7. **Pure Lua**: No dependencies on external modules (except optional lpeg for patterns) |
| 14 |
|
| 15 |
## Quick Start |
| 16 |
|
| 17 |
```lua |
| 18 |
local T = require "lua_shape.core" |
| 19 |
|
| 20 |
-- Define a schema |
| 21 |
local config_schema = T.table({ |
| 22 |
host = T.string({ min_len = 1 }), |
| 23 |
port = T.integer({ min = 1, max = 65535 }):with_default(8080), |
| 24 |
timeout = T.number({ min = 0 }):optional(), |
| 25 |
ssl = T.boolean():with_default(false) |
| 26 |
}) |
| 27 |
|
| 28 |
-- Validate configuration |
| 29 |
local ok, result = config_schema:check({ |
| 30 |
host = "localhost", |
| 31 |
port = 3000 |
| 32 |
}) |
| 33 |
|
| 34 |
if not ok then |
| 35 |
print("Validation error:") |
| 36 |
print(T.format_error(result)) |
| 37 |
end |
| 38 |
|
| 39 |
-- Transform with defaults applied |
| 40 |
local ok, config = config_schema:transform({ |
| 41 |
host = "example.com" |
| 42 |
}) |
| 43 |
-- config.port == 8080 (default applied) |
| 44 |
-- config.ssl == false (default applied) |
| 45 |
``` |
| 46 |
|
| 47 |
## Core Types |
| 48 |
|
| 49 |
### Scalars |
| 50 |
|
| 51 |
- `T.string(opts)` - String with optional constraints |
| 52 |
- `min_len`, `max_len` - Length constraints |
| 53 |
- `pattern` - Lua pattern for validation (e.g., `"^%d+$"` for digits only) |
| 54 |
- `lpeg` - Optional lpeg pattern for complex parsing |
| 55 |
- `T.number(opts)` - Number with optional range constraints (min, max) |
| 56 |
- Accepts both numbers and values convertible via `tonumber` |
| 57 |
- `T.integer(opts)` - Integer (number with integer constraint) |
| 58 |
- `T.boolean()` - Boolean value |
| 59 |
- `T.callable()` - Function/callable value |
| 60 |
- `T.enum(values)` - One of a fixed set of values |
| 61 |
- `T.literal(value)` - Exact value match |
| 62 |
|
| 63 |
### Structured Types |
| 64 |
|
| 65 |
- `T.array(item_schema, opts)` - Array with item validation |
| 66 |
- Underlying table must be a dense, 1-indexed array (no sparse or string keys) |
| 67 |
- `min_items`, `max_items` - Size constraints |
| 68 |
- `T.table(fields, opts)` - Table/object with field schemas |
| 69 |
- `open = true` - Allow additional fields not defined in schema |
| 70 |
- `open = false` (default) - Reject unknown fields |
| 71 |
- `extra = schema` - Schema for validating extra fields |
| 72 |
- `mixins` - Array of mixin schemas for composition (applied when the schema is resolved via the registry) |
| 73 |
- `T.one_of(variants)` - Sum type (match exactly one alternative) |
| 74 |
|
| 75 |
### Composition |
| 76 |
|
| 77 |
- `schema:optional()` - Make schema optional |
| 78 |
- `schema:with_default(value)` - Add default value (can be a function for dynamic defaults) |
| 79 |
- `schema:doc(doc_table)` - Add documentation |
| 80 |
- `schema:transform_with(fn)` - Apply transformation |
| 81 |
- `T.transform(schema, fn)` - Transform wrapper |
| 82 |
- `T.ref(id)` - Reference to registered schema |
| 83 |
- `T.mixin(schema, opts)` - Mixin for table composition |
| 84 |
|
| 85 |
## Examples |
| 86 |
|
| 87 |
### Basic Types with Constraints |
| 88 |
|
| 89 |
```lua |
| 90 |
-- String with length constraint |
| 91 |
local name_schema = T.string({ min_len = 3, max_len = 50 }) |
| 92 |
|
| 93 |
-- String with Lua pattern (validates format) |
| 94 |
local email_schema = T.string({ pattern = "^[%w%.]+@[%w%.]+$" }) |
| 95 |
local ipv4_schema = T.string({ pattern = "^%d+%.%d+%.%d+%.%d+$" }) |
| 96 |
|
| 97 |
-- Integer with range |
| 98 |
local age_schema = T.integer({ min = 0, max = 150 }) |
| 99 |
|
| 100 |
-- Enum |
| 101 |
local level_schema = T.enum({"debug", "info", "warning", "error"}) |
| 102 |
``` |
| 103 |
|
| 104 |
### Arrays and Tables |
| 105 |
|
| 106 |
```lua |
| 107 |
-- Array of strings |
| 108 |
local tags_schema = T.array(T.string()) |
| 109 |
|
| 110 |
-- Table with required and optional fields |
| 111 |
local user_schema = T.table({ |
| 112 |
name = T.string(), |
| 113 |
email = T.string(), |
| 114 |
age = T.integer():optional(), |
| 115 |
role = T.enum({"admin", "user"}):with_default("user") |
| 116 |
}) |
| 117 |
|
| 118 |
-- Closed table (default): rejects unknown fields |
| 119 |
local strict_config = T.table({ |
| 120 |
host = T.string(), |
| 121 |
port = T.integer() |
| 122 |
}, { open = false }) |
| 123 |
|
| 124 |
-- Open table: allows additional fields not in schema |
| 125 |
local flexible_config = T.table({ |
| 126 |
host = T.string(), |
| 127 |
port = T.integer() |
| 128 |
}, { open = true }) |
| 129 |
-- Accepts: { host = "localhost", port = 8080, custom_field = "value" } |
| 130 |
``` |
| 131 |
|
| 132 |
### one_of with Intersection |
| 133 |
|
| 134 |
```lua |
| 135 |
-- Multiple config variants |
| 136 |
local config_schema = T.one_of({ |
| 137 |
{ |
| 138 |
name = "file_config", |
| 139 |
schema = T.table({ |
| 140 |
type = T.literal("file"), |
| 141 |
path = T.string() |
| 142 |
}) |
| 143 |
}, |
| 144 |
{ |
| 145 |
name = "redis_config", |
| 146 |
schema = T.table({ |
| 147 |
type = T.literal("redis"), |
| 148 |
host = T.string(), |
| 149 |
port = T.integer():with_default(6379) |
| 150 |
}) |
| 151 |
} |
| 152 |
}) |
| 153 |
|
| 154 |
-- Error messages show intersection: |
| 155 |
-- "all alternatives require: type (string)" |
| 156 |
``` |
| 157 |
|
| 158 |
### Transforms |
| 159 |
|
| 160 |
`T.transform(accepted_type, transformer)` validates input against `accepted_type`, then applies `transformer` function. |
| 161 |
|
| 162 |
```lua |
| 163 |
-- Accept string, convert to number |
| 164 |
local num_from_string = T.transform(T.string(), tonumber) |
| 165 |
|
| 166 |
-- Accept number or string, convert both to number |
| 167 |
local flexible_number = T.one_of({ |
| 168 |
T.number(), |
| 169 |
T.transform(T.string(), tonumber) |
| 170 |
}) |
| 171 |
|
| 172 |
-- Accept string, parse time interval to number |
| 173 |
local timeout_schema = T.one_of({ |
| 174 |
T.number({ min = 0 }), |
| 175 |
T.transform(T.string(), parse_time_interval) -- "5s" -> 5.0 |
| 176 |
}) |
| 177 |
``` |
| 178 |
|
| 179 |
**Semantics:** |
| 180 |
1. Input is validated against accepted type (first argument) |
| 181 |
2. If valid, transformer function is called with pcall |
| 182 |
3. If transformer returns `nil` or errors, validation fails |
| 183 |
4. Otherwise, result is accepted without type checking |
| 184 |
|
| 185 |
> **Note:** Transform functions run only in `schema:transform(...)` mode. In `schema:check(...)` mode, only the input type is validated. |
| 186 |
|
| 187 |
### Callable Defaults |
| 188 |
|
| 189 |
Defaults can be functions that are called each time a default is needed: |
| 190 |
|
| 191 |
```lua |
| 192 |
local function get_current_timestamp() |
| 193 |
return os.time() |
| 194 |
end |
| 195 |
|
| 196 |
local event_schema = T.table({ |
| 197 |
name = T.string(), |
| 198 |
timestamp = T.number():with_default(get_current_timestamp), -- Function called each time |
| 199 |
priority = T.integer():with_default(0) -- Static default |
| 200 |
}) |
| 201 |
|
| 202 |
-- Each transform gets a fresh timestamp |
| 203 |
local ok, event1 = event_schema:transform({ name = "login" }) |
| 204 |
-- event1.timestamp will be the current time when transform was called |
| 205 |
``` |
| 206 |
|
| 207 |
### Schema Registry |
| 208 |
|
| 209 |
```lua |
| 210 |
local Registry = require "lua_shape.registry" |
| 211 |
local reg = Registry.global() |
| 212 |
|
| 213 |
-- Define reusable schemas |
| 214 |
local redis_schema = reg:define("redis.options", T.table({ |
| 215 |
servers = T.array(T.string()), |
| 216 |
db = T.integer({ min = 0, max = 15 }):with_default(0) |
| 217 |
})) |
| 218 |
|
| 219 |
-- Reference in other schemas |
| 220 |
local app_schema = T.table({ |
| 221 |
cache = T.ref("redis.options") |
| 222 |
}) |
| 223 |
|
| 224 |
-- Resolve references |
| 225 |
local resolved = reg:resolve_schema(app_schema) |
| 226 |
-- Validate/transforms should use the resolved schema so mixins/references are applied |
| 227 |
local ok, cfg_or_err = resolved:transform({ |
| 228 |
cache = { |
| 229 |
servers = {"redis:6379"} |
| 230 |
} |
| 231 |
}) |
| 232 |
``` |
| 233 |
|
| 234 |
### Mixins with Origin Tracking |
| 235 |
|
| 236 |
```lua |
| 237 |
-- Base mixin |
| 238 |
local redis_mixin = T.table({ |
| 239 |
redis_host = T.string(), |
| 240 |
redis_port = T.integer():with_default(6379) |
| 241 |
}) |
| 242 |
|
| 243 |
-- Use mixin in another schema |
| 244 |
local plugin_schema = T.table({ |
| 245 |
enabled = T.boolean(), |
| 246 |
plugin_option = T.string() |
| 247 |
}, { |
| 248 |
mixins = { |
| 249 |
T.mixin(redis_mixin, { as = "redis" }) |
| 250 |
} |
| 251 |
}) |
| 252 |
|
| 253 |
-- Documentation will show: |
| 254 |
-- Direct fields: enabled, plugin_option |
| 255 |
-- Mixin "redis": redis_host, redis_port |
| 256 |
``` |
| 257 |
|
| 258 |
Mixins are merged into the resulting table schema by `Registry:resolve_schema` (or `Registry:define`). Always validate against the resolved schema so that mixin fields participate in `:check` / `:transform` and emit proper documentation/JSON Schema output. |
| 259 |
|
| 260 |
### JSON Schema Export |
| 261 |
|
| 262 |
```lua |
| 263 |
local jsonschema = require "lua_shape.jsonschema" |
| 264 |
|
| 265 |
-- Export single schema |
| 266 |
local json = jsonschema.from_schema(config_schema, { |
| 267 |
id = "https://rspamd.com/schema/config", |
| 268 |
title = "Application Config" |
| 269 |
}) |
| 270 |
|
| 271 |
-- Export all schemas from registry |
| 272 |
local all_schemas = jsonschema.export_registry(Registry.global()) |
| 273 |
``` |
| 274 |
|
| 275 |
### Documentation Generation |
| 276 |
|
| 277 |
```lua |
| 278 |
local docs = require "lua_shape.docs" |
| 279 |
|
| 280 |
-- Generate documentation IR |
| 281 |
local doc_tree = docs.for_schema(config_schema) |
| 282 |
|
| 283 |
-- Render as markdown |
| 284 |
local markdown_lines = docs.render_markdown(doc_tree.schema_doc) |
| 285 |
for _, line in ipairs(markdown_lines) do |
| 286 |
print(line) |
| 287 |
end |
| 288 |
``` |
| 289 |
|
| 290 |
## Error Reporting |
| 291 |
|
| 292 |
### Structured Errors |
| 293 |
|
| 294 |
Errors are represented as trees: |
| 295 |
|
| 296 |
```lua |
| 297 |
{ |
| 298 |
kind = "table_invalid", |
| 299 |
path = "config", |
| 300 |
details = { |
| 301 |
errors = { |
| 302 |
port = { |
| 303 |
kind = "constraint_violation", |
| 304 |
path = "config.port", |
| 305 |
details = { constraint = "max", expected = 65535, got = 99999 } |
| 306 |
} |
| 307 |
} |
| 308 |
} |
| 309 |
} |
| 310 |
``` |
| 311 |
|
| 312 |
### Human-Readable Formatting |
| 313 |
|
| 314 |
```lua |
| 315 |
local T = require "lua_shape.core" |
| 316 |
print(T.format_error(error_tree)) |
| 317 |
``` |
| 318 |
|
| 319 |
Output: |
| 320 |
``` |
| 321 |
table validation failed at config: |
| 322 |
constraint violation at config.port: max (expected: 65535, got: 99999) |
| 323 |
``` |
| 324 |
|
| 325 |
### one_of Intersection Errors |
| 326 |
|
| 327 |
When all variants of a one_of fail, the error shows common requirements: |
| 328 |
|
| 329 |
``` |
| 330 |
value does not match any alternative at : |
| 331 |
all alternatives require: |
| 332 |
- name: string |
| 333 |
- type: string |
| 334 |
some alternatives also expect: |
| 335 |
- path: string (in file_config variant) |
| 336 |
- host: string (in redis_config variant) |
| 337 |
tried alternatives: |
| 338 |
- file_config: ... |
| 339 |
- redis_config: ... |
| 340 |
``` |
| 341 |
|
| 342 |
## API Reference |
| 343 |
|
| 344 |
### Core Module (`lua_shape.core`) |
| 345 |
|
| 346 |
#### Type Constructors |
| 347 |
|
| 348 |
- `T.string(opts?)` - String type |
| 349 |
- opts: `min_len`, `max_len`, `pattern`, `lpeg`, `doc` |
| 350 |
- `T.number(opts?)` - Number type |
| 351 |
- opts: `min`, `max`, `doc` |
| 352 |
- `T.integer(opts?)` - Integer type (number with integer=true) |
| 353 |
- opts: `min`, `max`, `doc` |
| 354 |
- `T.boolean(opts?)` - Boolean type |
| 355 |
- `T.enum(values, opts?)` - Enum type |
| 356 |
- `T.literal(value, opts?)` - Literal value type |
| 357 |
- `T.array(item_schema, opts?)` - Array type |
| 358 |
- opts: `min_items`, `max_items`, `doc` |
| 359 |
- `T.table(fields, opts?)` - Table type |
| 360 |
- opts: `open`, `extra`, `mixins`, `doc` |
| 361 |
- `T.one_of(variants, opts?)` - Sum type |
| 362 |
- `T.optional(schema, opts?)` - Optional wrapper |
| 363 |
- `T.default(schema, value)` - Default value wrapper |
| 364 |
- `T.transform(accepted_type, transformer, opts?)` - Transform wrapper (validates input against accepted_type, then applies transformer) |
| 365 |
- `T.ref(id, opts?)` - Schema reference placeholder (must be resolved via the registry before validation) |
| 366 |
- `T.mixin(schema, opts?)` - Mixin definition |
| 367 |
|
| 368 |
#### Schema Methods |
| 369 |
|
| 370 |
- `schema:check(value, ctx?)` - Validate value |
| 371 |
- `schema:transform(value, ctx?)` - Transform and validate (tableshape-compatible `(result)` / `(nil, err)` contract) |
| 372 |
- `schema:optional(opts?)` - Make optional |
| 373 |
- `schema:with_default(value)` - Add default |
| 374 |
- `schema:doc(doc_table)` - Add documentation |
| 375 |
- `schema:transform_with(fn, opts?)` - Add transformation |
| 376 |
|
| 377 |
### Registry Module (`lua_shape.registry`) |
| 378 |
|
| 379 |
- `Registry.global()` - Get/create global registry |
| 380 |
- `registry:define(id, schema)` - Register schema with ID (returns the resolved version with mixins/reference chains applied) |
| 381 |
- `registry:get(id)` - Get schema by ID |
| 382 |
- `registry:resolve_schema(schema)` - Resolve references and mixins (recurses into nested arrays/one_of/options and caches the result) |
| 383 |
- `registry:list()` - List all schema IDs |
| 384 |
- `registry:export_all()` - Export all schemas |
| 385 |
|
| 386 |
### Core Utilities |
| 387 |
|
| 388 |
The core module also includes utility functions: |
| 389 |
|
| 390 |
- `T.format_error(err)` - Format error tree as human-readable string |
| 391 |
- `T.deep_clone(value)` - Deep clone value for immutable transformations |
| 392 |
|
| 393 |
### JSON Schema Module (`lua_shape.jsonschema`) |
| 394 |
|
| 395 |
- `jsonschema.from_schema(schema, opts?)` - Convert to JSON Schema |
| 396 |
- `jsonschema.export_registry(registry, opts?)` - Export all schemas |
| 397 |
|
| 398 |
### Docs Module (`lua_shape.docs`) |
| 399 |
|
| 400 |
- `docs.for_schema(schema, opts?)` - Generate documentation IR |
| 401 |
- `docs.for_registry(registry, opts?)` - Generate docs for all schemas |
| 402 |
- `docs.render_markdown(doc_tree, indent?)` - Render as markdown |
| 403 |
|
| 404 |
## Migration from tableshape |
| 405 |
|
| 406 |
See [MIGRATION.md](MIGRATION.md) for detailed migration guide. |
| 407 |
|
| 408 |
Quick reference: |
| 409 |
|
| 410 |
| tableshape | lua_shape | |
| 411 |
|------------|---------------| |
| 412 |
| `ts.string` | `T.string()` | |
| 413 |
| `ts.number` | `T.number()` | |
| 414 |
| `ts.array_of(ts.string)` | `T.array(T.string())` | |
| 415 |
| `ts.shape({...})` | `T.table({...})` | |
| 416 |
| `field:is_optional()` | `field:optional()` or `{ schema = ..., optional = true }` | |
| 417 |
| `ts.string + ts.number` | `T.one_of({ T.string(), T.number() })` | |
| 418 |
| `ts.string / fn` | `T.string():transform_with(fn)` or `T.transform(T.string(), fn)` | |
| 419 |
| `field:describe("...")` | `field:doc({ summary = "..." })` | |
| 420 |
|
| 421 |
## Files |
| 422 |
|
| 423 |
- `core.lua` - Core type system, validation, and utilities |
| 424 |
- `registry.lua` - Schema registration and reference resolution |
| 425 |
- `jsonschema.lua` - JSON Schema export |
| 426 |
- `docs.lua` - Documentation generation |
| 427 |
- `MIGRATION.md` - Migration guide from tableshape |
| 428 |
- `README.md` - This file |
| 429 |
|
| 430 |
## License |
| 431 |
|
| 432 |
Apache License 2.0 - Same as Rspamd |