Debian Codegraph @

sdk/log/DESIGN.md

raw
1 # Logs SDK
2
3 ## Abstract
4
5 `go.opentelemetry.io/otel/sdk/log` provides Logs SDK compliant with the
6 [specification](https://opentelemetry.io/docs/specs/otel/logs/sdk/).
7
8 The prototype was created in
9 [#4955](https://github.com/open-telemetry/opentelemetry-go/pull/4955).
10
11 ## Background
12
13 The goal is to design the exported API of the SDK would have low performance
14 overhead. Most importantly, have a design that reduces the amount of heap
15 allocations and even make it possible to have a zero-allocation implementation.
16 Eliminating the amount of heap allocations reduces the GC pressure which can
17 produce some of the largest improvements in performance.[^1]
18
19 The main and recommended use case is to configure the SDK to use an OTLP
20 exporter with a batch processor.[^2] Therefore, the implementation aims to be
21 high-performant in this scenario. Some users that require high throughput may
22 also want to use e.g. an [user_events](https://docs.kernel.org/trace/user_events.html),
23 [LLTng](https://lttng.org/docs/v2.13/#doc-tracing-your-own-user-application)
24 or [ETW](https://learn.microsoft.com/en-us/windows/win32/etw/about-event-tracing)
25 exporter with a simple processor. Users may also want to use
26 [OTLP File](https://opentelemetry.io/docs/specs/otel/protocol/file-exporter/)
27 or [Standard Output](https://opentelemetry.io/docs/specs/otel/logs/sdk_exporters/stdout/)
28 exporter in order to emit logs to standard output/error or files.
29
30 ## Modules structure
31
32 The SDK is published as a single `go.opentelemetry.io/otel/sdk/log` Go module.
33
34 The exporters are going to be published as following Go modules:
35
36 - `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc`
37 - `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp`
38 - `go.opentelemetry.io/otel/exporters/stdout/stdoutlog`
39
40 ## LoggerProvider
41
42 The [LoggerProvider](https://opentelemetry.io/docs/specs/otel/logs/sdk/#loggerprovider)
43 is implemented as `LoggerProvider` struct in [provider.go](provider.go).
44
45 ## LogRecord limits
46
47 The [LogRecord limits](https://opentelemetry.io/docs/specs/otel/logs/sdk/#logrecord-limits)
48 can be configured using following options:
49
50 ```go
51 func WithAttributeCountLimit(limit int) LoggerProviderOption
52 func WithAttributeValueLengthLimit(limit int) LoggerProviderOption
53 ```
54
55 The limits can be also configured using the `OTEL_LOGRECORD_*` environment variables as
56 [defined by the specification](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#logrecord-limits).
57
58 ### Processor
59
60 The [LogRecordProcessor](https://opentelemetry.io/docs/specs/otel/logs/sdk/#logrecordprocessor)
61 is defined as `Processor` interface in [processor.go](processor.go).
62
63 The user set processors for the `LoggerProvider` using
64 `func WithProcessor(processor Processor) LoggerProviderOption`.
65
66 The user can configure custom processors and decorate built-in processors.
67
68 The specification may add new operations to the
69 [LogRecordProcessor](https://opentelemetry.io/docs/specs/otel/logs/sdk/#logrecordprocessor).
70 If it happens, [CONTRIBUTING.md](../../CONTRIBUTING.md#how-to-change-other-interfaces)
71 describes how the SDK can be extended in a backwards-compatible way.
72
73 ### SimpleProcessor
74
75 The [Simple processor](https://opentelemetry.io/docs/specs/otel/logs/sdk/#simple-processor)
76 is implemented as `SimpleProcessor` struct in [simple.go](simple.go).
77
78 ### BatchProcessor
79
80 The [Batching processor](https://opentelemetry.io/docs/specs/otel/logs/sdk/#batching-processor)
81 is implemented as `BatchProcessor` struct in [batch.go](batch.go).
82
83 The `Batcher` can be also configured using the `OTEL_BLRP_*` environment variables as
84 [defined by the specification](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#batch-logrecord-processor).
85
86 ### Exporter
87
88 The [LogRecordExporter](https://opentelemetry.io/docs/specs/otel/logs/sdk/#logrecordexporter)
89 is defined as `Exporter` interface in [exporter.go](exporter.go).
90
91 The slice passed to `Export` must not be retained by the implementation
92 (like e.g. [`io.Writer`](https://pkg.go.dev/io#Writer))
93 so that the caller can reuse the passed slice
94 (e.g. using [`sync.Pool`](https://pkg.go.dev/sync#Pool))
95 to avoid heap allocations on each call.
96
97 The specification may add new operations to the
98 [LogRecordExporter](https://opentelemetry.io/docs/specs/otel/logs/sdk/#logrecordexporter).
99 If it happens, [CONTRIBUTING.md](../../CONTRIBUTING.md#how-to-change-other-interfaces)
100 describes how the SDK can be extended in a backwards-compatible way.
101
102 ### Record
103
104 The [ReadWriteLogRecord](https://opentelemetry.io/docs/specs/otel/logs/sdk/#readwritelogrecord)
105 is defined as `Record` struct in [record.go](record.go).
106
107 The `Record` is designed similarly to [`log.Record`](https://pkg.go.dev/go.opentelemetry.io/otel/log#Record)
108 in order to reduce the number of heap allocations when processing attributes.
109
110 The SDK does not have have an additional definition of
111 [ReadableLogRecord](https://opentelemetry.io/docs/specs/otel/logs/sdk/#readablelogrecord)
112 as the specification does not say that the exporters must not be able to modify
113 the log records. It simply requires them to be able to read the log records.
114 Having less abstractions reduces the API surface and makes the design simpler.
115
116 ## Benchmarking
117
118 The benchmarks are supposed to test end-to-end scenarios
119 and avoid I/O that could affect the stability of the results.
120
121 The benchmark results can be found in [the prototype](https://github.com/open-telemetry/opentelemetry-go/pull/4955).
122
123 ## Rejected alternatives
124
125 ### Represent both LogRecordProcessor and LogRecordExporter as Exporter
126
127 Because the [LogRecordProcessor](https://opentelemetry.io/docs/specs/otel/logs/sdk/#logrecordprocessor)
128 and the [LogRecordProcessor](https://opentelemetry.io/docs/specs/otel/logs/sdk/#logrecordexporter)
129 abstractions are so similar, there was a proposal to unify them under
130 single `Exporter` interface.[^3]
131
132 However, introducing a `Processor` interface makes it easier
133 to create custom processor decorators[^4]
134 and makes the design more aligned with the specification.
135
136 ### Embed log.Record
137
138 Because [`Record`](#record) and [`log.Record`](https://pkg.go.dev/go.opentelemetry.io/otel/log#Record)
139 are very similar, there was a proposal to embed `log.Record` in `Record` definition.
140
141 [`log.Record`](https://pkg.go.dev/go.opentelemetry.io/otel/log#Record)
142 supports only adding attributes.
143 In the SDK, we also need to be able to modify the attributes (e.g. removal)
144 provided via API.
145
146 Moreover it is safer to have these abstraction decoupled.
147 E.g. there can be a need for some fields that can be set via API and cannot be modified by the processors.
148
149 ### Processor.OnEmit to accept Record values
150
151 There was a proposal to make the [Processor](#processor)'s `OnEmit`
152 to accept a [Record](#record) value instead of a pointer to reduce allocations
153 as well as to have design similar to [`slog.Handler`](https://pkg.go.dev/log/slog#Handler).
154
155 There have been long discussions within the OpenTelemetry Specification SIG[^5]
156 about whether such a design would comply with the specification. The summary
157 was that the current processor design flaws are present in other languages as
158 well. Therefore, it would be favorable to introduce new processing concepts
159 (e.g. chaining processors) in the specification that would coexist with the
160 current "mutable" processor design.
161
162 The performance disadvantages caused by using a pointer (which at the time of
163 writing causes an additional heap allocation) may be mitigated by future
164 versions of the Go compiler, thanks to improved escape analysis and
165 profile-guided optimization (PGO)[^6].
166
167 On the other hand, [Processor](#processor)'s `Enabled` is fine to accept
168 a [Record](#record) value as the processors should not mutate the passed
169 parameters.
170
171 [^1]: [A Guide to the Go Garbage Collector](https://tip.golang.org/doc/gc-guide)
172 [^2]: [OpenTelemetry Logging](https://opentelemetry.io/docs/specs/otel/logs)
173 [^3]: [Conversation on representing LogRecordProcessor and LogRecordExporter via a single Exporter interface](https://github.com/open-telemetry/opentelemetry-go/pull/4954#discussion_r1515050480)
174 [^4]: [Introduce Processor](https://github.com/pellared/opentelemetry-go/pull/9)
175 [^5]: [Log record mutations do not have to be visible in next registered processors](https://github.com/open-telemetry/opentelemetry-specification/pull/4067)
176 [^6]: [Profile-guided optimization](https://go.dev/doc/pgo)