| 1 |
# Logs SDK |
| 2 |
|
| 3 |
## Abstract |
| 4 |
|
| 5 |
`go.opentelemetry.io/otel/sdk/log` provides Logs SDK compliant with the |
| 6 |
[specification](https://opentelemetry.io/docs/specs/otel/logs/sdk/). |
| 7 |
|
| 8 |
The prototype was created in |
| 9 |
[#4955](https://github.com/open-telemetry/opentelemetry-go/pull/4955). |
| 10 |
|
| 11 |
## Background |
| 12 |
|
| 13 |
The goal is to design the exported API of the SDK would have low performance |
| 14 |
overhead. Most importantly, have a design that reduces the amount of heap |
| 15 |
allocations and even make it possible to have a zero-allocation implementation. |
| 16 |
Eliminating the amount of heap allocations reduces the GC pressure which can |
| 17 |
produce some of the largest improvements in performance.[^1] |
| 18 |
|
| 19 |
The main and recommended use case is to configure the SDK to use an OTLP |
| 20 |
exporter with a batch processor.[^2] Therefore, the implementation aims to be |
| 21 |
high-performant in this scenario. Some users that require high throughput may |
| 22 |
also want to use e.g. an [user_events](https://docs.kernel.org/trace/user_events.html), |
| 23 |
[LLTng](https://lttng.org/docs/v2.13/#doc-tracing-your-own-user-application) |
| 24 |
or [ETW](https://learn.microsoft.com/en-us/windows/win32/etw/about-event-tracing) |
| 25 |
exporter with a simple processor. Users may also want to use |
| 26 |
[OTLP File](https://opentelemetry.io/docs/specs/otel/protocol/file-exporter/) |
| 27 |
or [Standard Output](https://opentelemetry.io/docs/specs/otel/logs/sdk_exporters/stdout/) |
| 28 |
exporter in order to emit logs to standard output/error or files. |
| 29 |
|
| 30 |
## Modules structure |
| 31 |
|
| 32 |
The SDK is published as a single `go.opentelemetry.io/otel/sdk/log` Go module. |
| 33 |
|
| 34 |
The exporters are going to be published as following Go modules: |
| 35 |
|
| 36 |
- `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc` |
| 37 |
- `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp` |
| 38 |
- `go.opentelemetry.io/otel/exporters/stdout/stdoutlog` |
| 39 |
|
| 40 |
## LoggerProvider |
| 41 |
|
| 42 |
The [LoggerProvider](https://opentelemetry.io/docs/specs/otel/logs/sdk/#loggerprovider) |
| 43 |
is implemented as `LoggerProvider` struct in [provider.go](provider.go). |
| 44 |
|
| 45 |
## LogRecord limits |
| 46 |
|
| 47 |
The [LogRecord limits](https://opentelemetry.io/docs/specs/otel/logs/sdk/#logrecord-limits) |
| 48 |
can be configured using following options: |
| 49 |
|
| 50 |
```go |
| 51 |
func WithAttributeCountLimit(limit int) LoggerProviderOption |
| 52 |
func WithAttributeValueLengthLimit(limit int) LoggerProviderOption |
| 53 |
``` |
| 54 |
|
| 55 |
The limits can be also configured using the `OTEL_LOGRECORD_*` environment variables as |
| 56 |
[defined by the specification](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#logrecord-limits). |
| 57 |
|
| 58 |
### Processor |
| 59 |
|
| 60 |
The [LogRecordProcessor](https://opentelemetry.io/docs/specs/otel/logs/sdk/#logrecordprocessor) |
| 61 |
is defined as `Processor` interface in [processor.go](processor.go). |
| 62 |
|
| 63 |
The user set processors for the `LoggerProvider` using |
| 64 |
`func WithProcessor(processor Processor) LoggerProviderOption`. |
| 65 |
|
| 66 |
The user can configure custom processors and decorate built-in processors. |
| 67 |
|
| 68 |
The specification may add new operations to the |
| 69 |
[LogRecordProcessor](https://opentelemetry.io/docs/specs/otel/logs/sdk/#logrecordprocessor). |
| 70 |
If it happens, [CONTRIBUTING.md](../../CONTRIBUTING.md#how-to-change-other-interfaces) |
| 71 |
describes how the SDK can be extended in a backwards-compatible way. |
| 72 |
|
| 73 |
### SimpleProcessor |
| 74 |
|
| 75 |
The [Simple processor](https://opentelemetry.io/docs/specs/otel/logs/sdk/#simple-processor) |
| 76 |
is implemented as `SimpleProcessor` struct in [simple.go](simple.go). |
| 77 |
|
| 78 |
### BatchProcessor |
| 79 |
|
| 80 |
The [Batching processor](https://opentelemetry.io/docs/specs/otel/logs/sdk/#batching-processor) |
| 81 |
is implemented as `BatchProcessor` struct in [batch.go](batch.go). |
| 82 |
|
| 83 |
The `Batcher` can be also configured using the `OTEL_BLRP_*` environment variables as |
| 84 |
[defined by the specification](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#batch-logrecord-processor). |
| 85 |
|
| 86 |
### Exporter |
| 87 |
|
| 88 |
The [LogRecordExporter](https://opentelemetry.io/docs/specs/otel/logs/sdk/#logrecordexporter) |
| 89 |
is defined as `Exporter` interface in [exporter.go](exporter.go). |
| 90 |
|
| 91 |
The slice passed to `Export` must not be retained by the implementation |
| 92 |
(like e.g. [`io.Writer`](https://pkg.go.dev/io#Writer)) |
| 93 |
so that the caller can reuse the passed slice |
| 94 |
(e.g. using [`sync.Pool`](https://pkg.go.dev/sync#Pool)) |
| 95 |
to avoid heap allocations on each call. |
| 96 |
|
| 97 |
The specification may add new operations to the |
| 98 |
[LogRecordExporter](https://opentelemetry.io/docs/specs/otel/logs/sdk/#logrecordexporter). |
| 99 |
If it happens, [CONTRIBUTING.md](../../CONTRIBUTING.md#how-to-change-other-interfaces) |
| 100 |
describes how the SDK can be extended in a backwards-compatible way. |
| 101 |
|
| 102 |
### Record |
| 103 |
|
| 104 |
The [ReadWriteLogRecord](https://opentelemetry.io/docs/specs/otel/logs/sdk/#readwritelogrecord) |
| 105 |
is defined as `Record` struct in [record.go](record.go). |
| 106 |
|
| 107 |
The `Record` is designed similarly to [`log.Record`](https://pkg.go.dev/go.opentelemetry.io/otel/log#Record) |
| 108 |
in order to reduce the number of heap allocations when processing attributes. |
| 109 |
|
| 110 |
The SDK does not have have an additional definition of |
| 111 |
[ReadableLogRecord](https://opentelemetry.io/docs/specs/otel/logs/sdk/#readablelogrecord) |
| 112 |
as the specification does not say that the exporters must not be able to modify |
| 113 |
the log records. It simply requires them to be able to read the log records. |
| 114 |
Having less abstractions reduces the API surface and makes the design simpler. |
| 115 |
|
| 116 |
## Benchmarking |
| 117 |
|
| 118 |
The benchmarks are supposed to test end-to-end scenarios |
| 119 |
and avoid I/O that could affect the stability of the results. |
| 120 |
|
| 121 |
The benchmark results can be found in [the prototype](https://github.com/open-telemetry/opentelemetry-go/pull/4955). |
| 122 |
|
| 123 |
## Rejected alternatives |
| 124 |
|
| 125 |
### Represent both LogRecordProcessor and LogRecordExporter as Exporter |
| 126 |
|
| 127 |
Because the [LogRecordProcessor](https://opentelemetry.io/docs/specs/otel/logs/sdk/#logrecordprocessor) |
| 128 |
and the [LogRecordProcessor](https://opentelemetry.io/docs/specs/otel/logs/sdk/#logrecordexporter) |
| 129 |
abstractions are so similar, there was a proposal to unify them under |
| 130 |
single `Exporter` interface.[^3] |
| 131 |
|
| 132 |
However, introducing a `Processor` interface makes it easier |
| 133 |
to create custom processor decorators[^4] |
| 134 |
and makes the design more aligned with the specification. |
| 135 |
|
| 136 |
### Embed log.Record |
| 137 |
|
| 138 |
Because [`Record`](#record) and [`log.Record`](https://pkg.go.dev/go.opentelemetry.io/otel/log#Record) |
| 139 |
are very similar, there was a proposal to embed `log.Record` in `Record` definition. |
| 140 |
|
| 141 |
[`log.Record`](https://pkg.go.dev/go.opentelemetry.io/otel/log#Record) |
| 142 |
supports only adding attributes. |
| 143 |
In the SDK, we also need to be able to modify the attributes (e.g. removal) |
| 144 |
provided via API. |
| 145 |
|
| 146 |
Moreover it is safer to have these abstraction decoupled. |
| 147 |
E.g. there can be a need for some fields that can be set via API and cannot be modified by the processors. |
| 148 |
|
| 149 |
### Processor.OnEmit to accept Record values |
| 150 |
|
| 151 |
There was a proposal to make the [Processor](#processor)'s `OnEmit` |
| 152 |
to accept a [Record](#record) value instead of a pointer to reduce allocations |
| 153 |
as well as to have design similar to [`slog.Handler`](https://pkg.go.dev/log/slog#Handler). |
| 154 |
|
| 155 |
There have been long discussions within the OpenTelemetry Specification SIG[^5] |
| 156 |
about whether such a design would comply with the specification. The summary |
| 157 |
was that the current processor design flaws are present in other languages as |
| 158 |
well. Therefore, it would be favorable to introduce new processing concepts |
| 159 |
(e.g. chaining processors) in the specification that would coexist with the |
| 160 |
current "mutable" processor design. |
| 161 |
|
| 162 |
The performance disadvantages caused by using a pointer (which at the time of |
| 163 |
writing causes an additional heap allocation) may be mitigated by future |
| 164 |
versions of the Go compiler, thanks to improved escape analysis and |
| 165 |
profile-guided optimization (PGO)[^6]. |
| 166 |
|
| 167 |
On the other hand, [Processor](#processor)'s `Enabled` is fine to accept |
| 168 |
a [Record](#record) value as the processors should not mutate the passed |
| 169 |
parameters. |
| 170 |
|
| 171 |
[^1]: [A Guide to the Go Garbage Collector](https://tip.golang.org/doc/gc-guide) |
| 172 |
[^2]: [OpenTelemetry Logging](https://opentelemetry.io/docs/specs/otel/logs) |
| 173 |
[^3]: [Conversation on representing LogRecordProcessor and LogRecordExporter via a single Exporter interface](https://github.com/open-telemetry/opentelemetry-go/pull/4954#discussion_r1515050480) |
| 174 |
[^4]: [Introduce Processor](https://github.com/pellared/opentelemetry-go/pull/9) |
| 175 |
[^5]: [Log record mutations do not have to be visible in next registered processors](https://github.com/open-telemetry/opentelemetry-specification/pull/4067) |
| 176 |
[^6]: [Profile-guided optimization](https://go.dev/doc/pgo) |