Debian Codegraph @

Documentation/gpu/rfc/color_pipeline.rst

raw
1 .. SPDX-License-Identifier: GPL-2.0
2
3 ========================
4 Linux Color Pipeline API
5 ========================
6
7 What problem are we solving?
8 ============================
9
10 We would like to support pre-, and post-blending complex color
11 transformations in display controller hardware in order to allow for
12 HW-supported HDR use-cases, as well as to provide support to
13 color-managed applications, such as video or image editors.
14
15 It is possible to support an HDR output on HW supporting the Colorspace
16 and HDR Metadata drm_connector properties, but that requires the
17 compositor or application to render and compose the content into one
18 final buffer intended for display. Doing so is costly.
19
20 Most modern display HW offers various 1D LUTs, 3D LUTs, matrices, and other
21 operations to support color transformations. These operations are often
22 implemented in fixed-function HW and therefore much more power efficient than
23 performing similar operations via shaders or CPU.
24
25 We would like to make use of this HW functionality to support complex color
26 transformations with no, or minimal CPU or shader load. The switch between HW
27 fixed-function blocks and shaders/CPU must be seamless with no visible
28 difference when fallback to shaders/CPU is neceesary at any time.
29
30
31 How are other OSes solving this problem?
32 ========================================
33
34 The most widely supported use-cases regard HDR content, whether video or
35 gaming.
36
37 Most OSes will specify the source content format (color gamut, encoding transfer
38 function, and other metadata, such as max and average light levels) to a driver.
39 Drivers will then program their fixed-function HW accordingly to map from a
40 source content buffer's space to a display's space.
41
42 When fixed-function HW is not available the compositor will assemble a shader to
43 ask the GPU to perform the transformation from the source content format to the
44 display's format.
45
46 A compositor's mapping function and a driver's mapping function are usually
47 entirely separate concepts. On OSes where a HW vendor has no insight into
48 closed-source compositor code such a vendor will tune their color management
49 code to visually match the compositor's. On other OSes, where both mapping
50 functions are open to an implementer they will ensure both mappings match.
51
52 This results in mapping algorithm lock-in, meaning that no-one alone can
53 experiment with or introduce new mapping algorithms and achieve
54 consistent results regardless of which implementation path is taken.
55
56 Why is Linux different?
57 =======================
58
59 Unlike other OSes, where there is one compositor for one or more drivers, on
60 Linux we have a many-to-many relationship. Many compositors; many drivers.
61 In addition each compositor vendor or community has their own view of how
62 color management should be done. This is what makes Linux so beautiful.
63
64 This means that a HW vendor can now no longer tune their driver to one
65 compositor, as tuning it to one could make it look fairly different from
66 another compositor's color mapping.
67
68 We need a better solution.
69
70
71 Descriptive API
72 ===============
73
74 An API that describes the source and destination colorspaces is a descriptive
75 API. It describes the input and output color spaces but does not describe
76 how precisely they should be mapped. Such a mapping includes many minute
77 design decision that can greatly affect the look of the final result.
78
79 It is not feasible to describe such mapping with enough detail to ensure the
80 same result from each implementation. In fact, these mappings are a very active
81 research area.
82
83
84 Prescriptive API
85 ================
86
87 A prescriptive API describes not the source and destination colorspaces. It
88 instead prescribes a recipe for how to manipulate pixel values to arrive at the
89 desired outcome.
90
91 This recipe is generally an ordered list of straight-forward operations,
92 with clear mathematical definitions, such as 1D LUTs, 3D LUTs, matrices,
93 or other operations that can be described in a precise manner.
94
95
96 The Color Pipeline API
97 ======================
98
99 HW color management pipelines can significantly differ between HW
100 vendors in terms of availability, ordering, and capabilities of HW
101 blocks. This makes a common definition of color management blocks and
102 their ordering nigh impossible. Instead we are defining an API that
103 allows user space to discover the HW capabilities in a generic manner,
104 agnostic of specific drivers and hardware.
105
106
107 drm_colorop Object
108 ==================
109
110 To support the definition of color pipelines we define the DRM core
111 object type drm_colorop. Individual drm_colorop objects will be chained
112 via the NEXT property of a drm_colorop to constitute a color pipeline.
113 Each drm_colorop object is unique, i.e., even if multiple color
114 pipelines have the same operation they won't share the same drm_colorop
115 object to describe that operation.
116
117 Note that drivers are not expected to map drm_colorop objects statically
118 to specific HW blocks. The mapping of drm_colorop objects is entirely a
119 driver-internal detail and can be as dynamic or static as a driver needs
120 it to be. See more in the Driver Implementation Guide section below.
121
122 Each drm_colorop has three core properties:
123
124 TYPE: An enumeration property, defining the type of transformation, such as
125 * enumerated curve
126 * custom (uniform) 1D LUT
127 * 3x3 matrix
128 * 3x4 matrix
129 * 3D LUT
130 * etc.
131
132 Depending on the type of transformation other properties will describe
133 more details.
134
135 BYPASS: A boolean property that can be used to easily put a block into
136 bypass mode. The BYPASS property is not mandatory for a colorop, as long
137 as the entire pipeline can get bypassed by setting the COLOR_PIPELINE on
138 a plane to '0'.
139
140 NEXT: The ID of the next drm_colorop in a color pipeline, or 0 if this
141 drm_colorop is the last in the chain.
142
143 An example of a drm_colorop object might look like one of these::
144
145 /* 1D enumerated curve */
146 Color operation 42
147 ├─ "TYPE": immutable enum {1D enumerated curve, 1D LUT, 3x3 matrix, 3x4 matrix, 3D LUT, etc.} = 1D enumerated curve
148 ├─ "BYPASS": bool {true, false}
149 ├─ "CURVE_1D_TYPE": enum {sRGB EOTF, sRGB inverse EOTF, PQ EOTF, PQ inverse EOTF, …}
150 └─ "NEXT": immutable color operation ID = 43
151
152 /* custom 4k entry 1D LUT */
153 Color operation 52
154 ├─ "TYPE": immutable enum {1D enumerated curve, 1D LUT, 3x3 matrix, 3x4 matrix, 3D LUT, etc.} = 1D LUT
155 ├─ "BYPASS": bool {true, false}
156 ├─ "SIZE": immutable range = 4096
157 ├─ "DATA": blob
158 └─ "NEXT": immutable color operation ID = 0
159
160 /* 17^3 3D LUT */
161 Color operation 72
162 ├─ "TYPE": immutable enum {1D enumerated curve, 1D LUT, 3x3 matrix, 3x4 matrix, 3D LUT, etc.} = 3D LUT
163 ├─ "BYPASS": bool {true, false}
164 ├─ "SIZE": immutable range = 17
165 ├─ "DATA": blob
166 └─ "NEXT": immutable color operation ID = 73
167
168 drm_colorop extensibility
169 -------------------------
170
171 Unlike existing DRM core objects, like &drm_plane, drm_colorop is not
172 extensible. This simplifies implementations and keeps all functionality
173 for managing &drm_colorop objects in the DRM core.
174
175 If there is a need one may introduce a simple &drm_colorop_funcs
176 function table in the future, for example to support an IN_FORMATS
177 property on a &drm_colorop.
178
179 If a driver requires the ability to create a driver-specific colorop
180 object they will need to add &drm_colorop func table support with
181 support for the usual functions, like destroy, atomic_duplicate_state,
182 and atomic_destroy_state.
183
184
185 COLOR_PIPELINE Plane Property
186 =============================
187
188 Color Pipelines are created by a driver and advertised via a new
189 COLOR_PIPELINE enum property on each plane. Values of the property
190 always include object id 0, which is the default and means all color
191 processing is disabled. Additional values will be the object IDs of the
192 first drm_colorop in a pipeline. A driver can create and advertise none,
193 one, or more possible color pipelines. A DRM client will select a color
194 pipeline by setting the COLOR PIPELINE to the respective value.
195
196 NOTE: Many DRM clients will set enumeration properties via the string
197 value, often hard-coding it. Since this enumeration is generated based
198 on the colorop object IDs it is important to perform the Color Pipeline
199 Discovery, described below, instead of hard-coding color pipeline
200 assignment. Drivers might generate the enum strings dynamically.
201 Hard-coded strings might only work for specific drivers on a specific
202 pieces of HW. Color Pipeline Discovery can work universally, as long as
203 drivers implement the required color operations.
204
205 The COLOR_PIPELINE property is only exposed when the
206 DRM_CLIENT_CAP_PLANE_COLOR_PIPELINE is set. Drivers shall ignore any
207 existing pre-blend color operations when this cap is set, such as
208 COLOR_RANGE and COLOR_ENCODING. If drivers want to support COLOR_RANGE
209 or COLOR_ENCODING functionality when the color pipeline client cap is
210 set, they are expected to expose colorops in the pipeline to allow for
211 the appropriate color transformation.
212
213 Setting of the COLOR_PIPELINE plane property or drm_colorop properties
214 is only allowed for userspace that sets this client cap.
215
216 An example of a COLOR_PIPELINE property on a plane might look like this::
217
218 Plane 10
219 ├─ "TYPE": immutable enum {Overlay, Primary, Cursor} = Primary
220 ├─ …
221 └─ "COLOR_PIPELINE": enum {0, 42, 52} = 0
222
223
224 Color Pipeline Discovery
225 ========================
226
227 A DRM client wanting color management on a drm_plane will:
228
229 1. Get the COLOR_PIPELINE property of the plane
230 2. iterate all COLOR_PIPELINE enum values
231 3. for each enum value walk the color pipeline (via the NEXT pointers)
232 and see if the available color operations are suitable for the
233 desired color management operations
234
235 If userspace encounters an unknown or unsuitable color operation during
236 discovery it does not need to reject the entire color pipeline outright,
237 as long as the unknown or unsuitable colorop has a "BYPASS" property.
238 Drivers will ensure that a bypassed block does not have any effect.
239
240 An example of chained properties to define an AMD pre-blending color
241 pipeline might look like this::
242
243 Plane 10
244 ├─ "TYPE" (immutable) = Primary
245 └─ "COLOR_PIPELINE": enum {0, 44} = 0
246
247 Color operation 44
248 ├─ "TYPE" (immutable) = 1D enumerated curve
249 ├─ "BYPASS": bool
250 ├─ "CURVE_1D_TYPE": enum {sRGB EOTF, PQ EOTF} = sRGB EOTF
251 └─ "NEXT" (immutable) = 45
252
253 Color operation 45
254 ├─ "TYPE" (immutable) = 3x4 Matrix
255 ├─ "BYPASS": bool
256 ├─ "DATA": blob
257 └─ "NEXT" (immutable) = 46
258
259 Color operation 46
260 ├─ "TYPE" (immutable) = 1D enumerated curve
261 ├─ "BYPASS": bool
262 ├─ "CURVE_1D_TYPE": enum {sRGB Inverse EOTF, PQ Inverse EOTF} = sRGB EOTF
263 └─ "NEXT" (immutable) = 47
264
265 Color operation 47
266 ├─ "TYPE" (immutable) = 1D LUT
267 ├─ "SIZE": immutable range = 4096
268 ├─ "DATA": blob
269 └─ "NEXT" (immutable) = 48
270
271 Color operation 48
272 ├─ "TYPE" (immutable) = 3D LUT
273 ├─ "DATA": blob
274 └─ "NEXT" (immutable) = 49
275
276 Color operation 49
277 ├─ "TYPE" (immutable) = 1D enumerated curve
278 ├─ "BYPASS": bool
279 ├─ "CURVE_1D_TYPE": enum {sRGB EOTF, PQ EOTF} = sRGB EOTF
280 └─ "NEXT" (immutable) = 0
281
282
283 Color Pipeline Programming
284 ==========================
285
286 Once a DRM client has found a suitable pipeline it will:
287
288 1. Set the COLOR_PIPELINE enum value to the one pointing at the first
289 drm_colorop object of the desired pipeline
290 2. Set the properties for all drm_colorop objects in the pipeline to the
291 desired values, setting BYPASS to true for unused drm_colorop blocks,
292 and false for enabled drm_colorop blocks
293 3. Perform (TEST_ONLY or not) atomic commit with all the other KMS
294 states it wishes to change
295
296 To configure the pipeline for an HDR10 PQ plane and blending in linear
297 space, a compositor might perform an atomic commit with the following
298 property values::
299
300 Plane 10
301 └─ "COLOR_PIPELINE" = 42
302
303 Color operation 42
304 └─ "BYPASS" = true
305
306 Color operation 44
307 └─ "BYPASS" = true
308
309 Color operation 45
310 └─ "BYPASS" = true
311
312 Color operation 46
313 └─ "BYPASS" = true
314
315 Color operation 47
316 ├─ "DATA" = Gamut mapping + tone mapping + night mode
317 └─ "BYPASS" = false
318
319 Color operation 48
320 ├─ "CURVE_1D_TYPE" = PQ EOTF
321 └─ "BYPASS" = false
322
323
324 Driver Implementer's Guide
325 ==========================
326
327 What does this all mean for driver implementations? As noted above the
328 colorops can map to HW directly but don't need to do so. Here are some
329 suggestions on how to think about creating your color pipelines:
330
331 - Try to expose pipelines that use already defined colorops, even if
332 your hardware pipeline is split differently. This allows existing
333 userspace to immediately take advantage of the hardware.
334
335 - Additionally, try to expose your actual hardware blocks as colorops.
336 Define new colorop types where you believe it can offer significant
337 benefits if userspace learns to program them.
338
339 - Avoid defining new colorops for compound operations with very narrow
340 scope. If you have a hardware block for a special operation that
341 cannot be split further, you can expose that as a new colorop type.
342 However, try to not define colorops for "use cases", especially if
343 they require you to combine multiple hardware blocks.
344
345 - Design new colorops as prescriptive, not descriptive; by the
346 mathematical formula, not by the assumed input and output.
347
348 A defined colorop type must be deterministic. The exact behavior of the
349 colorop must be documented entirely, whether via a mathematical formula
350 or some other description. Its operation can depend only on its
351 properties and input and nothing else, allowed error tolerance
352 notwithstanding.
353
354
355 Driver Forward/Backward Compatibility
356 =====================================
357
358 As this is uAPI drivers can't regress color pipelines that have been
359 introduced for a given HW generation. New HW generations are free to
360 abandon color pipelines advertised for previous generations.
361 Nevertheless, it can be beneficial to carry support for existing color
362 pipelines forward as those will likely already have support in DRM
363 clients.
364
365 Introducing new colorops to a pipeline is fine, as long as they can be
366 bypassed or are purely informational. DRM clients implementing support
367 for the pipeline can always skip unknown properties as long as they can
368 be confident that doing so will not cause unexpected results.
369
370 If a new colorop doesn't fall into one of the above categories
371 (bypassable or informational) the modified pipeline would be unusable
372 for user space. In this case a new pipeline should be defined.
373
374
375 References
376 ==========
377
378 1. https://lore.kernel.org/dri-devel/QMers3awXvNCQlyhWdTtsPwkp5ie9bze_hD5nAccFW7a_RXlWjYB7MoUW_8CKLT2bSQwIXVi5H6VULYIxCdgvryZoAoJnC5lZgyK1QWn488=@emersion.fr/