| 1 |
.. SPDX-License-Identifier: GPL-2.0 |
| 2 |
|
| 3 |
======================== |
| 4 |
Linux Color Pipeline API |
| 5 |
======================== |
| 6 |
|
| 7 |
What problem are we solving? |
| 8 |
============================ |
| 9 |
|
| 10 |
We would like to support pre-, and post-blending complex color |
| 11 |
transformations in display controller hardware in order to allow for |
| 12 |
HW-supported HDR use-cases, as well as to provide support to |
| 13 |
color-managed applications, such as video or image editors. |
| 14 |
|
| 15 |
It is possible to support an HDR output on HW supporting the Colorspace |
| 16 |
and HDR Metadata drm_connector properties, but that requires the |
| 17 |
compositor or application to render and compose the content into one |
| 18 |
final buffer intended for display. Doing so is costly. |
| 19 |
|
| 20 |
Most modern display HW offers various 1D LUTs, 3D LUTs, matrices, and other |
| 21 |
operations to support color transformations. These operations are often |
| 22 |
implemented in fixed-function HW and therefore much more power efficient than |
| 23 |
performing similar operations via shaders or CPU. |
| 24 |
|
| 25 |
We would like to make use of this HW functionality to support complex color |
| 26 |
transformations with no, or minimal CPU or shader load. The switch between HW |
| 27 |
fixed-function blocks and shaders/CPU must be seamless with no visible |
| 28 |
difference when fallback to shaders/CPU is neceesary at any time. |
| 29 |
|
| 30 |
|
| 31 |
How are other OSes solving this problem? |
| 32 |
======================================== |
| 33 |
|
| 34 |
The most widely supported use-cases regard HDR content, whether video or |
| 35 |
gaming. |
| 36 |
|
| 37 |
Most OSes will specify the source content format (color gamut, encoding transfer |
| 38 |
function, and other metadata, such as max and average light levels) to a driver. |
| 39 |
Drivers will then program their fixed-function HW accordingly to map from a |
| 40 |
source content buffer's space to a display's space. |
| 41 |
|
| 42 |
When fixed-function HW is not available the compositor will assemble a shader to |
| 43 |
ask the GPU to perform the transformation from the source content format to the |
| 44 |
display's format. |
| 45 |
|
| 46 |
A compositor's mapping function and a driver's mapping function are usually |
| 47 |
entirely separate concepts. On OSes where a HW vendor has no insight into |
| 48 |
closed-source compositor code such a vendor will tune their color management |
| 49 |
code to visually match the compositor's. On other OSes, where both mapping |
| 50 |
functions are open to an implementer they will ensure both mappings match. |
| 51 |
|
| 52 |
This results in mapping algorithm lock-in, meaning that no-one alone can |
| 53 |
experiment with or introduce new mapping algorithms and achieve |
| 54 |
consistent results regardless of which implementation path is taken. |
| 55 |
|
| 56 |
Why is Linux different? |
| 57 |
======================= |
| 58 |
|
| 59 |
Unlike other OSes, where there is one compositor for one or more drivers, on |
| 60 |
Linux we have a many-to-many relationship. Many compositors; many drivers. |
| 61 |
In addition each compositor vendor or community has their own view of how |
| 62 |
color management should be done. This is what makes Linux so beautiful. |
| 63 |
|
| 64 |
This means that a HW vendor can now no longer tune their driver to one |
| 65 |
compositor, as tuning it to one could make it look fairly different from |
| 66 |
another compositor's color mapping. |
| 67 |
|
| 68 |
We need a better solution. |
| 69 |
|
| 70 |
|
| 71 |
Descriptive API |
| 72 |
=============== |
| 73 |
|
| 74 |
An API that describes the source and destination colorspaces is a descriptive |
| 75 |
API. It describes the input and output color spaces but does not describe |
| 76 |
how precisely they should be mapped. Such a mapping includes many minute |
| 77 |
design decision that can greatly affect the look of the final result. |
| 78 |
|
| 79 |
It is not feasible to describe such mapping with enough detail to ensure the |
| 80 |
same result from each implementation. In fact, these mappings are a very active |
| 81 |
research area. |
| 82 |
|
| 83 |
|
| 84 |
Prescriptive API |
| 85 |
================ |
| 86 |
|
| 87 |
A prescriptive API describes not the source and destination colorspaces. It |
| 88 |
instead prescribes a recipe for how to manipulate pixel values to arrive at the |
| 89 |
desired outcome. |
| 90 |
|
| 91 |
This recipe is generally an ordered list of straight-forward operations, |
| 92 |
with clear mathematical definitions, such as 1D LUTs, 3D LUTs, matrices, |
| 93 |
or other operations that can be described in a precise manner. |
| 94 |
|
| 95 |
|
| 96 |
The Color Pipeline API |
| 97 |
====================== |
| 98 |
|
| 99 |
HW color management pipelines can significantly differ between HW |
| 100 |
vendors in terms of availability, ordering, and capabilities of HW |
| 101 |
blocks. This makes a common definition of color management blocks and |
| 102 |
their ordering nigh impossible. Instead we are defining an API that |
| 103 |
allows user space to discover the HW capabilities in a generic manner, |
| 104 |
agnostic of specific drivers and hardware. |
| 105 |
|
| 106 |
|
| 107 |
drm_colorop Object |
| 108 |
================== |
| 109 |
|
| 110 |
To support the definition of color pipelines we define the DRM core |
| 111 |
object type drm_colorop. Individual drm_colorop objects will be chained |
| 112 |
via the NEXT property of a drm_colorop to constitute a color pipeline. |
| 113 |
Each drm_colorop object is unique, i.e., even if multiple color |
| 114 |
pipelines have the same operation they won't share the same drm_colorop |
| 115 |
object to describe that operation. |
| 116 |
|
| 117 |
Note that drivers are not expected to map drm_colorop objects statically |
| 118 |
to specific HW blocks. The mapping of drm_colorop objects is entirely a |
| 119 |
driver-internal detail and can be as dynamic or static as a driver needs |
| 120 |
it to be. See more in the Driver Implementation Guide section below. |
| 121 |
|
| 122 |
Each drm_colorop has three core properties: |
| 123 |
|
| 124 |
TYPE: An enumeration property, defining the type of transformation, such as |
| 125 |
* enumerated curve |
| 126 |
* custom (uniform) 1D LUT |
| 127 |
* 3x3 matrix |
| 128 |
* 3x4 matrix |
| 129 |
* 3D LUT |
| 130 |
* etc. |
| 131 |
|
| 132 |
Depending on the type of transformation other properties will describe |
| 133 |
more details. |
| 134 |
|
| 135 |
BYPASS: A boolean property that can be used to easily put a block into |
| 136 |
bypass mode. The BYPASS property is not mandatory for a colorop, as long |
| 137 |
as the entire pipeline can get bypassed by setting the COLOR_PIPELINE on |
| 138 |
a plane to '0'. |
| 139 |
|
| 140 |
NEXT: The ID of the next drm_colorop in a color pipeline, or 0 if this |
| 141 |
drm_colorop is the last in the chain. |
| 142 |
|
| 143 |
An example of a drm_colorop object might look like one of these:: |
| 144 |
|
| 145 |
/* 1D enumerated curve */ |
| 146 |
Color operation 42 |
| 147 |
├─ "TYPE": immutable enum {1D enumerated curve, 1D LUT, 3x3 matrix, 3x4 matrix, 3D LUT, etc.} = 1D enumerated curve |
| 148 |
├─ "BYPASS": bool {true, false} |
| 149 |
├─ "CURVE_1D_TYPE": enum {sRGB EOTF, sRGB inverse EOTF, PQ EOTF, PQ inverse EOTF, …} |
| 150 |
└─ "NEXT": immutable color operation ID = 43 |
| 151 |
|
| 152 |
/* custom 4k entry 1D LUT */ |
| 153 |
Color operation 52 |
| 154 |
├─ "TYPE": immutable enum {1D enumerated curve, 1D LUT, 3x3 matrix, 3x4 matrix, 3D LUT, etc.} = 1D LUT |
| 155 |
├─ "BYPASS": bool {true, false} |
| 156 |
├─ "SIZE": immutable range = 4096 |
| 157 |
├─ "DATA": blob |
| 158 |
└─ "NEXT": immutable color operation ID = 0 |
| 159 |
|
| 160 |
/* 17^3 3D LUT */ |
| 161 |
Color operation 72 |
| 162 |
├─ "TYPE": immutable enum {1D enumerated curve, 1D LUT, 3x3 matrix, 3x4 matrix, 3D LUT, etc.} = 3D LUT |
| 163 |
├─ "BYPASS": bool {true, false} |
| 164 |
├─ "SIZE": immutable range = 17 |
| 165 |
├─ "DATA": blob |
| 166 |
└─ "NEXT": immutable color operation ID = 73 |
| 167 |
|
| 168 |
drm_colorop extensibility |
| 169 |
------------------------- |
| 170 |
|
| 171 |
Unlike existing DRM core objects, like &drm_plane, drm_colorop is not |
| 172 |
extensible. This simplifies implementations and keeps all functionality |
| 173 |
for managing &drm_colorop objects in the DRM core. |
| 174 |
|
| 175 |
If there is a need one may introduce a simple &drm_colorop_funcs |
| 176 |
function table in the future, for example to support an IN_FORMATS |
| 177 |
property on a &drm_colorop. |
| 178 |
|
| 179 |
If a driver requires the ability to create a driver-specific colorop |
| 180 |
object they will need to add &drm_colorop func table support with |
| 181 |
support for the usual functions, like destroy, atomic_duplicate_state, |
| 182 |
and atomic_destroy_state. |
| 183 |
|
| 184 |
|
| 185 |
COLOR_PIPELINE Plane Property |
| 186 |
============================= |
| 187 |
|
| 188 |
Color Pipelines are created by a driver and advertised via a new |
| 189 |
COLOR_PIPELINE enum property on each plane. Values of the property |
| 190 |
always include object id 0, which is the default and means all color |
| 191 |
processing is disabled. Additional values will be the object IDs of the |
| 192 |
first drm_colorop in a pipeline. A driver can create and advertise none, |
| 193 |
one, or more possible color pipelines. A DRM client will select a color |
| 194 |
pipeline by setting the COLOR PIPELINE to the respective value. |
| 195 |
|
| 196 |
NOTE: Many DRM clients will set enumeration properties via the string |
| 197 |
value, often hard-coding it. Since this enumeration is generated based |
| 198 |
on the colorop object IDs it is important to perform the Color Pipeline |
| 199 |
Discovery, described below, instead of hard-coding color pipeline |
| 200 |
assignment. Drivers might generate the enum strings dynamically. |
| 201 |
Hard-coded strings might only work for specific drivers on a specific |
| 202 |
pieces of HW. Color Pipeline Discovery can work universally, as long as |
| 203 |
drivers implement the required color operations. |
| 204 |
|
| 205 |
The COLOR_PIPELINE property is only exposed when the |
| 206 |
DRM_CLIENT_CAP_PLANE_COLOR_PIPELINE is set. Drivers shall ignore any |
| 207 |
existing pre-blend color operations when this cap is set, such as |
| 208 |
COLOR_RANGE and COLOR_ENCODING. If drivers want to support COLOR_RANGE |
| 209 |
or COLOR_ENCODING functionality when the color pipeline client cap is |
| 210 |
set, they are expected to expose colorops in the pipeline to allow for |
| 211 |
the appropriate color transformation. |
| 212 |
|
| 213 |
Setting of the COLOR_PIPELINE plane property or drm_colorop properties |
| 214 |
is only allowed for userspace that sets this client cap. |
| 215 |
|
| 216 |
An example of a COLOR_PIPELINE property on a plane might look like this:: |
| 217 |
|
| 218 |
Plane 10 |
| 219 |
├─ "TYPE": immutable enum {Overlay, Primary, Cursor} = Primary |
| 220 |
├─ … |
| 221 |
└─ "COLOR_PIPELINE": enum {0, 42, 52} = 0 |
| 222 |
|
| 223 |
|
| 224 |
Color Pipeline Discovery |
| 225 |
======================== |
| 226 |
|
| 227 |
A DRM client wanting color management on a drm_plane will: |
| 228 |
|
| 229 |
1. Get the COLOR_PIPELINE property of the plane |
| 230 |
2. iterate all COLOR_PIPELINE enum values |
| 231 |
3. for each enum value walk the color pipeline (via the NEXT pointers) |
| 232 |
and see if the available color operations are suitable for the |
| 233 |
desired color management operations |
| 234 |
|
| 235 |
If userspace encounters an unknown or unsuitable color operation during |
| 236 |
discovery it does not need to reject the entire color pipeline outright, |
| 237 |
as long as the unknown or unsuitable colorop has a "BYPASS" property. |
| 238 |
Drivers will ensure that a bypassed block does not have any effect. |
| 239 |
|
| 240 |
An example of chained properties to define an AMD pre-blending color |
| 241 |
pipeline might look like this:: |
| 242 |
|
| 243 |
Plane 10 |
| 244 |
├─ "TYPE" (immutable) = Primary |
| 245 |
└─ "COLOR_PIPELINE": enum {0, 44} = 0 |
| 246 |
|
| 247 |
Color operation 44 |
| 248 |
├─ "TYPE" (immutable) = 1D enumerated curve |
| 249 |
├─ "BYPASS": bool |
| 250 |
├─ "CURVE_1D_TYPE": enum {sRGB EOTF, PQ EOTF} = sRGB EOTF |
| 251 |
└─ "NEXT" (immutable) = 45 |
| 252 |
|
| 253 |
Color operation 45 |
| 254 |
├─ "TYPE" (immutable) = 3x4 Matrix |
| 255 |
├─ "BYPASS": bool |
| 256 |
├─ "DATA": blob |
| 257 |
└─ "NEXT" (immutable) = 46 |
| 258 |
|
| 259 |
Color operation 46 |
| 260 |
├─ "TYPE" (immutable) = 1D enumerated curve |
| 261 |
├─ "BYPASS": bool |
| 262 |
├─ "CURVE_1D_TYPE": enum {sRGB Inverse EOTF, PQ Inverse EOTF} = sRGB EOTF |
| 263 |
└─ "NEXT" (immutable) = 47 |
| 264 |
|
| 265 |
Color operation 47 |
| 266 |
├─ "TYPE" (immutable) = 1D LUT |
| 267 |
├─ "SIZE": immutable range = 4096 |
| 268 |
├─ "DATA": blob |
| 269 |
└─ "NEXT" (immutable) = 48 |
| 270 |
|
| 271 |
Color operation 48 |
| 272 |
├─ "TYPE" (immutable) = 3D LUT |
| 273 |
├─ "DATA": blob |
| 274 |
└─ "NEXT" (immutable) = 49 |
| 275 |
|
| 276 |
Color operation 49 |
| 277 |
├─ "TYPE" (immutable) = 1D enumerated curve |
| 278 |
├─ "BYPASS": bool |
| 279 |
├─ "CURVE_1D_TYPE": enum {sRGB EOTF, PQ EOTF} = sRGB EOTF |
| 280 |
└─ "NEXT" (immutable) = 0 |
| 281 |
|
| 282 |
|
| 283 |
Color Pipeline Programming |
| 284 |
========================== |
| 285 |
|
| 286 |
Once a DRM client has found a suitable pipeline it will: |
| 287 |
|
| 288 |
1. Set the COLOR_PIPELINE enum value to the one pointing at the first |
| 289 |
drm_colorop object of the desired pipeline |
| 290 |
2. Set the properties for all drm_colorop objects in the pipeline to the |
| 291 |
desired values, setting BYPASS to true for unused drm_colorop blocks, |
| 292 |
and false for enabled drm_colorop blocks |
| 293 |
3. Perform (TEST_ONLY or not) atomic commit with all the other KMS |
| 294 |
states it wishes to change |
| 295 |
|
| 296 |
To configure the pipeline for an HDR10 PQ plane and blending in linear |
| 297 |
space, a compositor might perform an atomic commit with the following |
| 298 |
property values:: |
| 299 |
|
| 300 |
Plane 10 |
| 301 |
└─ "COLOR_PIPELINE" = 42 |
| 302 |
|
| 303 |
Color operation 42 |
| 304 |
└─ "BYPASS" = true |
| 305 |
|
| 306 |
Color operation 44 |
| 307 |
└─ "BYPASS" = true |
| 308 |
|
| 309 |
Color operation 45 |
| 310 |
└─ "BYPASS" = true |
| 311 |
|
| 312 |
Color operation 46 |
| 313 |
└─ "BYPASS" = true |
| 314 |
|
| 315 |
Color operation 47 |
| 316 |
├─ "DATA" = Gamut mapping + tone mapping + night mode |
| 317 |
└─ "BYPASS" = false |
| 318 |
|
| 319 |
Color operation 48 |
| 320 |
├─ "CURVE_1D_TYPE" = PQ EOTF |
| 321 |
└─ "BYPASS" = false |
| 322 |
|
| 323 |
|
| 324 |
Driver Implementer's Guide |
| 325 |
========================== |
| 326 |
|
| 327 |
What does this all mean for driver implementations? As noted above the |
| 328 |
colorops can map to HW directly but don't need to do so. Here are some |
| 329 |
suggestions on how to think about creating your color pipelines: |
| 330 |
|
| 331 |
- Try to expose pipelines that use already defined colorops, even if |
| 332 |
your hardware pipeline is split differently. This allows existing |
| 333 |
userspace to immediately take advantage of the hardware. |
| 334 |
|
| 335 |
- Additionally, try to expose your actual hardware blocks as colorops. |
| 336 |
Define new colorop types where you believe it can offer significant |
| 337 |
benefits if userspace learns to program them. |
| 338 |
|
| 339 |
- Avoid defining new colorops for compound operations with very narrow |
| 340 |
scope. If you have a hardware block for a special operation that |
| 341 |
cannot be split further, you can expose that as a new colorop type. |
| 342 |
However, try to not define colorops for "use cases", especially if |
| 343 |
they require you to combine multiple hardware blocks. |
| 344 |
|
| 345 |
- Design new colorops as prescriptive, not descriptive; by the |
| 346 |
mathematical formula, not by the assumed input and output. |
| 347 |
|
| 348 |
A defined colorop type must be deterministic. The exact behavior of the |
| 349 |
colorop must be documented entirely, whether via a mathematical formula |
| 350 |
or some other description. Its operation can depend only on its |
| 351 |
properties and input and nothing else, allowed error tolerance |
| 352 |
notwithstanding. |
| 353 |
|
| 354 |
|
| 355 |
Driver Forward/Backward Compatibility |
| 356 |
===================================== |
| 357 |
|
| 358 |
As this is uAPI drivers can't regress color pipelines that have been |
| 359 |
introduced for a given HW generation. New HW generations are free to |
| 360 |
abandon color pipelines advertised for previous generations. |
| 361 |
Nevertheless, it can be beneficial to carry support for existing color |
| 362 |
pipelines forward as those will likely already have support in DRM |
| 363 |
clients. |
| 364 |
|
| 365 |
Introducing new colorops to a pipeline is fine, as long as they can be |
| 366 |
bypassed or are purely informational. DRM clients implementing support |
| 367 |
for the pipeline can always skip unknown properties as long as they can |
| 368 |
be confident that doing so will not cause unexpected results. |
| 369 |
|
| 370 |
If a new colorop doesn't fall into one of the above categories |
| 371 |
(bypassable or informational) the modified pipeline would be unusable |
| 372 |
for user space. In this case a new pipeline should be defined. |
| 373 |
|
| 374 |
|
| 375 |
References |
| 376 |
========== |
| 377 |
|
| 378 |
1. https://lore.kernel.org/dri-devel/QMers3awXvNCQlyhWdTtsPwkp5ie9bze_hD5nAccFW7a_RXlWjYB7MoUW_8CKLT2bSQwIXVi5H6VULYIxCdgvryZoAoJnC5lZgyK1QWn488=@emersion.fr/ |