Better integrate with OS-level support for HDR displays

[doug-walker]

OCIO has support for using an ICC monitor profile to instantiate a "virtual display" that may be added to a config and used with a specific set of views. However, as operating systems such as macOS have added support for HDR (aka XDR) display technology, they have been using proprietary methods that do not rely on ICC profiles. (This is partly because the ICC format does not really support HDR monitors.)

As part of this transition to HDR displays, it has become less and less possible to bypass OS-level color management. So rather than OCIO calculating the exact RGB code values that are sent to the monitor, increasingly the application must accept that the OS is going to be modifying the values. This implies that the host application needs to tag the images in accordance with whatever processing has been done by OCIO. Therefore an OCIO display color space will need to contain some extra information to allow the application to use the appropriate OS-specific tag.

Furthermore, the application may need to do some negotiation between the color space tags that are in the config and the color space tags supported by the host OS and the capabilities of the connected displays. This will requires some further additions to the OCIO API.

[brechtvl]

When implementing HDR support in Blender, I found that there are a number of things to balance and choices to make:

• Are you drawing images:
  • Fullscreen
  • In a dedicated window
  • In a window mixed with other user interface elements
• Which color spaces does your OS / compositor / GPU driver support?
• Want to use as much gamut and luminance as the display supports, or limit to an intended output target.
• Let the OS handle HLG tone mapping, or do it yourself (or not at all).
• Do you try to draw into the native color space of the compositor, display or GPU to minimize transforms and improve performance / memory usage.

And in practice it seems that:

• On Windows, 16 bit linear extended sRGB is the only well supported choice in windowed mode. In fullscreen the Windows compositor is bypassed and GPU drivers also offer other choices like 10 bit PQ.
• On Linux + Wayland linear extended sRGB also seemed the to be the choice that is widely supported and well defined.
• Different applications (video players, browsers) handle HLG tone mapping with different algorithms. On macOS you can pass HLG metadata like max luminance to the OS. My understanding is that on Windows you have to do tone mapping yourself, and Linux + Wayland does support the metadata but compositors may not.

In Blender we have a cross platform application, and the image is displayed in the same window as other UI elements. We found the simplest was just to target (linear) extended sRGB that we know to be well supported on all platforms and not bother negotiating the color space. We clip to the gamut and luminance for the display chosen by the user, so they can preview what e.g. an sRGB image will look like when saved to file even if the monitor supports a wider gamut.

I don't know if that's the right choice for all applications. But I suspect that in practice it might not be worth negotiating much unless you really want to squeeze out maximum performance or care about macOS doing HLG tone mapping for you. In other cases I suspect the OS just ends up doing the same conversion that are possible with OpenColorIO too. But I may well have missed some use cases.

[doug-walker]

Brecht, this is really great input, thank you for sharing your thoughts on this!

Reading your post made me want to add a few more thoughts, partially in reply to you and partly just to elaborate on the thinking behind the project.

One of the core goals of OCIO is to give the VFX facility color scientists control over as much of the color pipeline as possible. Historically, the expectation was that the output of the OCIO display & view was sent directly to the monitor. With HDR we are entering this brave new world where additional processing is being inserted into that pipeline, outside of their control.

On one level, we could argue nothing new is needed since OCIO v2 empowers applications to do the necessary post-processing, e.g. by inverting the selected display color space and converting into a different color space to hand to the window system. However, part of my motivation here is to start a conversation about how configs authors would prefer the applications do this, taking into account that, as you point out, there are these different scenarios (what does the OS/GPU support, is a window full screen, are there UI widgets, etc.).

I could imagine config authors saying that they want more control, e.g., to put the required color spaces for HDR in their config and insist that applications only use those and not do their own post-processing. And further, they may have preferences about which are used, if the application has options. I'm curious to hear what people say about this.

I'm not aware of any OCIO applications that support HDR from arbitrary configs, they are already placing various requirements on config authors. Perhaps there should be a standard set of things which config authors should be expected to include to enable HDR support? There could then be an OCIO API call to validate that and ociocheck could print whether a config supports HDR.

The presence of the new interop IDs in display color spaces and the cie_xyz_d65_interchange role definitely simplify things for application developers, so these are the types of things that could be considered.

One output of this process I'd like to see is a set of recommended practices for config authors and application developers. I do think it's possible some API additions could be helpful as well (more to come on that), but we'll see what people think.

We're planning on devoting the May 4 OCIO working group meeting to this topic, please drop by if you're available.