HDMI FRL: EDID max bit depth is preferred even when it forces DSC

Hi Harry,

A question about the bit depth and DSC policy for HDMI FRL, with test
data from an RX 6800 (DCN 3.0) driving an LG 42C2 at 3840x2160p120,
stock v7.2-rc7 with amdgpu.dcfeaturemask=0x402, direct HDMI, no PCON.
As an aside, with Tomasz's out of tree HDMI VRR patches on top of the
same base, FRL and native HDMI VRR come up and stay active together on
this DCN generation, which I had not seen reported anywhere before.

By default the stream ends up at 12bpc with DSC on, at 3 lanes and
6Gbps. The path there:

- FRL verification trains at rate 5 (4 lanes at 10Gbps) on the first
  attempt with no fallback, so the link is not the constraint.
- Color depth is committed before DSC is considered and defaults to
  the EDID maximum, 12bpc. At this timing
  dc_bandwidth_in_kbps_from_timing() gives 42.77 Gbps for 12bpc RGB,
  above the 40 Gbps dc_link_frl_bandwidth_kbps() reports for rate 5,
  so apply_dsc_policy_for_stream() enables DSC.
- Nothing afterwards reconsiders the depth. dc_dsc_compute_config()
  is called with the 12bpc timing and the sink advertises DSC decode
  at 12bpc, so it succeeds. The DSC target comes from
  frl_dsc_max_frl_rate, which is 3 on this sink, so 24 Gbps, and
  hdmi_frl_decide_link_settings() then walks up from the lowest rate
  and stops at the first one that validates, 3 lanes at 6Gbps.

Capping GNOME's max bpc setting to 10, via a maxbpc element in the
monitor entry of monitors.xml, gives the other outcome on the same
kernel, cable and display: 10bpc RGB needs 35.64 Gbps, fits the
verified rate uncompressed, so no DSC, and the stream runs at 4 lanes
and 10Gbps.

I am honestly not sure which of the two is preferable. I have not done
careful side by side testing and I cannot see a difference between
12bpc with DSC and 10bpc without it. What made me look in the first
place is that the AMD Windows driver ends up at YCbCr444 10bpc at 4
lanes and 10Gbps on the same GPU, cable and TV, and without DSC. The
TV's info panel flags DSC when it is active, and it does flag it in the
default Linux case but not on Windows. So I thought you should know the
Linux default lands somewhere else. One other detail that may matter:
this sink advertises DSC decode at 12bpc only, with a DSC FRL rate cap
of 3, so uncompressed 12bpc at this timing is not reachable on this
link at all, and the panel is 10 bit in any case.

So the question: is preferring the maximum advertised depth deliberate
policy for HDMI FRL, or would it make sense to try lower depths
against the verified link rate before enabling DSC? The max bpc
property does get there, but it needs the user to know that, and on a
GNOME desktop it means hand editing a config file that configuration
tools can rewrite, while the driver already has the information.

One thing worth flagging, since it bears on any step down: for
DC_LINK_ENCODING_HDMI_FRL, dc_bandwidth_in_kbps_from_timing() returns
the raw pixel rate times bpp product and it is compared against the
raw FRL line rate, with no 16b/18b coding overhead on either side, so
the comparison is optimistic by about 12 percent. 10bpc at this timing
is therefore a lot closer to the rate 5 limit than 35.64 against 40
Gbps makes it look. What actually gates the mode is the capacity check
reached from frl_validate_mode_timing(), which did accept it, so
nothing unsafe happens today. Two related loose ends there: the
dc_crtc_timing field that would supply the accurate figure,
frl_uncompressed_video_bandwidth_in_kbps, is read in
dc_bandwidth_in_kbps_from_timing() but never assigned anywhere in the
tree, and dc_link_frl_margin_check_uncompressed_video() is exported in
dc.h and wired up in link_factory.c but has no callers. If a depth
step down were ever added, gating it on that check rather than on the
coarse comparison seems like the right basis.

I also have a debugfs proof of concept that does the step down inside
apply_dsc_policy_for_stream(), written quickly with Claude Sonnet while
I was diagnosing this, which I am happy to send if it is useful. It is
not needed to reproduce any of the above.

Happy to run further tests or provide EDID and FRL logs from this
hardware.

Thanks,
Antoine Richermoz (rcmz)