From 9a754434f1c3cd49877911bbd84cf1ecad781152 Mon Sep 17 00:00:00 2001 From: Robert Mader Date: Tue, 11 Jun 2024 14:35:53 +0200 Subject: [PATCH] gl-renderer: Fix limited range quantization MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The highest possible value is 255, not 256, like in previous part of the calculation. While on it, use the values directly, making the code more readable. Fix suggested by Pekka Paalanen and Loïc Molinari. Signed-off-by: Robert Mader --- libweston/renderer-gl/fragment.glsl | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/libweston/renderer-gl/fragment.glsl b/libweston/renderer-gl/fragment.glsl index 27a5640d..332149b0 100644 --- a/libweston/renderer-gl/fragment.glsl +++ b/libweston/renderer-gl/fragment.glsl @@ -102,12 +102,11 @@ yuva2rgba(vec4 yuva) /* ITU-R BT.601 & BT.709 quantization (limited range) */ - /* Y = 255/219 * (x - 16/256) */ - Y = 1.16438356 * (yuva.x - 0.0625); + Y = 255.0/219.0 * (yuva.x - 16.0/255.0); - /* Remove offset 128/256, but the 255/224 multiplier comes later */ - su = yuva.y - 0.5; - sv = yuva.z - 0.5; + /* Remove offset 128/255, but the 255/224 multiplier comes later */ + su = yuva.y - 128.0/255.0; + sv = yuva.z - 128.0/255.0; /* * ITU-R BT.601 encoding coefficients (inverse), with the