codec/progressive: Fix wrong usage of subband diffing flag (#8076)

Currently, all Calista Progressive encoded streams contain tile
artifacts, when the RFX_SUBBAND_DIFFING is used, but not the
RFX_DWT_REDUCE_EXTRAPOLATE flag.
The reason is the wrong usage of the context and tile flags.
The RFX_SUBBAND_DIFFING flag should have no actual impact on the
decoder itself.
Especially, it does not affect the band sizes within a 64x64 tile.
The RFX_DWT_REDUCE_EXTRAPOLATE flag, on the other hand, MUST have an
effect on the band sizes.
However, FreeRDP currently uses the RFX_SUBBAND_DIFFING flag when
decoding a component to determine whether the Reduce-Extrapolate method
is used, resulting in tile artifacts, when that method was actually not
used.
The current behaviour did not result in tile artifacts with the MS
Windows RDS, as that server always sets both flags.

So, fix this issue by using the correct flag, when decoding a tile.
This commit is contained in:
Pascal Nowack 2022-07-15 08:32:33 +02:00 committed by GitHub
parent dfa231c0a5
commit c7d1a2cdb5
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

View File

@ -907,7 +907,7 @@ static INLINE int progressive_rfx_decode_component(PROGRESSIVE_CONTEXT* progress
return status;
CopyMemory(sign, buffer, 4096 * 2);
if (!subbandDiff)
if (!extrapolate)
{
rfx_differential_decode(buffer + 4032, 64);
progressive_rfx_decode_block(prims, &buffer[0], 1024, shift->HL1); /* HL1 */