fix 2Gb integer overflow in in VNC tight and zlib encodings
As found by Roland Dreier <roland@purestorage.com> (excellent
catch!), when amount of VNC compressed data produced by zlib
and sent to client exceeds 2Gb, integer overflow occurs because
currently, we calculate amount of data produced at each step by
comparing saved total_out with new total_out, and total_out is
something which grows without bounds. Compare it with previous
avail_out instead of total_out, and leave total_out alone.
The same code is used in vnc-enc-tight.c and vnc-enc-zlib.c,
so fix both cases.
There, there's no actual need to save previous_out value, since
capacity-offset (which is how that value is calculated) stays
the same so it can be recalculated again after call to deflate(),
but whole thing becomes less readable this way.
Reported-by: Roland Dreier <roland@purestorage.com>
Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>
Signed-off-by: Corentin Chary <corentin.chary@gmail.com>
Signed-off-by: Aurelien Jarno <aurelien@aurel32.net>
Add some missing #ifdefs to fix compilation failures in the !CONFIG_VNC_JPEG
case introduced by commit ce702e93.
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
This option allow to disable adaptive behaviors in some encodings.
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
Use the new update frequency infrastructure to use jpeg for regions with
high update frequency.
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
bswap_NN() variants are not always available in CONFIG_MACHINE_BSWAP_H case
and bswapNN() are public APIs in "bswap.h".
Signed-off-by: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
Commit d167f9bc06 missed this one:
/src/qemu/ui/vnc-enc-tight.c:1483: warning: 'ret' may be used uninitialized in this function
Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
Commit 7bccf57383 missed this one:
/src/qemu/ui/vnc-enc-tight.c: In function 'send_sub_rect':
/src/qemu/ui/vnc-enc-tight.c:1527: warning: comparison is always true due to limited range of data type
Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
Fix mingw32 build errors like
/src/qemu/ui/vnc-enc-tight.c: In function 'tight_detect_smooth_image24':
/src/qemu/ui/vnc-enc-tight.c:119: error: 'uint' undeclared (first use in this function)
Replace 'uint' with proper 'unsigned int'.
Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
Fix a warning with some GCCs:
/src/qemu/ui/vnc-enc-tight.c: In function `send_sub_rect_nojpeg':
/src/qemu/ui/vnc-enc-tight.c:1458: warning: `ret' might be used uninitialized in this function
Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
Fix the following warnings:
/src/qemu/hw/ide/core.c: In function `ide_drive_pio_post_load':
/src/qemu/hw/ide/core.c:2767: warning: comparison is always false due to limited range of data type
/src/qemu/ui/vnc-enc-tight.c: In function `tight_detect_smooth_image':
/src/qemu/ui/vnc-enc-tight.c:284: warning: comparison is always true due to limited range of data type
/src/qemu/ui/vnc-enc-tight.c:297: warning: comparison is always true due to limited range of data type
/src/qemu/ui/vnc-enc-tight.c: In function `tight_encode_indexed_rect16':
/src/qemu/ui/vnc-enc-tight.c:456: warning: comparison is always false due to limited range of data type
/src/qemu/ui/vnc-enc-tight.c: In function `tight_encode_indexed_rect32':
/src/qemu/ui/vnc-enc-tight.c:457: warning: comparison is always false due to limited range of data type
Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
Split send_sub_rect in send_sub_rect_jpeg and send_sub_rect_nojpeg to
remove all these #ifdef CONFIG_JPEG.
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
rgb_prepare_row bpp depends on the server display surface, not
the client.
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
This will allow to implement the threaded VNC server in a
more cleaner way.
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
Profiling with callgrind seems to show that a lot of time is spent
in the palette code (mostly due to memory allocation and qdict to int
conversion).
This patch adds a VncPalette implementation. The palette is stored
in a hash table, like qdict, but which does way less memory allocations,
and doesn't suffer from the QObject overhead.
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
Disable png filters for lower compression levels. This should lower
the CPU consumption and reduce encoding time.
This isn't in tight_conf because:
* tight_conf structure must not change, because it's shared with other
tight implementations (libvncserver, etc..).
* it'd exceed the 80 col limit.
* PNG_ macros are only defined if CONFIG_VNC_PNG is defined
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
Introduce a new encoding: VNC_ENCODING_TIGHT_PNG [1] (-269) with a new
tight filter VNC_TIGHT_PNG (0x0A). When the client tells it supports the Tight PNG
encoding, the server will use tight, but will always send encoding pixels using
PNG instead of zlib. If the client also told it support JPEG, then the server can
send JPEG, because PNG will only be used in the cases zlib was used in normal tight.
This encoding was introduced to speed up HTML5 based VNC clients like noVNC [2], but
can also be used on devices like iPhone where PNG can be rendered in hardware.
[1] http://wiki.qemu.org/VNC_Tight_PNG
[2] http://github.com/kanaka/noVNC/
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
While using indexed colors, the last color was never added to the palette.
Triggered with ubuntu livecd.
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
For the same reason that we don't use vnc-authentication-sasl.c but
vnc-auth-sals.c. Because it's tooooo long.
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
