From b52991537c0efe27ee0c1955eb28a4584226d8b5 Mon Sep 17 00:00:00 2001 From: Gonglei Date: Wed, 28 May 2014 21:21:35 +0800 Subject: [PATCH] vnc-enc-tight: Fix divide-by-zero in tight_detect_smooth_image{16,24,32} Spotted by Coverity: (1) Event assignment: Assigning: "pixels" = "0". (2) Event cond_true: Condition "y < h", taking true branch (3) Event cond_false: Condition "x < w", taking false branch (4) Event loop_end: Reached end of loop (5) Event divide_by_zero: In expression "(stats[0] + stats[1]) * 100U / pixels", division by expression "pixels" which may be zero has undefined behavior. 290 DEFINE_DETECT_FUNCTION(16) 291 DEFINE_DETECT_FUNCTION(32) Signed-off-by: Gonglei Signed-off-by: Gerd Hoffmann --- ui/vnc-enc-tight.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/ui/vnc-enc-tight.c b/ui/vnc-enc-tight.c index 59b59c0c79..f02352cc46 100644 --- a/ui/vnc-enc-tight.c +++ b/ui/vnc-enc-tight.c @@ -181,6 +181,10 @@ tight_detect_smooth_image24(VncState *vs, int w, int h) } } + if (pixels == 0) { + return 0; + } + /* 95% smooth or more ... */ if (stats[0] * 33 / pixels >= 95) { return 0; @@ -267,7 +271,9 @@ tight_detect_smooth_image24(VncState *vs, int w, int h) y += w; \ } \ } \ - \ + if (pixels == 0) { \ + return 0; \ + } \ if ((stats[0] + stats[1]) * 100 / pixels >= 90) { \ return 0; \ } \