integer overflow in XvCreateImage() [CVE-2013-1989 3/3]

num_planes is a CARD32 and needs to be bounds checked before bit shifting
and adding to sizeof(XvImage) to come up with the total size to allocate,
to avoid integer overflow leading to underallocation and writing data from
the network past the end of the allocated buffer.

Reported-by: Ilja Van Sprundel <ivansprundel@ioactive.com>
Signed-off-by: Alan Coopersmith <alan.coopersmith@oracle.com>

1 files changed, 4 insertions, 1 deletions

diff --git a/src/Xv.c b/src/Xv.c
index 0a07d9d..f268f8e 100644
--- a/src/Xv.c
+++ b/src/Xv.c
@@ -992,7 +992,10 @@ XvImage * XvCreateImage (
       return NULL;
    }
 
-   if((ret = (XvImage*)Xmalloc(sizeof(XvImage) + (rep.num_planes << 3)))) {
+   if (rep.num_planes < ((INT_MAX >> 3) - sizeof(XvImage)))
+       ret = Xmalloc(sizeof(XvImage) + (rep.num_planes << 3));
+
+   if (ret != NULL) {
 	ret->id = id;
 	ret->width = rep.width;
 	ret->height = rep.height;