Resolves: fdo#86493 Fix crash while scaling large bitmaps.

Fast bitmap scaling overflowed the LUT used by the nearest-neighbor algorithm.
When a bitmap has 46k pixel on a side and is enlarged, the scaling code
overflows the 32-bit long, resulting in negative indexes, which then segfaults.

This isn't as rare as it sounds. At least in web-view in writer the border/shadow
bitmap is as long as the document (which is an issue in its own right,)
which can overflow for large documents during scaling and segfault.

Reviewed-on: https://gerrit.libreoffice.org/14597
Reviewed-by: Caolán McNamara <caolanm@redhat.com>
Tested-by: Caolán McNamara <caolanm@redhat.com>
(cherry picked from commit c91bfb9ac7d110c5dca0ea34ec0e1668a985b34c)

Change-Id: I1ccf73d02469f6601a9a7e67b30524cb497cf6bc
Reviewed-on: https://gerrit.libreoffice.org/14809
Reviewed-by: Miklos Vajna <vmiklos@collabora.co.uk>
Tested-by: Miklos Vajna <vmiklos@collabora.co.uk>
(cherry picked from commit e40f78753e10be6ca867aac593b6f0be166f3b73)
Signed-off-by: Michael Stahl <mstahl@redhat.com>

1 files changed, 7 insertions, 6 deletions

diff --git a/vcl/source/gdi/bitmap3.cxx b/vcl/source/gdi/bitmap3.cxx
index 0384e0005fa1..2ed8c65244f4 100644
--- a/vcl/source/gdi/bitmap3.cxx
+++ b/vcl/source/gdi/bitmap3.cxx
@@ -1047,18 +1047,19 @@ bool Bitmap::ImplScaleFast( const double& rScaleX, const double& rScaleY )
                 const long nScanlineSize = pWriteAcc->GetScanlineSize();
                 const long nNewWidth1 = nNewWidth - 1L;
                 const long nNewHeight1 = nNewHeight - 1L;
-                const long nWidth = pReadAcc->Width();
-                const long nHeight = pReadAcc->Height();
-                boost::scoped_array<long> pLutX(new long[ nNewWidth ]);
-                boost::scoped_array<long> pLutY(new long[ nNewHeight ]);
 
                 if( nNewWidth1 && nNewHeight1 )
                 {
+                    const double nWidth = pReadAcc->Width();
+                    const double nHeight = pReadAcc->Height();
+                    boost::scoped_array<long> pLutX(new long[ nNewWidth ]);
+                    boost::scoped_array<long> pLutY(new long[ nNewHeight ]);
+
                     for( long nX = 0L; nX < nNewWidth; nX++ )
-                        pLutX[ nX ] = nX * nWidth / nNewWidth;
+                        pLutX[ nX ] = long(nX * nWidth / nNewWidth);
 
                     for( long nY = 0L; nY < nNewHeight; nY++ )
-                        pLutY[ nY ] = nY * nHeight / nNewHeight;
+                        pLutY[ nY ] = long(nY * nHeight / nNewHeight);
 
                     long nActY = 0L;
                     while( nActY < nNewHeight )