Fix -fsanitize=shift-exponent

...as seen with `--convert-to pdf cdr/fdo55522-1.cdr` with cdr/fdo55522-1.cdr as
obtained by bin/get-bugzilla-attachments-by-mimetype (i.e., the attachment at
<https://bugs.documentfoundation.org/show_bug.cgi?id=55522#c0>):

> vcl/source/fontsubset/cff.cxx:737:35: runtime error: shift exponent 32 is too large for 32-bit type 'unsigned int'
>  #0 in CffSubsetterContext::convertOneTypeOp() at vcl/source/fontsubset/cff.cxx:737:35 (instdir/program/libvcllo.so +0x9489ce3)
>  #1 in CffSubsetterContext::convert2Type1Ops(CffLocal*, unsigned char const*, int, unsigned char*) at vcl/source/fontsubset/cff.cxx:1117:9 (instdir/program/libvcllo.so +0x94970d3)
>  #2 in CffSubsetterContext::emitAsType1(Type1Emitter&, unsigned short const*, unsigned char const*, int*, int, FontSubsetInfo&) at vcl/source/fontsubset/cff.cxx:1969:28 (instdir/program/libvcllo.so +0x94a9ec8)
[...]

If any of these "overflow" bits of nHintMask should have been set by the
preceding for loop, mbIgnoreHints would have been set and this for loop wouldn't
be reached.

Change-Id: I0fd6de10610b52300e081770e9df1078e7ee5f92
Reviewed-on: https://gerrit.libreoffice.org/73247
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>

1 files changed, 3 insertions, 2 deletions

diff --git a/vcl/source/fontsubset/cff.cxx b/vcl/source/fontsubset/cff.cxx
index d9abcca4ef45..17112310ece0 100644
--- a/vcl/source/fontsubset/cff.cxx
+++ b/vcl/source/fontsubset/cff.cxx
@@ -713,6 +713,7 @@ void CffSubsetterContext::convertOneTypeOp()
         int nCntrBits[2] = {0,0};
         U8 nMaskBit = 0;
         U8 nMaskByte = 0;
+        int const MASK_BITS = 8*sizeof(nHintMask);
         for( i = 0; i < mnHintSize; i+=2, nMaskBit>>=1) {
             if( !nMaskBit) {
                 nMaskByte = *(mpReadPtr++);
@@ -720,7 +721,7 @@ void CffSubsetterContext::convertOneTypeOp()
             }
             if( !(nMaskByte & nMaskBit))
                 continue;
-            if( i >= 8*int(sizeof(nHintMask)))
+            if( i >= MASK_BITS)
                 mbIgnoreHints = true;
             if( mbIgnoreHints)
                 continue;
@@ -734,7 +735,7 @@ void CffSubsetterContext::convertOneTypeOp()
             break;
 
         for( i = 0; i < mnHintSize; i+=2) {
-            if( !(nHintMask & (1U << i)))
+            if(i >= MASK_BITS || !(nHintMask & (1U << i)))
                 continue;
             writeType1Val( mnHintStack[i]);
             writeType1Val( mnHintStack[i+1] - mnHintStack[i]);