Fix xmlSecSize size mismatch

Since commit 9630a2dfc79b08e3417e6e69b083f5124614499c,
CppunitTest_xmlsecurity_signing on Win64 segfaults:

===
[CUT] xmlsecurity_signing
/usr/bin/sh: line 1: 10188 Segmentation fault      ( PATH="C:\lo\src\core\instdir\program;C:\lo\src\core\instdir\program;C:\lo\src\core\workdir\LinkTarget\Library;C:\lo\src\core\workdir\UnpackedTarball\cppunit\src\cppunit\DebugDll;$PATH" $W/LinkTarget/Executable/cppunittester.exe $W/LinkTarget/CppunitTest/test_xmlsecurity_signing.dll --headless "-env:BRAND_BASE_DIR=file:///$S/instdir" "-env:BRAND_SHARE_SUBDIR=share" "-env:BRAND_SHARE_RESOURCE_SUBDIR=program/resource" "-env:UserInstallation=file:///$W/CppunitTest/xmlsecurity_signing.test.user" "-env:CONFIGURATION_LAYERS=xcsxcu:file:///$I/share/registry xcsxcu:file:///$W/unittest/registry" "-env:UNO_TYPES=file:///$I/program/types.rdb file:///$I/program/types/offapi.rdb" "-env:UNO_SERVICES=file:///$W/Rdb/ure/services.rdb file:///$W/Rdb/services.rdb" -env:URE_INTERNAL_LIB_DIR=file:///$I/program -env:LO_LIB_DIR=file:///$I/program -env:LO_JAVA_DIR=file:///$I/program/classes --protector $W/LinkTarget/Library/unoexceptionprotector.dll unoexceptionprotector --protector $W/LinkTarget/Library/unobootstrapprotector.dll unobootstrapprotector --protector $W/LinkTarget/Library/vclbootstrapprotector.dll vclbootstrapprotector "-env:CPPUNITTESTTARGET=$W/CppunitTest/xmlsecurity_signing.test" ) > $W/CppunitTest/xmlsecurity_signing.test.log 2>&1
warn:sfx.appl:18084:18824:sfx2/source/appl/app.cxx:191: No DDE-Service possible. Error: 16399
warn:vcl:18084:18824:vcl/win/window/salframe.cxx:1084: WinSalFrame::SetIcon(): Could not load large icon !
warn:vcl:18084:18824:vcl/win/window/salframe.cxx:1085: WinSalFrame::SetIcon(): Could not load small icon !
warn:basic:18084:18824:basic/source/uno/namecont.cxx:973: Cannot access extensions!
warn:basic:18084:18824:basic/source/uno/namecont.cxx:973: Cannot access extensions!
warn:xmlsecurity.xmlsec:18084:18824:xmlsecurity/source/xmlsec/errorcallback.cxx:51: ..\src\xmldsig.c:793: xmlSecDSigCtxProcessKeyInfoNode() '' '' 45 'details=NULL' Operation completed successfully.
warn:xmlsecurity.xmlsec:18084:18824:xmlsecurity/source/xmlsec/errorcallback.cxx:51: ..\src\xmldsig.c:508: xmlSecDSigCtxProcessSignatureNode() '' 'xmlSecDSigCtxProcessKeyInfoNode' 1 ' ' Operation completed successfully.
warn:xmlsecurity.xmlsec:18084:18824:xmlsecurity/source/xmlsec/errorcallback.cxx:51: ..\src\xmldsig.c:291: xmlSecDSigCtxSign() '' 'xmlSecDSigCtxProcessSignatureNode' 1 ' ' Operation completed successfully.
HEAP CORRUPTION DETECTED: after Normal block (#1570713) at 0x00000197AC7E5AB0.
CRT detected that the application wrote to memory after end of heap buffer.
warn:xmlsecurity.xmlsec:18084:18824:xmlsecurity/source/xmlsec/errorcallback.cxx:51: ..\src\xmldsig.c:793: xmlSecDSigCtxProcessKeyInfoNode() '' '' 45 'details=NULL' Operation completed successfully.
warn:xmlsecurity.xmlsec:18084:18824:xmlsecurity/source/xmlsec/errorcallback.cxx:51: ..\src\xmldsig.c:508: xmlSecDSigCtxProcessSignatureNode() '' 'xmlSecDSigCtxProcessKeyInfoNode' 1 ' ' Operation completed successfully.
warn:xmlsecurity.xmlsec:18084:18824:xmlsecurity/source/xmlsec/errorcallback.cxx:51: ..\src\xmldsig.c:346: xmlSecDSigCtxVerify() '' 'xmlSecDSigCtxProcessSignatureNode' 1 ' ' Operation completed successfully.
HEAP CORRUPTION DETECTED: after Normal block (#1585431) at 0x00000197AC7E7BF0.
CRT detected that the application wrote to memory after end of heap buffer.
SigningTest::testDescription finished in: 3332ms
warn:vcl:18084:18824:vcl/win/window/salframe.cxx:1084: WinSalFrame::SetIcon(): Could not load large icon !
warn:vcl:18084:18824:vcl/win/window/salframe.cxx:1085: WinSalFrame::SetIcon(): Could not load small icon !
SigningTest::testECDSA finished in: 550ms
warn:vcl:18084:18824:vcl/win/window/salframe.cxx:1084: WinSalFrame::SetIcon(): Could not load large icon !
warn:vcl:18084:18824:vcl/win/window/salframe.cxx:1085: WinSalFrame::SetIcon(): Could not load small icon !
SigningTest::testECDSAOOXML finished in: 466ms
warn:vcl:18084:18824:vcl/win/window/salframe.cxx:1084: WinSalFrame::SetIcon(): Could not load large icon !
warn:vcl:18084:18824:vcl/win/window/salframe.cxx:1085: WinSalFrame::SetIcon(): Could not load small icon !
warn:vcl.gdi:18084:18824:vcl/source/outdev/map.cxx:694: Please record only relative MapModes!
warn:vcl.gdi:18084:18824:vcl/source/outdev/map.cxx:694: Please record only relative MapModes!
warn:vcl.gdi:18084:18824:vcl/source/outdev/map.cxx:694: Please record only relative MapModes!
warn:vcl.gdi:18084:18824:vcl/source/outdev/map.cxx:694: Please record only relative MapModes!
warn:vcl.gdi:18084:18824:vcl/source/outdev/map.cxx:694: Please record only relative MapModes!
warn:vcl.gdi:18084:18824:vcl/source/outdev/map.cxx:694: Please record only relative MapModes!
SigningTest::testECDSAPDF finished in: 433ms
warn:xmlsecurity.xmlsec:18084:18824:xmlsecurity/source/xmlsec/errorcallback.cxx:51: ..\src\xmldsig.c:793: xmlSecDSigCtxProcessKeyInfoNode() '' '' 45 'details=NULL' Operation completed successfully.
warn:xmlsecurity.xmlsec:18084:18824:xmlsecurity/source/xmlsec/errorcallback.cxx:51: ..\src\xmldsig.c:508: xmlSecDSigCtxProcessSignatureNode() '' 'xmlSecDSigCtxProcessKeyInfoNode' 1 ' ' Operation completed successfully.
warn:xmlsecurity.xmlsec:18084:18824:xmlsecurity/source/xmlsec/errorcallback.cxx:51: ..\src\xmldsig.c:346: xmlSecDSigCtxVerify() '' 'xmlSecDSigCtxProcessSignatureNode' 1 ' ' Operation completed successfully.
===

The problem is mismatch between sizeof(xmlSecSize) in xmlsec and LO:
xmlsec uses 32-bit integer, while LO uses 64-bit. Crash happens in
XMLSignature_MSCryptImpl::validate(), when pDsigCtx->manifestReferences
address is incorrectly retrieved and passed to xmlSecPtrListGetSize.

Despite the comment in xmlsecurity/inc/xmlsec-wrapper.h mentioned that
the XMLSEC_NO_SIZE_T isn't used in xmlsec for MSVC, it's actually used
there since commit 1cf0cd6f0f19c34a23228f7de691187887081dff. So we need
to enable it for MSVC, too.

Change-Id: I05a4f4f6700c178d28886a7ac203469c41d7048b
Reviewed-on: https://gerrit.libreoffice.org/62676
Reviewed-by: Miklos Vajna <vmiklos@collabora.co.uk>
Tested-by: Mike Kaganski <mike.kaganski@collabora.com>

1 files changed, 2 insertions, 2 deletions

diff --git a/xmlsecurity/inc/xmlsec-wrapper.h b/xmlsecurity/inc/xmlsec-wrapper.h
index 0633bd3c1585..c060c8bf23b8 100644
--- a/xmlsecurity/inc/xmlsec-wrapper.h
+++ b/xmlsecurity/inc/xmlsec-wrapper.h
@@ -24,8 +24,8 @@
 
 #include <sal/types.h>
 
-// Cf. xmlsec's configure.in (but which isn't used for MSVC):
-#if !defined _MSC_VER && SAL_TYPES_SIZEOFPOINTER != 4 && !defined SYSTEM_XMLSEC
+// Cf. xmlsec's configure.in:
+#if SAL_TYPES_SIZEOFPOINTER != 4 && !defined SYSTEM_XMLSEC
 #define XMLSEC_NO_SIZE_T
 #endif