tdf#126127: Make nTmp smaller still, avoid -fsanitize=signed-integer-overflow

...after f2e3655255db4032738849cd4b77ce67a6e2c984 "Avoid
 -fsanitize=signed-integer-overflow" had already reduced it from using LONG_MAX
to TWIPS_MAX/2 in the past.  This time, avoid the computation of

> const sal_uInt64 nCurrentDist = sal_Int64(aDiff.getX()) * sal_Int64(aDiff.getX()) + sal_Int64(aDiff.getY()) * sal_Int64(aDiff.getY()); // opt: no sqrt

in GetFrameOfModify (sw/source/core/layout/frmtool.cxx) from overflowing (where
aDiff.getY() derives from nTmp and can be close to it in magnitude, so computing
its square would overflow on platforms where TWIPS_MAX is a large sal_Int64
value).

(The "empirically shown to be large enough in practice" in the comment is a
successful `make check` on Linux 64-bit with UBSan.)

Change-Id: Ic7f058bd6853ff04ccb50a150509e98f850d12d2
Reviewed-on: https://gerrit.libreoffice.org/74801
Reviewed-by: Michael Stahl <michael.stahl@cib.de>
Tested-by: Jenkins
(cherry picked from commit 8723ac4e20eda87a82393f2f6c7d28ece8514238)

1 files changed, 8 insertions, 1 deletions

diff --git a/sw/source/core/text/frmform.cxx b/sw/source/core/text/frmform.cxx
index 176914b69040..959daa4ea9a1 100644
--- a/sw/source/core/text/frmform.cxx
+++ b/sw/source/core/text/frmform.cxx
@@ -842,7 +842,14 @@ bool SwTextFrame::CalcPreps()
                 }
                 else
                 {
-                    SwTwips nTmp  = TWIPS_MAX/2 - (getFrameArea().Top()+10000);
+                    // nTmp should be very large, but not so large as to cause overflow later (e.g.,
+                    // GetFrameOfModify in sw/source/core/layout/frmtool.cxx calculates nCurrentDist
+                    // from, among others, the square of aDiff.getY(), which can be close to nTmp);
+                    // the previously used value TWIPS_MAX/2 (i.e., (LONG_MAX - 1)/2) depended on
+                    // the range of 'long', while the value (SAL_MAX_INT32 - 1)/2 (which matches the
+                    // old value on platforms where 'long' is 'sal_Int32') is empirically shown to
+                    // be large enough in practice even on platforms where 'long' is 'sal_Int64':
+                    SwTwips nTmp  = (SAL_MAX_INT32 - 1)/2 - (getFrameArea().Top()+10000);
                     SwTwips nDiff = nTmp - getFrameArea().Height();
 
                     {