tdf#55960 sc: fix floating point noise in FillSeriesHEADmaster

When Calc generates a linear fill series (e.g. -2.0, -1.95, -1.90, ...),
the result of nStartVal + nStepValue * nIndex is computed in IEEE 754
double arithmetic. Because 0.05 is not exactly representable in binary,
the accumulated value drifts from the mathematically expected result,
producing user-visible noise such as 0.69999999999999996 instead of 0.7,
or a tiny residual like 5.5e-17 instead of 0.0.

For example, starting at -2.0 with increment 0.05 and filling downward:
previously cell A41 (the value that should be 0.0) showed 5.55111512312578E-17
and cell A55 (expected 0.7) showed 0.69999999999999996. After this fix
both cells display the exact expected values.

This patch applies rtl::math::approxValue() to the computed value inside
the FILL_LINEAR / non-duration branch. approxValue() rounds the value to
its 16th significant decimal digit, which removes the representational
noise while preserving meaningful precision. The function handles zero,
very large values, and infinities without special-casing: zero stays
zero (no division involved), large/infinite values simply pass through
unchanged because they already have fewer than 16 significant digits of
mantissa.

The rounding is placed directly in the else-clause of the duration check
so no additional runtime condition is needed. Both fill loops for numeric
cells and string-with-number cells receive the same treatment.

Change-Id: If2b9bbb9bd56c7b069decf4dd4b94658e919d162
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/201071
Tested-by: Jenkins
Reviewed-by: Mike Kaganski <mike.kaganski@collabora.com>

0 files changed, 0 insertions, 0 deletions