tdf#102694, bnc#957991: Improve pivot cache reading performance

This is an aggregated patch, containing all related changes on
master. Fixes some functional issues also next to the performance
problem and add all new test cases related this bug fix.

bnc#957991: Improve pivot cache reading performance

Performance problem was caused by grouping. Pivot
tables with the same source are linked to each other
by the pivot cache and so all the pivot tables were
updated when one group was added to one of the tables.

This code change first of all fixes a functional issue:
group name was wrongly imported and so pivot tables using
group fields were broken. This caused by calling
RefreshPivotTableGroups() on a pivot table which is not part
of the cache yet and so update was not called on this table.
Calling ReloadGroupTableData() solve this problem.

Second part of the codechange is about the consistency
of the pivot cache. We have an invariant in this code to
have the same groups for tabels with the same source.
To keep this invariant we update every newly inserted
tables adding the neccessary groups.
The performance improvement here is that until the table
is not part of the cache, it does not updates other tables.
Group syncronization is done when the table is inserted to
the ScDPCollection.

(cherry picked from commit 3b64a198568d5b2bb14066581aca112cc6182fd7)

This parameter can be const.

(cherry picked from commit b905116dc41a4101e4e44d39d946bbba6a3a334d)

Test the case for pivot tables sharing a group

Fixed with 3b64a198568d5b2bb14066581aca112cc6182fd7

(cherry picked from commit 049d2ffd329f88934a92b72e4ec650ea46bf8c61)

Test XLSX import of date groups

(cherry picked from commit f1a97fab957629d617b0b0ad44de5beb3d343f5c)

Test XLSX import of number groups

(cherry picked from commit 93098854c2d2b814a21161bd88071918cb382e87)

Pivot tables: Fix wrong group field number in cache

getCache should not append new group fields to the cache,
but use those which are stored in the dimension data.

(cherry picked from commit 7e1fd9aa86ab8d5e86f35df8615f438a3383af98)

Pivot tables: Fix XLSX import of nested date groups

Same fix which was added for other kind of groups:
f697d7aa5c26f9fcfd717b76a4827a5bcb38325e

(cherry picked from commit 9d19634c8e719a80674ca4b3dfc3c7e49f049e5b)

Pivot table XLSX import: PivotCache is not handled as a const object.

(cherry picked from commit db481dd30521f9ce498873cdc945cf9544bf983c)

tdf#102694, bnc#957991: Improve pivot cache reading performance

When two or more tables have the same source data, then
the grouping of source fields are shared between these
tables, so don't need to import these grouping for each
tables.
The added code checkes whether we already imported the
group fields and don't create them again using API
functions, but apply the exisiting groups of an other table
sharing the source.

(cherry picked from commit 03a1143cc75161dab56b20f1ab9e723ddd0caa8e)

Add better comment for this function.

(cherry picked from commit 97ebbb892993507fe19ca069534015637fde497b)

Change-Id: I04d81295491c4c0f351200670f9ebdae5644f858
Reviewed-on: https://gerrit.libreoffice.org/30048
Tested-by: Jenkins <ci@libreoffice.org>
Reviewed-by: Andras Timar <andras.timar@collabora.com>

2 files changed, 77 insertions, 5 deletions

diff --git a/sc/source/filter/oox/pivotcachebuffer.cxx b/sc/source/filter/oox/pivotcachebuffer.cxx
index c7b75e3b3a5c..687a47d976e7 100644
--- a/sc/source/filter/oox/pivotcachebuffer.cxx
+++ b/sc/source/filter/oox/pivotcachebuffer.cxx
@@ -1250,6 +1250,11 @@ sal_Int32 PivotCache::getCacheFieldCount() const
     return static_cast< sal_Int32 >( maFields.size() );
 }
 
+PivotCacheField* PivotCache::getCacheField( sal_Int32 nFieldIdx )
+{
+    return maFields.get( nFieldIdx ).get();
+}
+
 const PivotCacheField* PivotCache::getCacheField( sal_Int32 nFieldIdx ) const
 {
     return maFields.get( nFieldIdx ).get();
diff --git a/sc/source/filter/oox/pivottablebuffer.cxx b/sc/source/filter/oox/pivottablebuffer.cxx
index bc3c05ce8c82..04a432ae8d92 100644
--- a/sc/source/filter/oox/pivottablebuffer.cxx
+++ b/sc/source/filter/oox/pivottablebuffer.cxx
@@ -49,6 +49,9 @@
 #include "dapiuno.hxx"
 #include "dpobject.hxx"
 #include "dpsave.hxx"
+#include "dpdimsave.hxx"
+#include "document.hxx"
+#include "documentimport.hxx"
 
 namespace oox {
 namespace xls {
@@ -424,11 +427,12 @@ void PivotTableField::finalizeDateGroupingImport( const Reference< XDataPilotFie
 {
     if( maDPFieldName.isEmpty() )    // prevent endless loops if file format is broken
     {
-        if( const PivotCacheField* pCacheField = mrPivotTable.getCacheField( mnFieldIndex ) )
+        if( PivotCacheField* pCacheField = mrPivotTable.getCacheField( mnFieldIndex ) )
         {
             if( !pCacheField->isDatabaseField() && pCacheField->hasDateGrouping() && (pCacheField->getGroupBaseField() == nBaseFieldIdx) )
             {
                 maDPFieldName = pCacheField->createDateGroupField( rxBaseDPField );
+                pCacheField->setFinalGroupName(maDPFieldName);
                 OSL_ENSURE( !maDPFieldName.isEmpty(), "PivotTableField::finalizeDateGroupingImport - cannot create date group field" );
             }
         }
@@ -439,7 +443,7 @@ void PivotTableField::finalizeParentGroupingImport( const Reference< XDataPilotF
 {
     if( maDPFieldName.isEmpty() )    // prevent endless loops if file format is broken
     {
-        if( const PivotCacheField* pCacheField = mrPivotTable.getCacheField( mnFieldIndex ) )
+        if( PivotCacheField* pCacheField = mrPivotTable.getCacheField( mnFieldIndex ) )
         {
             // data field can have user defined groupname captions, apply them
             // if they do
@@ -449,10 +453,11 @@ void PivotTableField::finalizeParentGroupingImport( const Reference< XDataPilotF
                 if ( aIt->mnType == XML_data  && aIt->msCaption.getLength() )
                     captionList.push_back( IdCaptionPair( aIt->mnCacheItem, aIt->msCaption ) );
             }
-            // #FIXME find another way out of this const nightmare prison
             if ( !captionList.empty() )
-                const_cast<PivotCacheField*>( pCacheField )->applyItemCaptions( captionList );
+                pCacheField->applyItemCaptions( captionList );
+
             maDPFieldName = pCacheField->createParentGroupField( rxBaseDPField, rBaseCacheField, orItemNames );
+            pCacheField->setFinalGroupName(maDPFieldName);
             // on success, try to create nested group fields
             Reference< XDataPilotField > xDPField = mrPivotTable.getDataPilotField( maDPFieldName );
             if( xDPField.is() )
@@ -461,6 +466,33 @@ void PivotTableField::finalizeParentGroupingImport( const Reference< XDataPilotF
     }
 }
 
+void PivotTableField::finalizeImportBasedOnCache( const Reference< XDataPilotDescriptor >& rxDPDesc)
+{
+    /*  Process all fields based on source data, other fields (e.g. group
+        fields) are processed based on cache fields.*/
+    Reference< XDataPilotField > xDPField;
+    sal_Int32 nDatabaseIdx = mrPivotTable.getCacheDatabaseIndex( mnFieldIndex );
+    if( (nDatabaseIdx >= 0) && rxDPDesc.is() ) try
+    {
+        // Try to get the source field and its name from passed DataPilot descriptor
+        Reference< XIndexAccess > xDPFieldsIA( rxDPDesc->getDataPilotFields(), UNO_SET_THROW );
+        xDPField.set( xDPFieldsIA->getByIndex( nDatabaseIdx ), UNO_QUERY_THROW );
+        Reference< XNamed > xDPFieldName( xDPField, UNO_QUERY_THROW );
+        maDPFieldName = xDPFieldName->getName();
+        SAL_WARN_IF( maDPFieldName.isEmpty(), "sc.filter", "PivotTableField::finalizeImportBasedOnCache - no field name in source data found" );
+    }
+    catch( Exception& )
+    {
+    }
+
+    // Use group names already generated for another table using the same group field.
+    if( const PivotCacheField* pCacheField = mrPivotTable.getCacheField( mnFieldIndex ) )
+    {
+        if(!pCacheField->getFinalGroupName().isEmpty())
+            maDPFieldName = pCacheField->getFinalGroupName();
+    }
+}
+
 void PivotTableField::convertRowField()
 {
     convertRowColPageField( XML_axisRow );
@@ -1190,7 +1222,7 @@ void PivotTable::finalizeImport()
                 aDescProp.setProperty( PROP_DrillDownOnDoubleClick, maDefModel.mbEnableDrill );
 
                 // finalize all fields, this finds field names and creates grouping fields
-                maFields.forEachMem(&PivotTableField::finalizeImport, ::std::cref(mxDPDescriptor));
+                finalizeFieldsImport();
 
                 // all row fields
                 for( IndexVector::iterator aIt = maRowFields.begin(), aEnd = maRowFields.end(); aIt != aEnd; ++aIt )
@@ -1250,6 +1282,36 @@ void PivotTable::finalizeImport()
     }
 }
 
+void PivotTable::finalizeFieldsImport()
+{
+    if (maFields.empty())
+        return;
+
+    /* Check whether group fields are already imported for an other table
+       sharing the same groups. */
+    ScDPObject* pDPObj = getDPObject();
+    const ScDocument& rDoc = getDocImport().getDoc();
+    if (rDoc.HasPivotTable())
+    {
+        const ScDPCollection* pDPCollection = rDoc.GetDPCollection();
+        assert(pDPCollection != nullptr);
+        const ScDPDimensionSaveData* pGroups = nullptr;
+        bool bRefFound = pDPCollection->GetReferenceGroups(*pDPObj, &pGroups);
+        // Apply reference groups on this table.
+        if (bRefFound && pGroups && pGroups->HasGroupDimensions()) {
+            ScDPSaveData* pSaveData = pDPObj->GetSaveData();
+            if (pSaveData) {
+                pSaveData->SetDimensionData(pGroups);
+                pDPObj->ReloadGroupTableData();
+                maFields.forEachMem(&PivotTableField::finalizeImportBasedOnCache, ::std::cref(mxDPDescriptor));
+                return;
+            }
+
+        }
+    }
+    maFields.forEachMem(&PivotTableField::finalizeImport, ::std::cref(mxDPDescriptor));
+}
+
 void PivotTable::finalizeDateGroupingImport( const Reference< XDataPilotField >& rxBaseDPField, sal_Int32 nBaseFieldIdx )
 {
     // process all fields, there is no chaining information in the cache fields
@@ -1300,6 +1362,11 @@ Reference< XDataPilotField > PivotTable::getDataLayoutField() const
     return xDPField;
 }
 
+PivotCacheField* PivotTable::getCacheField( sal_Int32 nFieldIdx )
+{
+    return mpPivotCache ? mpPivotCache->getCacheField( nFieldIdx ) : nullptr;
+}
+
 const PivotCacheField* PivotTable::getCacheField( sal_Int32 nFieldIdx ) const
 {
     return mpPivotCache ? mpPivotCache->getCacheField( nFieldIdx ) : nullptr;