Analytics - Caching Query & Search Results for Large Datasets
Available from version 0.4.1
Setting up Data Caching for Long Running Queries or Large Datasets
When setting up direct query data sources using SuperSync Analytics, the length of time to execute larger searches or the amount of data returned can become problematic.
For large datasets, NetSuite can sometimes return a server side error as too many rows are being held in memory. This usually just shows up as an error code 500, internal error. This error can often be avoided by setting a page length on the data source configurations.
Long running queries which time out however, may need a different approach. Pre-generating search or query results in NetSuite using a scheduled script and caching the results will handle this as the data is simply being served when the direct link is called, the time to run the search or query is removed from the equation.
Scheduling Cached Data Scripts
In the data source setups page, a "Cashed Result" option can be chosen as a data sync destination instead of sending the data to Sharepoint or Azure Data Store.
After this value is set, when viewing the data source record you will have the options to Schedule or Run an upload. When running the the first time you will only have the option to "Schedule Upload" and will be prompted to schedule a script file to run in the background which will generate the files.
If the Data Sync Destination is set to Cached Result, SuperSync Analytics will NOT attempt to run the query real time but will look to stored files in the file cabinet and return the result, if the background script has not yet run for the first time, it will be unable to return any data.
Caching results ahead of time significantly decreases the time taken for direct query links to run and mitigates errors related to the size of the result set.
As of version 0.4.1, only JSON format is supported for Cached Results.