Improvements

The improvements in pdrLoader version v7.6.1 release include some enhancements and defect fixes from the previous release.

Functional enhancements

The functional enhancements in pdrLoader v7.6.1 include:

  • File purge configuration now supports additional mask_mode logic.

  • New data source "internal" can be defined to use an alternate persistent data store (e.g. cloud blob storage). The default is a local filesystem
  • Performance and logging files for pdrMonitor are now prefixed with pdrBatcher to ensure uniqueness
  • Now automatically restarts worker processing threads if they are detected as dead
  • Add functions to DATA_FILTER row filter "matches", "exists", "!exists"
  • New optional property to customise PDR_* configuration in memory cache timeout in seconds.
  • New optional property to customise activity timeout on reconciliation in minutes
  • New optional property to allow a delay to be introduced to the reconciliation process.
  • Now supports loading of flat CSV files without column headings, the column headings are implied by the database table

  • Add support for managing configuration of non data model files configuration

  • Added sparse data row filter

  • LastChanged row filter now supports operator in PDR_REPORT_RECORDS.ROW_FILTER_PARAM2. Default is "<"

  • Now supports file handler lookup based on filename as well as extension, e.g. .gas.csv is equivalent to .gascsv

  • Add native support for Market JSON (IDX). Filename mask for this processor is *.market.json

  • Enhance processing of plain JSON file to handle either nest object or array under "data" element

  • Added EXTRACTED_AT row filter for supporting load of public WEM data

  • Added generic XML file processor loading capabilities

  • Enhancements to file process order to support PRIORITY and sort chaining definitions

  • Added new column transform DATABASE_SEQUENCE, DEFAULT_VALUE, DEFAULT_VALUE_SQL

  • Now populates PDR_REPORT_AUDIT.FILE_PROCESS_START column when a report starts processing

  • Now supports inactivity timeout on loader internal data source

  • Add functionality to gracefully handle primary key violations on concurrent threads

  • Added row filters COLUMNS_MATCH and COLUMNS_NO_NOT_MATCH

Bug fixes

The bug fixes in pdrLoader v7.6.1 include:

  • correct defect with process_priority and file bump operations - ensure resumption of file load starts from where file load was previously paused

  • Defect fix for purging manifest request rows from PDR_REQUEST_LOG for mySQL

  • Defect fix for DELETE transactions

  • Correct defect with error handling with loss of database connection after file process but before audit table update

  • Correct issue with reconciliation which could result in unnecessary report re-requests

  • Add PDR_DATA_FILTER_CONFIG to support dump file

Security Upgrades

The security upgrades in pdrLoader v7.6.1 include:

  • Added capability for configuring SSL encrypted application web services.

  • Security upgrades to numerous 3rd party libraries