Improvements v8.0

The latest version of the pdrBatcher is now referenced as the AEMO Gateway Software v8.0.0 release, which is an upgrade from the pdrBatcher v7.6.0 installation.

Functional enhancements

The functional enhancements in AEMO Gateway Software v8.0.0 release include:

  • Encrypting variables in the properties file using ${encrypted:XXXX} syntax.
    batcher_{DS}_password_encrypted is discontinued. Replaced by ${encrypted:xxx} on batcher_{DS}_password.

  • Support for OAuth2 authentication.
    Change to data source configuration for Google cloud to reference OAuth2 framework.

  • Renaming application package from au.com.nemmco.* to au.com.aemo.*.

  • Each thread now using its own subfolder within the holding directory.

  • SYNCHRONISE mode now supporting delete operations at destination using configuration.
    New configuration properties:

  • batcher_thread_{N}_mode_param1

  • Property renaming batcher_thread_{N}_source_ack_translator to batcher_thread_{N}_ack_translator.

  • Adding extensibility documentation and examples.

  • Extensibility methods always execute in a daemon thread to reduce risk of custom code impacting overall performance.

  • Deprecated property:

  • batcher_thread_{N}_max_size_kB
    replaced with
  • batcher_thread_{N}_max_size_bytes

  • File sizes can now be entered in human readable format.

  • Azure blob supporting OAuth2 client credentials against EntraId.

  • pdrConnectionTest having command line parameters --configuration and --sources allowing the specific data source to test.

  • Adding SFTP and FTPS as data source providers to make these service types easier to configure. The system continues to support the legacy configuration of SFTP/FTPS under FTP.

  • Adding DEBUG log capability for transform configuration. New property batcher_translator_debug_dir and new --debug parameter for transform test CLI tool.

  • An example for configuring structured logging for JSON output.

  • Adding a feature for archiving transactions. New properties:

  • batcher_transaction_archive_data_source
  • batcher_transaction_archive_root_dir
  • batcher_transaction_archive_data_dir
  • batcher_thread_{N}_transaction_archive
  • batcher_thread_{N}_transaction_archive_data_dir

  • Adding a stop file status API for monitor reporting.

  • Adding parameters to control thread check and reporting interval:

  • batcher_thread_check_interval
  • batcher_thread_report_interval

  • Adding a command line tool for testing a sample file for validation against registered schemas.

  • Setting content-type based on filename when uploading to cloud blob.

  • Windows service wrapper migrating from WinRun4J (legacy) to Apache Procrun.

  • Various new file translation functionalities added:

  • XML_VARIABLE: Creates a variable based on an XSLT path specification
  • ALIAS_VARIABLE: Creates a variable based on an expression
  • JSON_TO_XML: Converts JSON to XML
  • XML_TO_JSON: Converts XML to JSON
  • SIGNATURE: Creates or verifies a digital signature

Bug fixes

The bug fixes in AGS includes:

  • File translators are now loaded in a predictable order, ensuring it applies the correct filemask.

  • Source and destination connections are now tested if there is an error in processing a file.

Security upgrades

Security upgrades to numerous third-party libraries.

Upgrading AEMO Gateway

Refer to the pdrBatcher/AEMO Gateway Software implementation instructions for details on how to perform an upgrade install.