Implementation Instructions

Depending on your current version, see the following sections for upgrade steps:

Full Install

To install pdrLoader v7.6.2 as a new implementation, you can choose one of the following files from the Data Interchange Software:

File

File name

Benefit

Participant Data Replication Loader GUI Installer v7.6.2

Participant Data Replication Loader GUI Installer for <Database> v7.6.2.zip

Requires the minimum data entry for localisation

All implementation steps are automatic

A manual installation is possible by exiting the installer and using a text editor to set the configuration properties

Upgrading from pdrLoader v7.5.0 to v7.6.2

A successful upgrade involves understanding the changes between the versions. See Improvements .

The v7.6.2 release has shipped with updated JDBC drivers across all database platforms. Please confirm the compatibility of the JDBC driver with your local database infrastructure.

A SSL encrypted connection to your local database would be considered best security practice, and participants wanting to configure this should discuss signed certificates across their infrastructure with their system administrator.

Participants who do not require encrypted communications can generally apply configuration to the JDBC connection string to indicate a non-encrypted communication channel.

Please refer to the OEM JDBC driver documentation for how to configure the connection string to your organisations requirements.

Properties file updates

The release of pdrLoader v7.6.2 introduces a number of new optional configurations in the .properties file.

For help configuring the application properties, see Guide to Participant Data Replication Loader.

Property

Detail

file_purge_${config}_mask_mode

The logic to apply to file inclusion and exclusion masks.

Valid values are:

EXCLUSIVE:

     include mask (if defined) is evaluated first

     exclude mask is only evaluated where there is not a match on include mask

AND:

     both include and exclude masks must be satisfied

         Default value is EXCLUSIVE

db_handler

Refactoring of database handler classes to a new package structure au.com.aemo.Common.Database.*

loader_data_sources

This property allows for the configuration of the persistance layer for stateful data

There is only a single option supported:

    internal

which directs the loader application to process the connectivity parameters for the "internal" data source

Refer to examples at the bottom of the properties file which illustrate how to configure cloud storage for Azure, AWS and Google clouds

The definition of the following properties will be required to be adjusted to point to a pre-created container on the cloud filesystem

    file_reports_dir

    file_archive_dir

    file_log_dir

    file_request_dir

    file_trickle_reports_dir

    file_error_dir

    performance_overflow_dir

If the loader_data_sources property is not defined, then a default persistance layer of the local filesystem is used and the above properties will point to local folders which are created when the loader is installed.

loader_internal_mode

Persistance layer for interaction with source content. Valid modes are:

    local - for a locally mounted filesystem

    aws - for Amazon S3 blob storage

    azure - For Azure blob storage

    google - For Google blob storage

file_queue_dir

Local directory to place reports to be loaded that are already queued.

This configuration is now mandatory

config_cache_timeout

The timeout for the configuration cache sourced from PDR_* management tables. Defaults to 120 seconds.

file_load_automated_recovery_inactivity_timeout

Inactivity timeout in minutes. If the reconciliation process does not execute within this timeout period, then a warning message is logged. Default value is (2 * file_load_automated_recovery_interval)

file_load_automated_recovery_recon_delay_mins

The reconciliation delay in minutes from current date time. This allows for a delayed assessment of whether a report is determined as missing. Rows will remain in PDR_MANIFEST_LOG in the RECONCILE state until this time delay passes.

file_load_thread_{N}_process_order

The process order for files in a source directory.

Valid settings are:

    NEWEST - based on modified date, most recent file first

    OLDEST - based on modified date, oldest file first

    NAME:parameter - based on the filename in alphabetical order, parameter can be either ASCENDING (default) or DESCENDING

    NAME_REVERSE - based on the filename in reverse alphabetical order

    PRIORITY:parameter - based on the priority extracted from the filename. Example file "nmidh_GLOPOOL_20240208102721001.zip" is high priority (H), denoted by last character of first filename segment

            Supported characters are H - High, M - Medium, L - Low

            parameter can be either ASCENDING or DESCENDING (default)

    MODIFIED_DATE:parameter - based on modified date, parameter can be either ASCENDING (default) or DESCENDING

    SIZE:parameter - based on file size, parameter can be either ASCENDING (default) or DESCENDING

    NAME_FILTER:parameter - based on filename, parameter is a comma delimited list of filemasks

                                                e.g. NAME_FILTER:*_DISPATCHIS_2*,*_P5MIN_*

        would result in files being processed in the order of DISPATCHIS, P5MIN, then everything else.

Process order can be chained, so for example:

    PRIORITY | MODIFIED_DATE

will result in the process order being sorted first by priority (high then medium then low) and then by modified date within each priority level

Default setting is NAME:ASCENDING

web_server_https_keystore_file

The keystore file which contains certificates to support SSL configuration of the web server

web_server_https_keystore_type

The type of keystore file. Default is PKCS12.

web_server_https_store_password

The key store password associated with KeyStoreFile. Defaults to encryption key stored in pdr_key.properties

web_server_https_key_password

The certificate password. Defaults to encryption key stored in pdr_key.properties

web_server_https_protocol

The SSL protocol to use. Options include TLS, TLSv1.2. Default is TLS.

 

Properties file deletions

Properties made obsolete in v7.6.2.

Property

Detail

file_load_threads

Is replaced by file_load_threads_active

GUI installation process

The GUI installation process is a simple, clean way to upgrade. Alternatively, you can use the Implementation Instructions

To install:

  1. Validate the JRE installation is Open JDK 21 by running the following in a command prompt:

    java -version

    The response should be similar to:

    openjdk version "21" 2023-09-19
    OpenJDK Runtime Environment (build 21+35-2513)
    OpenJDK 64-Bit Server VM (build 21+35-2513, mixed mode, sharing)

  2. If a different Java version is detected, refer to the documentation to install the certified JRE version.
  3. Validate the .JAR file signature. From the command line prompt, run the following command:

    jarsigner -verify "Participant Data Replication Loader GUI Installer v7.6.2.jar"

    For more information, see https://docs.oracle.com/en/java/javase/21/docs/specs/man/jarsigner.html

  4. Start the GUI installer using one of the following methods:

    • Double-click the JAR file in a Windows environment, if *.jar files are associated with a Java Runtime Environment (JRE).
    • Run the installer from the command line:

      • Open a command prompt.
      • Navigate to the folder where the installer is stored.
      • Run:

        • java -jar <insert installer name here>.jar

    • For headless environments, such as Linux, run the installer in a terminal session using the -console flag:

      • java -jar <insert installer name here>.jar -console

      Provides the same capabilities as the GUI installer but runs entirely in text mode.

  5. Install the pdrLoader v7.6.2 to the current installation directory (for example, C:\Pdr\Loader). The current installation is backed up to a folder alongside the current installation with a timestamp appended to the folder name.
  6. If running as a window service, then ensure that the windows service name entered into the GUI installer matches the existing service name.
  7. Check the pdrLoader.properties file changes and re-apply any customised configurations. SeeImplementation Instructions.
  8. Run the pdrLoader.bat or the Windows service and check the pdr.log in the Log directory for any errors.

  9. The GUI installer enables secure API services, by default, unless you deselect this installation option.

    You need to revalidate the application connection in the pdrMonitor after completing this upgrade process.

    You need pdrMonitor v1.3 to communicate to the pdrBatcher secured API services.

Manual installation process

The generic steps are:

  1. Backup your v7.5.0 pdrLoader installation folders.
  2. Remove the pdrLoader v7.5.0 Windows service by running:

    pdrServiceUninstall.bat <insert_Windows_service_name_here>.

  3. Remove all existing jar files from the Lib folder
  4. Copy the following content from the pdrLoader v7.6.2 installation media into your Lib folder
    • All jar files
    • New scripts:
      • pdrCertificateManager.*
      • pdrServiceInstall.bat
      • pdrKeyManager.*
  5. Run the following command, after unzipping the command line prompt in the Lib folder:

    jarsigner -verify "AppPdrLoader.jar"

  6. If not already, install JRE 21 and update the JRE path variable in:
    • pdrEnvironment.bat
    • pdrServiceInstall.bat
    • WinRun4J.ini files
  7. Define property file_queue_dir as follows:

    file_queue_dir=${file_reports_dir}/Queue

  8. Update the db_handler property to pint to the correct package. For example,

    db_handler=au.com.nemmco.Util.DbHandlerSqlServer

    becomes

    db_handler=au.com.aemo.Common.Database.DbHandlerSqlServer

  9. Add placeholder configuration to enable SSL encrypted web API services:
    • web_server_https_keystore_file=
    • web_server_https_keystore_type=PKCS12
    • web_server_https_protocol=TLS
    • This capability can only be enabled once pdrMonitor has been upgraded to v1.3
  10. Consider aligning your pdrLoader.properties to the standard configuration file unless you have specific customisation requirements. For participants with customisation requirements, it is recommended that these customisations be merged into a standard v7.5.0 properties file definition
  11. Apply the database update scripts applicable to your database platform in the following order:
    • <database_platform>_alter_pdr.sql
    • populate_parser_config_metadata_delta.sql
    • commit
  12. Install a new Windows service by running:

    pdrServiceInstall.bat <insert_Windows_service_name_here>.

  13. Check the database password is correctly encrypted before running the service by running the following command to avoid any password lockout:

    pdrConnectionTest.bat

  14. Run pdrLoader.bat or the Windows service and check the pdr.log in the Log directory for any errors. For help, see Guide to Participant Data Replication Loader.

Upgrading from v7.6.1 to v7.6.2 (Linux only)

These fixes are for Linux environments only.

If you are on Windows installation for 7.6.1, you do not have to upgrade to 7.6.2.

The v7.6.2 release of pdrLoader fixes the following issues:

  • The shebang line of supplied shell scripts which causes scripts to fail unless “.” is in the PATH environment variable.
  • Creating self-signed certificates using pdrCertificateManager.sh when the Secured API Services installation option is selected

For any existing v7.6.1 installations on Linux please complete the upgrade to v7.6.2 with the following steps:

  1. Stop pdrLoader service.
  2. Backup Lib folder from the installation directory.
  3. Extract the contents of the CLI release Participant Data Replication Loader v7.6.2.zip to a working folder.
  4. Update Lib/AppPdrLoader.jar in the installation folder with equivalent file from the new release.
  5. Update Lib/*.sh in the installation folder with equivalent file from the new release except pdrEnvironment.sh.
  6. To ensure the updated shell scripts are executable, execute the following commands in a terminal session in the Lib folder:
    • dos2unix *.sh
    • chmod + *.sh
  7. Restart pdrLoader service.
  8. Review application logs and confirm correct version installed and service is operating correctly.

The upgrade in the v7.6.2 GUI installer is delivered as an increment on a v7.5.0 base and should not be used to complete this upgrade from a v7.6.1 base.