Implementation Instructions

Depending on your current version, see the following sections for upgrade steps:

Full install

To install pdrBatcher v7.6.1 as a new implementation, select the following file from the Data Interchange Software:

File

Filename

Benefit

Participant Data Replication Batcher GUI Installer v7.6.1

Participant Data Replication Batcher GUI Installer v7.6.1.zip

Requires the minimum data entry for localisation.

All implementation steps are automatic.

A manual installation is possible by exiting the installer and using a text editor to set the configuration properties.

Upgrading from v7.6.x to v7.6.1.3

Improvements

The pdrBatcher v7.6.1.3 patch release includes a single defect fix from the previous release. This patch corrects an issue when consuming data from nemweb.com.au by supporting case-sensitive URL references. This is a new requirement for a planned future change to the underlying hosting platform supporting the AEMO NEMWeb service. See NEMWeb Platform Migration.

If this patch release is relevant to you, ensure you are using the https protocol for consuming data from nemweb.com.au. See Decommissioning HTTP Support.

You do not need to implement this patch release if your pdrBatcher installation does not consume data from nemweb.com.au.

Supportability

The pdrBatcher v7.6.1.3 patch is supported for pdrBatcher v7.6.0 (Windows users) and pdrBatcher v7.6.1 (Linux users).

Windows users currently running v7.6.0, do not need to upgrade to v7.6.1 before applying the patch.

Properties file updates

No properties are introduced for v7.6.1.3.

Properties file deletions

No properties are made obsolete for v7.6.1.3.

Manual installation process

Follow the steps below to manually upgrade an existing Participant Data Replication Batcher Software v7.6.x to v7.6.1.3 installation:

  1. Confirm you are running pdrBatcher v7.6.0 (Windows users) or pdrBatcher v7.6.1 (Linux users).
  2. Shutdown the pdrBatcher v7.6.x service.
  3. Backup your pdrBatcher v7.6.x installation folder.
  4. Replace the AppPdrBatcher.jar in your local pdrBatcher lib folder by copying the patched AppPdrBatcher.jar from the pdrBatcher v7.6.1.3 Releases folder.
  5. Check the timestamp of the AppPdrBatcher.jar to ensure it is the latest one.
  6. Check pdrBatcher.properties and validate the NEMWeb data source is using the https protocol. A NEMWeb data source is identified by a configuration operating in “web” mode. For example, if the data sourced is named “NEMWeb”, update its configuration block so its referenced URL uses the https protocol.

    7.   # define "NEMWeb" data source

      batcher_NEMWeb_mode=web

      batcher_NEMWeb_url=https://nemweb.com.au

      batcher_NEMWeb_timeout_connection=10

      batcher_NEMWeb_timeout_transfer=120

      batcher_NEMWeb_proxy_host=nemproxy

      batcher_NEMWeb_proxy_port=8080

      batcher_NEMWeb_dir=/Reports/Current

  7. Ensure all NEMWeb directory references in pdrBatcher.properties match the exact case as displayed in the folder names within the NEMWeb directory listing. For example:

    8.   # define "NEMWeb" data source

      batcher_NEMWeb_mode=web

      batcher_NEMWeb_url=https://nemweb.com.au

      batcher_NEMWeb_timeout_connection=10

      batcher_NEMWeb_timeout_transfer=120

      batcher_NEMWeb_proxy_host=nemproxy

      batcher_NEMWeb_proxy_port=8080

      batcher_NEMWeb_dir= /Reports/CURRENT

  8. Ensure all batcher thread folder references in pdrBatcher.properties use the exact case as listed in the NEMWeb directory listing. For example:

    9.  # Thread 1 : EMMS high priority dispatch thread

     batcher_thread_1_source=NEMWeb

     batcher_thread_1_dest=local

     batcher_thread_1_inc=PUBLIC_*

     batcher_thread_1_timeout=120

     batcher_thread_1_polling_interval=15

     batcher_thread_1_pause_interval=0.5

     batcher_thread_1_activity_monitor=Y

     batcher_thread_1_mode=CONSUMER_CACHE

     batcher_thread_1_sub_dir= DispatchIS_Reports

     batcher_thread_1_sub_dir_dest=N

    Use the folder names exactly as displayed in the NEMWeb directory listing (the hyperlink entries). Do not rely on the case currently shown in the browser URL or the page header, as these may reflect the case entered in the request rather than the stored directory name.

  9. Start the pdrBatcher v7.6.1.3 service
  10. Check the pdr.log in the Log folder to confirm the correct version is running and to identify any errors. For help, see Participant Data Replication Batcher.

Upgrading from v7.5.0 to v7.6.1

The release of pdrBatcher v7.6.1 introduces a number of new optional configurations in the .properties file. For details, see Properties file updates

Properties file updates

For help configuring these properties, see Participant Data Replication Batcher User Guide.

Property

Detail

ftp_provider

FTP provider classes changed to package au.aemo.Common.FTP.*

batcher_{DS}_ftp_provider

Determines whether the batcher uses plain FTP or SFTP/FTPS. Default is FTP. Valid values are

For FTP connection (default) - au.com.aemo.Common.FTP.FtpConnectionIntEDT

For FTPS configuration          - au.com.aemo.Common.FTP.FtpConnectionIntEDT_FTPS

For SFTP configuration          - au.com.aemo.Common.FTP.FtpConnectionIntEDT_SFTP

file_purge_${config}_mask_mode

The logic to apply to file inclusion and exclusion masks.

Valid values are:

EXCLUSIVE:

     include mask (if defined) is evaluated first

     exclude mask is only evaluated where there is not a match on include mask

AND:

     both include and exclude masks must be satisfied

         Default value is EXCLUSIVE

batcher_thread_{N}_pause_interval

Now supports fractional seconds

batcher_thread_{N}_min_size_bytes

Allows the specification of a minimum file size filter on a processing thread

participant_identifier

The single specifier of the participant identifier which is referenced in various other properties definitions. Refer to the standard properties configuration file for how this is merged into other properties.

batcher_thread_{N}_mask_mode

The logic to apply to file inclusion and exclusion masks. Valid values are:

EXCLUSIVE:

  • include mask (if defined) is evaluated first
  • exclude mask is only evaluated where there is not a match on include mask

AND:

  • both include and exclude masks must be satisfied

Default value is EXCLUSIVE

batcher_thread_{N}_file_translator_1

Multiple Translators can be specified for a single processing thread. Example:

batcher_thread_{N}_file_translator_1=<insert translator specification as described above for csv files>

batcher_thread_{N}_file_translator_1_inc=*.csv

batcher_thread_{N}_file_translator_2=<insert translator specification as described above for zip files>

batcher_thread_{N}_file_translator_2_inc=*.zip

batcher_thread_{N}_file_translator_1_inc

batcher_thread_${N}_source_dir

Can now be defined as a comma delimited specification of source directories. This allows for many:1 and many:many data replication configurations

batcher_thread_${N}_source_dir

Can now be defined as a comma delimited specification of source directories. This allows for many:1 and many:many data replication configurations

batcher_{DS}_api_business_response_codes

A set of HTTP response codes for an API call which are a business response to a transaction.

These allow batcher to identify whether a business transaction has been processed.

Codes outside of this list are considered a technical error.

batcher_thread_{N}_source_file_mask

The filemask specification to extract variables out of incoming filenames

Example:    For a file of this name:

                                            PARTID_BID20240110_20240110102307.zip

Applying this configuration setting

                        (?<ParticipantID>[a-zA-Z0-9]+)_(?<BidIdentifier>[a-zA-Z0-9]+)_(?<CreationDate>[a-zA-Z0-9]+).(?<Extension>[a-zA-Z0-9]+)

Results in the following variables:

                                            ParticipantID=PARTID

                                            BidIdentifier=BID20240110

                                            CreationDate=20240110102307

                                            Extension=zip

These variables can be used in RENAME translator

batcher_thread_{N}_source_api_header_values

The header values to add to a HTTP request. Relevant to API only. Example:

                            X-Market:NEM,X-initiatingParticipantID:{ParticipantID}

Adds the HTTP request headers X-Market and X-initiatingParticipantID, resolving the value of the latter

with a defined variable. Use in conjunction with batcher_thread_{N}_input_file_mask

batcher_thread_{N}_source_ack_api_header_values

Same as batcher_thread_{N}_source_api_header_values except applied when sending an acknowledement to the source system

batcher_thread_{N}_dest_file_mask

The filemask specification to output data at the destination

Only valid for GRAPHQL and API sources where there is not a native source filename

Example:    public_{event_id}_{date:yyyyMMddHHmmssSSS}.json

batcher_thread_{N}_dest_api_header_values

Same as batcher_thread_{N}_source_api_header_values except applied to destination

batcher_thread_{N}_source_ack_dir

The directory at the source connection to deliver a transaction acknowledgement

batcher_thread_{N}_source_ack_translator

Ability to perform a translation on an acknowledgment transaction back to source

Configuration option per batcher_thread_{N}_file_translator

batcher_thread_{N}_api_business_response_codes

A set of HTTP response codes for an API call which are a business response to a transaction.

These allow batcher to identify whether a business transaction has been processed.

Codes outside of this list are considered a technical error.

Overrides any specification of batcher_{DS}_api_business_response_codes at the data source level

client_https_keystore_file

The keystore file which contains x509 certificates to support SSL authentication where this application is consuming a SSL service

client_https_keystore_type

The type of keystore file. Default is PKCS12.

client_https_store_password

The key store password associated with KeyStoreFile. Defaults to encryption key stored in pdr_key.properties

client_https_key_password

The certificate password. Defaults to encryption key stored in pdr_key.properties

client_https_protocol

The SSL protocol to use. Options include TLS, TLSv1.2. Default is TLS.

batcher_thread_{N}_process_order

The process order for files in a source directory.

Valid settings are:

                            NEWEST - based on modified date, most recent file first

                            OLDEST - based on modified date, oldest file first

                            NAME:parameter - based on the filename in alphabetical order, parameter can be either ASCENDING (default) or DESCENDING

                            NAME_REVERSE - based on the filename in reverse alphabetical order

                            PRIORITY:parameter - based on the priority extracted from the filename. Example file "nmidh_GLOPOOL_20240208102721001.zip" is high priority (H), denoted by last character of first filename segment

                                                    Supported characters are H - High, M - Medium, L - Low

                                                    parameter can be either ASCENDING or DESCENDING (default)

                            MODIFIED_DATE:parameter - based on modified date, parameter can be either ASCENDING (default) or DESCENDING

                            SIZE:parameter - based on file size, parameter can be either ASCENDING (default) or DESCENDING

                            NAME_FILTER:parameter - based on filename, parameter is a comma delimited list of filemasks

                                                e.g. NAME_FILTER:*_DISPATCHIS_2*,*_P5MIN_*

                                        would result in files being processed in the order of DISPATCHIS, P5MIN, then everything else.

Process order can be chained, so for example:

                            PRIORITY | MODIFIED_DATE

will result in the process order being sorted first by priority (high then medium then low) and then by modified date within each priority level

Default setting is NAME:ASCENDING

batcher_{DS}_send_inactivity_timeout

The data inactivity timeout threshold in seconds within which data should be sent. If this threshold is breached then the connections associated with this data source are all reset

batcher_{DS}_receive_inactivity_timeout

The receive data inactivity timeout threshold in seconds within which data should be received. If this threshold is breached then the connections associated with this data source are all reset The type of keystore file. Default is PKCS12.

web_server_https_keystore_file

The keystore file which contains certificates to support SSL configuration of the web server

web_server_https_keystore_type

The type of keystore file. Default is PKCS12.

web_server_https_store_password

The key store password associated with KeyStoreFile. Defaults to encryption key stored in pdr_key.properties

web_server_https_key_password

The certificate password. Defaults to encryption key stored in pdr_key.properties

web_server_https_protocol

The SSL protocol to use. Options include TLS, TLSv1.2. Default is TLS.

Properties file deletions

No properties are made obsolete for v7.6.1.

GUI Installation process

The GUI installation process is a simple, clean way to upgrade. Alternatively, you can use the Implementation Instructions

To upgrade an existing Participant Data Replication Batcher Software v7.5.0 to v7.6.1 installation:

  1. Validate the JRE installation is Open JDK 21 by running the following in a command prompt:

    java -version

    The response should be similar to:

    openjdk version "21" 2023-09-19
    OpenJDK Runtime Environment (build 21+35-2513)
    OpenJDK 64-Bit Server VM (build 21+35-2513, mixed mode, sharing)

  2. If a different Java version is detected, refer to the documentation to install the certified JRE version.
  3. Validate the jar file signature. From the command line prompt, run the following command:

    jarsigner -verify "Participant Data Replication Batcher GUI Installer v7.6.1.jar"

    For more information, see https://docs.oracle.com/en/java/javase/21/docs/specs/man/jarsigner.html

  4. Start the GUI installer using one of the following methods:

    • Double-click the JAR file in a Windows environment, if *.jar files are associated with a Java Runtime Environment (JRE).
    • Run the installer from the command line:

      • Open a command prompt.
      • Navigate to the folder where the installer is stored.
      • Run:

        • java -jar <insert installer name here>.jar

    • For headless environments, such as Linux, run the installer in a terminal session using the -console flag:

      • java -jar <insert installer name here>.jar -console

      Provides the same capabilities as the GUI installer but runs entirely in text mode.

  5. Install the pdrBatcher v7.6.1 to the current installation folder (for example, C:\Pdr\Batcher).
    The current installation is backed up to a folder alongside the current installation with a timestamp appended to the folder name.
  6. If running as a Windows service, ensure the Windows service name entered into the GUI installer matches the existing service name.
  7. Check the pdrBatcher.properties file changes and re-apply any customised configurations. See Properties file updates and deletions Page 1.
  8. Run the pdrBatcher.bat or the Windows service and check the pdr.log in the Log folder for any errors.

  9. The GUI installer enables secure API services, by default, unless you deselect this installation option.

    You need to revalidate the application connection in the pdrMonitor after completing this upgrade process.

    You need pdrMonitor v1.3 to communicate to the pdrBatcher secured API services.

Manual Installation process

Follow the steps below to manually upgrade an existing Participant Data Replication Batcher Software v7.5.0 to v7.6.1 installation:

  1. Backup your pdrBatcher v7.5.0 installation folder.
  2. Remove the pdrBatcher v7.5.0 Windows service by running:

    pdrServiceUninstall.bat <insert_Windows_service_name_here>.

  3. Remove all existing jar files from the Lib folder
  4. Copy the following content from the pdrBatcher v7.6.1 installation media into your Lib folder
    • All jar files
    • New scripts:
      • pdrCertificateManager.*
      • pdrServiceInstall.bat
  5. Run the following command, after unzipping the command line prompt in the Lib folder:

    jarsigner -verify "AppPdrBatcher.jar"

  6. If not already, install JRE 21 and update the JRE path variable in:
    • pdrEnvironment.bat
    • pdrServiceInstall.bat
    • WinRun4J.ini files
  7. Add placeholder configuration to enable SSL encrypted web API services:
    • web_server_https_keystore_file=
    • web_server_https_keystore_type=PKCS12
    • web_server_https_protocol=TLS

    This capability can only be enabled once pdrMonitor has been upgraded to v1.3

  8. Consider aligning your pdrBatcher.properties to the standard configuration file unless you have specific customisation requirements. For participants with customisation requirements, it is recommended that these customisations be merged into a standard v7.6.1 properties file definition
  9. Install a new Windows service by running:

    pdrServiceInstall.bat <insert_Windows_service_name_here>.

  10. Check the NEMNET password is correctly encrypted before running the service by running the following command to avoid any password lockout:

    pdrConnectionTest.bat

  11. Run pdrBatcher.bat or the Windows service and check the pdr.log in the Log folder for any errors. For help, see Guide to Participant Data Replication Batcher.

Upgrading from v7.6.0 to v7.6.1 (Linux only)

These fixes are for Linux environments only.

If you are on Windows installation for 7.6.0, you do not have to upgrade to 7.6.1.

The v7.6.1 release of pdrBatcher fixes the following issues:

  • The shebang line of supplied shell scripts which causes scripts to fail unless “.” is in the PATH environment variable.
  • Creating self-signed certificates using pdrCertificateManager.sh when the Secured API Services installation option is selected

For any existing v7.6.0 installations on Linux please complete the upgrade to v7.6.1 with the following steps:

  1. Stop pdrBatcher service.
  2. Backup Lib folder from the installation directory.
  3. Extract the contents of the CLI release Participant Data Replication Batcher v7.6.1.zip to a working folder.
  4. Update Lib/AppPdrBatcher.jar in the installation folder with equivalent file from the new release.
  5. Update Lib/*.sh in the installation folder with equivalent file from the new release except pdrEnvironment.sh.
  6. To ensure the updated shell scripts are executable, execute the following commands in a terminal session in the Lib folder:
    • dos2unix *.sh
    • chmod + *.sh
  7. Restart pdrBatcher service.
  8. Review application logs and confirm correct version installed and service is operating correctly.

The upgrade in the v7.6.1 GUI installer is delivered as an increment on a v7.5.0 base and should not be used to complete this upgrade from a v7.6.0 base.