3. Automation Module¶
The automation module includes control logic located on an external batch server (“RED-server”) interacting with the RED database repository. This module is an integral part of the core logic and mandatory prerequisite for every other module.
RED-PSS guide describes the process part on the RED-server.
3.1. Scheduling, Logging and Monitoring¶
Since the Exasol database comes without a scheduling mechanism, the RED framework adds all these features. There is no need for an external scheduler. Logging and monitoring are integral parts of the RED framework as well.
RED provides all control and status information in views and tables in the database. Even the status of the RED-PSS can be monitored in this way using SQL-queries only. Therefore, it is not strictly necessary to monitor individual processes on OS-level. Detailed documentation of individual columns in the referenced tables and views can be found directly in the database itself as column comments.
3.1.1. Monitoring Views¶
Database objects whose name begins with “STA_” contain condensed overview information about the status and configuration of the system. Some objects hold ALERT and ALERT_REASON columns that indicate whether there is an immediate need for action. The information is prepared for usage in external monitoring tools or direct user queries. The alert levels and their meaning are:
0…OK
1…Warning
2…Error
Furthermore the ALERT_REASON column might show a textual description, if a warning or an error occured.
Views starting with “STA_” are always granted to public, while all other views and tables are only granted to the application manager.
3.1.2. Object Status (REDADM)¶
see Object Status
3.1.3. TSA-Loader Status (REDADM)¶
The related views in REDADM show status information of all TSA-connectors in the system, i.e. DBI, CFI and custom connectors.
STA_TSA_QUEUED_LOADER: Shows the status of all TSA-loaders which are currently queued for execution (status: NOT SCHEDULED).
STA_TSA_ACTIVE_LOADER: Shows the status of all TSA-loaders which are currently in status SCHEDULED, RUNNING or FAILED. The column RUNTIME shows the actual runtime in seconds while RUNTIME_AVG shows the average runtime of this loader and RUNTIME_STDDEV shows the corresponding standard deviation of the sample.
The ALERT column switches to 1 if:
the loader has been more than 30s in status SCHEDULED
the RUNTIME > RUNTIME_AVG + 3 x RUNTIME_STDDEV (3-sigma rule)
The ALERT column switches to 2 if:
the loader has been more than 60s in status SCHEDULED
the RUNTIME > RUNTIME_AVG + 4 x RUNTIME_STDDEV (4-sigma rule)
STA_TSA_LOADER_COUNT: Gives an overview (total count) of all not scheduled, scheduled, running and failed TSA loaders in the framework.
3.1.4. PSA-Loader Status (REDADM)¶
The related views in REDADM show status information of currently active PSA-loader processes.
STA_PSA_QUEUED_LOADER: Shows the status of all PSA-loaders which are currently queued for execution (status: NOT SCHEDULED).
STA_PSA_ACTIVE_LOADER: Shows the status of all PSA-loaders which are currently in status SCHEDULED, RUNNING or FAILED. The column RUNTIME shows the actual runtime in seconds while RUNTIME_AVG shows the average runtime of this loader and RUNTIME_STDDEV shows the corresponding standard deviation of the sample.
The ALERT column switches to 1 if:
the loader has been more than 30s in status SCHEDULED
the RUNTIME > RUNTIME_AVG + 3 x RUNTIME_STDDEV (3-sigma rule)
The ALERT column switches to 2 if:
the loader has been more than 60s in status SCHEDULED
the RUNTIME > RUNTIME_AVG + 4 x RUNTIME_STDDEV (4-sigma rule)
STA_PSA_LOADER_COUNT: Gives an overview (total count) of all not scheduled, scheduled, running and failed PSA loaders in the framework.
STA_PSA_CONTROL_DETAIL: Shows the full history of each and every PSA loader process.
3.1.5. Application Status (REDADM)¶
STA_APP_OBJECT_STATUS: Shows the object status of all components of an application. The OBJECT_STATUS takes its values from STA_OBJECT_STATUS and LAST_CHANGE is taken from LAST_PSA_FINISH. Alert values are based on STA_OBJECT_STATUS as well. Note that this view ignores all loads which are not planned by a corresponding load cycle since they not relevant for an application status.
STA_APP_STATUS: Applications
RED_APP_ACT_STATUS: Shows the status application actions for each BDate (information about the last run only).
RED_APP_ACT_CONTROL_DETAIL: Shows detailed information of each action run (full history).
3.1.6. Connector Logs¶
For detailed log information regarding the data import from source to TSA check out the views CFIADM.CFI_IMPLOG_DETAILS and DBIADM.DBI_CDC_LOG_DETAILS, respectively. An aggregated status can be found in DBIADM.DBI_CDC_STATUS and CFIADM.CFI_IMP_STATUS, respectively.
3.1.7. redinfo¶
redinfo is a commandline tool which gives a (constantly refreshing) overview of the RED framework including watchers, jobs and active loaders as well as of the general status of the DB. It may be used for interactive monitoring.
3.2. Deployment¶
The RED framework tracks two vtypes of versions:
the technical version (TV): This is the version of the RED framework itself.
the business version (BV): This is the version of the metadata, i.e. the version of loaders, tables, application marts, tec.
The version history (TV & BV) can be seen in the ADM_VERSION_HISTORY view.
A business version must be opened before the application manager can do any change. When a BV should be propagated to a higher environment (test or production), the BV must be closed and all changes should be put to some source code versioning system like GIT.
3.2.1. Transaction Brackets¶
Changes within API calls usually are committed as soon as the call is finished successfully. If you run a script with multiple calls in sequence (as it happens during a deployment) each change would be committed immediately. If something goes wrong the previous changes cannot be rolled back. Transaction brackets guarantee an atomic transaction over all singe API calls hich means that an error occurring in the middle of the deployment triggers a rollback of the complete deployment.
Important
Only users with a private sandbox can use transaction brackets.
Start a transaction by issuing the following command:
1EXECUTE SCRIPT API_REDSYS.TRX_BEGIN();
The exceute all your API calls and in the end terminate the transaction by calling:
1EXECUTE SCRIPT API_REDSYS.TRX_COMMIT();
Note
TRX_COMMIT
may take some time because only at this point in time the changes are applied physically within a single database transaction.