9. Monitoring¶
RED provides all control and status information in views and tables in the database. Even the status of RED-PSS can be monitored with SQL queries only, so it is not strictly necessary to monitor individual processes at operating system level. Detailed documentation of the individual columns can be found in the database itself as column comments.
RED exposes this monitoring data through the following interfaces (see the figure below):
- REST API
Dedicated monitoring endpoints, including RED and Exasol database status data, see RED REST Monitoring Endpoints.
- SQL API
Direct access to all control and status views in the database, see Monitoring Views.
- syslog
The Exasol syslog information. Refer to the Exasol documentation for more information on monitoring.
- RED Vision
The web GUI of RED for interactive monitoring.
9.1. RED REST Monitoring Endpoints¶
The monitoring endpoints are part of the REST API; their documentation (Swagger/OpenAPI) is available at https://<red-server>:<gu_port>/docs. Each request is authenticated with the application key of a REST user in the HTTP header app_key.
ENDPOINT |
DESCRIPTION |
|---|---|
/mon/adm_current_tversion |
current technical version (TV) of the database |
/mon/adm_current_bversion |
current business version (BV) |
/mon/adm_jmon_current_load |
current number of active processes in the worker queues of the job monitor, the configured queue sizes, and the queue utilisation |
/mon/adm_jmon_job_status |
job name, job ID, execution time, and next execution time of the latest job executions started by the job monitor in all schemas |
/mon/all_adm_job_status |
job execution status with alert evaluation, see below |
/mon/all_adm_watcher_exec_status |
error log of watcher processes in all schemas, including the alert evaluation for execution retries |
/mon/all_adm_watcher_status |
alert evaluation of watcher processes in all schemas for the last processing |
/mon/all_db_size_last_day |
raw size, memory size, object count, and storage use every 30 minutes |
/mon/all_monitor_last_day |
CPU utilisation, disk read ratios, network traffic, swap ratio, etc. every 30 seconds |
/mon/all_usage_last_day |
number of logged-in users and queries every second |
/mon/app_status |
applications in alert state: schema, table name, minimum and maximum business date, number of affected days, alert code, and alert reason |
/mon/check_python_vm |
availability and latency of the Python VM (script language container) |
/mon/db_backup_list |
list of usable database backups |
/mon/exa_dba_transaction_conflicts_last_day |
number of transaction conflicts per minute and the maximum conflict duration in seconds |
/mon/exa_license_check |
Exasol licence check regarding expiration and raw data overflow (prognosis) |
/mon/exa_loadavg |
current CPU load of each database node |
/mon/exa_statistics_object_sizes |
sizes of all statistical system tables, aggregated by type |
/mon/exa_volume_usage |
usage of the storage volumes |
/mon/jmon_status |
log data of the latest job executions started by the job monitor in all schemas |
/mon/object_status |
objects in alert state: schema, table name, minimum and maximum business date, number of affected days, alert code, and alert reason |
The alert level of /mon/all_adm_job_status depends on the return codes and on statistical runtime deviations:
1 (warning): the execution was delayed between 30 and 60 seconds after the scheduled start time, or the runtime was 3 to 4 standard deviations above the average runtime.
2 (error): an error occurred, the execution was delayed by more than 60 seconds, or the runtime was more than 4 standard deviations above the average runtime.
9.2. SQL API and Monitoring Views¶
Database objects whose name begins with STA_ contain condensed information about the status and configuration of the system.
Many of them have the columns ALERT and ALERT_REASON, which indicate whether there is an immediate need for action.
The information is prepared for external monitoring tools as well as for direct queries.
The alert levels are:
0: OK
1: warning
2: error
The column ALERT_REASON describes the cause of a warning or an error.
Views starting with STA_ are granted to public, while all other views and tables are only granted to the Application Manager and the monitoring role.
All status values used in these views, together with their severity and description, are listed in the status catalogue REDSYS.VIS_STATUS_CODES.
9.2.1. Object Status (REDADM)¶
STA_OBJECT_STATUSThe status of each PSA object per BDomain and business date, see Object and Application Status.
STA_OBJECT_UNFINISHED_OVERVIEWOverview of all objects that have not reached the status
FINISHED, grouped by status and alert level, with the oldest and newest affected business date.STA_PSA_OBJECT_LINEAGETable lineage from the source via the loader and TSA table to the PSA table.
9.2.2. TSA Loader Status (REDADM)¶
The following views show the status of all TSA loaders, i.e. of the DBI, CFI, and custom connectors:
STA_TSA_PENDING_LOADERAll pending TSA loaders (running and queued) with alert reason and source state (input waiting, scheduled, running, error).
STA_TSA_QUEUED_LOADERTSA loaders that are queued for execution (status
NOT SCHEDULED).STA_TSA_ACTIVE_LOADERTSA loaders in status
SCHEDULED,RUNNING, orERROR. The columnRUNTIMEshows the current runtime in seconds,RUNTIME_AVGthe average runtime of the loader, andRUNTIME_STDDEVthe standard deviation.STA_TSA_LOADER_COUNTNumber of not scheduled, scheduled, running, and failed TSA loaders per connector (the column with the number of loaders in status
ERRORis namedFAILED).
The ALERT column of the active loaders switches to 1 if the loader has been in status SCHEDULED for more than 30 seconds or its runtime exceeds the average by more than 3 standard deviations, and to 2 at more than 60 seconds or 4 standard deviations.
9.2.3. PSA Loader Status (REDADM)¶
The following views show the status of the PSA loader processes:
STA_PSA_PENDING_LOADERAll pending PSA loaders (running and queued) with the number of unfulfilled dependencies (
BLOCKING_COUNT), alert reason, and pipeline state (waiting, blocked, not scheduled, scheduled, running, error).STA_PSA_QUEUED_LOADERPSA loaders that are queued for execution (status
NOT SCHEDULED).STA_PSA_ACTIVE_LOADERPSA loaders in status
SCHEDULED,RUNNING, orERROR, with runtime statistics and alert evaluation as for the TSA loaders.STA_PSA_LOADER_COUNTNumber of not scheduled, scheduled, running, and failed PSA loaders.
STA_PSA_PIPELINE_STATUSandSTA_PSA_RUN_LINEAGEFor an object, BDomain, and business date: the expected deliveries and the state of each layer, and the executed chain of runs from the source via TSA to PSA.
9.2.4. Application Status (REDADM)¶
STA_APP_STATUSConsolidated status of the applications: loaded, preliminary, final, data quality, and materialisation status, as well as the number of components that are waiting, loaded, and finished. See Applications.
STA_APP_OBJECT_STATUSThe object status of all components of an application. The object status is taken from
STA_OBJECT_STATUS, the columnLAST_CHANGEfromLAST_PSA_FINISHED. Loads that are not planned by a load cycle are ignored, since they are not relevant for the application status.STA_APP_UNFINISHED_OVERVIEWUnfinished applications by alert level, with the oldest and newest affected business date.
STA_APP_ACTION_STATUSStatus of each application action per business date (latest run only).
STA_APP_ACTION_DETAILDetails of each action run (full history).
STA_APP_ACTION_UNFINISHED_OVERVIEWAll pending or running application actions.
9.2.5. Jobs and Watchers (REDSYS)¶
STA_JOB_STATUS,STA_JOB_OVERVIEW, andSTA_WATCHER_OVERVIEWStatus, errors, and alerts of the jobs and watchers of all schemas. If no job has been started for a longer time, RED-PSS (the job monitor) is most likely stopped.
9.2.6. Connector Logs¶
Detailed log information about the data import from the source into the TSA layer can be found in the views CFIADM.CFI_IMPLOG_DETAILS and DBIADM.DBI_CDC_LOG_DETAILS. An aggregated status is available in CFIADM.CFI_IMP_STATUS and DBIADM.DBI_CDC_STATUS.
9.3. Licences and Integrity¶
RED monitors its own licence, the Exasol licence, and the integrity of the framework objects:
- RED licence
REDSYS.STA_RED_LICENSE_INFOshows the validity, the days until expiration, and the contracted limits of the RED licence. The licence is verified by the job monitor at start-up, daily, and on resume. If the verification fails, the job monitor suspends all DB jobs. Where to place the licence file is described in the RED-PSS Guide.- Exasol licence and capacity
EXASYS.STA_EXA_LICENSE_INFOprojects the growth of the database volume with linear regressions over 5, 31, and 365 days and raises an alert before the licensed raw data size is reached.- Object integrity
As part of the licence verification, RED checks that the framework objects in the database match the delivered technical version. Deviations, e.g. manually changed framework scripts, make the verification fail.
- On-demand check
redctl checkchecks the licence and the framework objects at any time and lists changed or missing framework objects. It is read-only: it does not suspend any jobs (see the RED-PSS Guide).
Licence status and storage prognosis are also shown in RED Vision.