9. Monitoring

RED provides all control and status information in views and tables in the database. Even the status of RED-PSS can be monitored with SQL queries only, so it is not strictly necessary to monitor individual processes at operating system level. Detailed documentation of the individual columns can be found in the database itself as column comments.

RED exposes this monitoring data through the following interfaces (see the figure below):

REST API

Dedicated monitoring endpoints, including RED and Exasol database status data, see RED REST Monitoring Endpoints.

SQL API

Direct access to all control and status views in the database, see Monitoring Views.

syslog

The Exasol syslog information. Refer to the Exasol documentation for more information on monitoring.

RED Vision

The web GUI of RED for interactive monitoring.

Monitoring overview: Exasol views and the RED server feed customer monitoring via SQL API, REST API and syslog; RED Vision serves application and security managers RED Customer-specific Exasol RED status and monitoring views RED server REST API RED Vision Monitoring server e.g. Prometheus, Nagios, Icinga, CheckMK Dashboard Notification Application Manager Security Manager SQL REST API syslog SQL API RED Vision
Monitoring overview

9.1. RED REST Monitoring Endpoints

The monitoring endpoints are part of the REST API; their documentation (Swagger/OpenAPI) is available at https://<red-server>:<gu_port>/docs. Each request is authenticated with the application key of a REST user in the HTTP header app_key.

ENDPOINT

DESCRIPTION

/mon/adm_current_tversion

current technical version (TV) of the database

/mon/adm_current_bversion

current business version (BV)

/mon/adm_jmon_current_load

current number of active processes in the worker queues of the job monitor, the configured queue sizes, and the queue utilisation

/mon/adm_jmon_job_status

job name, job ID, execution time, and next execution time of the latest job executions started by the job monitor in all schemas

/mon/all_adm_job_status

job execution status with alert evaluation, see below

/mon/all_adm_watcher_exec_status

error log of watcher processes in all schemas, including the alert evaluation for execution retries

/mon/all_adm_watcher_status

alert evaluation of watcher processes in all schemas for the last processing

/mon/all_db_size_last_day

raw size, memory size, object count, and storage use every 30 minutes

/mon/all_monitor_last_day

CPU utilisation, disk read ratios, network traffic, swap ratio, etc. every 30 seconds

/mon/all_usage_last_day

number of logged-in users and queries every second

/mon/app_status

applications in alert state: schema, table name, minimum and maximum business date, number of affected days, alert code, and alert reason

/mon/check_python_vm

availability and latency of the Python VM (script language container)

/mon/db_backup_list

list of usable database backups

/mon/exa_dba_transaction_conflicts_last_day

number of transaction conflicts per minute and the maximum conflict duration in seconds

/mon/exa_license_check

Exasol licence check regarding expiration and raw data overflow (prognosis)

/mon/exa_loadavg

current CPU load of each database node

/mon/exa_statistics_object_sizes

sizes of all statistical system tables, aggregated by type

/mon/exa_volume_usage

usage of the storage volumes

/mon/jmon_status

log data of the latest job executions started by the job monitor in all schemas

/mon/object_status

objects in alert state: schema, table name, minimum and maximum business date, number of affected days, alert code, and alert reason

The alert level of /mon/all_adm_job_status depends on the return codes and on statistical runtime deviations:

  • 1 (warning): the execution was delayed between 30 and 60 seconds after the scheduled start time, or the runtime was 3 to 4 standard deviations above the average runtime.

  • 2 (error): an error occurred, the execution was delayed by more than 60 seconds, or the runtime was more than 4 standard deviations above the average runtime.

9.2. SQL API and Monitoring Views

Database objects whose name begins with STA_ contain condensed information about the status and configuration of the system. Many of them have the columns ALERT and ALERT_REASON, which indicate whether there is an immediate need for action. The information is prepared for external monitoring tools as well as for direct queries. The alert levels are:

  • 0: OK

  • 1: warning

  • 2: error

The column ALERT_REASON describes the cause of a warning or an error.

Views starting with STA_ are granted to public, while all other views and tables are only granted to the Application Manager and the monitoring role.

All status values used in these views, together with their severity and description, are listed in the status catalogue REDSYS.VIS_STATUS_CODES.

9.2.1. Object Status (REDADM)

STA_OBJECT_STATUS

The status of each PSA object per BDomain and business date, see Object and Application Status.

STA_OBJECT_UNFINISHED_OVERVIEW

Overview of all objects that have not reached the status FINISHED, grouped by status and alert level, with the oldest and newest affected business date.

STA_PSA_OBJECT_LINEAGE

Table lineage from the source via the loader and TSA table to the PSA table.

9.2.2. TSA Loader Status (REDADM)

The following views show the status of all TSA loaders, i.e. of the DBI, CFI, and custom connectors:

STA_TSA_PENDING_LOADER

All pending TSA loaders (running and queued) with alert reason and source state (input waiting, scheduled, running, error).

STA_TSA_QUEUED_LOADER

TSA loaders that are queued for execution (status NOT SCHEDULED).

STA_TSA_ACTIVE_LOADER

TSA loaders in status SCHEDULED, RUNNING, or ERROR. The column RUNTIME shows the current runtime in seconds, RUNTIME_AVG the average runtime of the loader, and RUNTIME_STDDEV the standard deviation.

STA_TSA_LOADER_COUNT

Number of not scheduled, scheduled, running, and failed TSA loaders per connector (the column with the number of loaders in status ERROR is named FAILED).

The ALERT column of the active loaders switches to 1 if the loader has been in status SCHEDULED for more than 30 seconds or its runtime exceeds the average by more than 3 standard deviations, and to 2 at more than 60 seconds or 4 standard deviations.

9.2.3. PSA Loader Status (REDADM)

The following views show the status of the PSA loader processes:

STA_PSA_PENDING_LOADER

All pending PSA loaders (running and queued) with the number of unfulfilled dependencies (BLOCKING_COUNT), alert reason, and pipeline state (waiting, blocked, not scheduled, scheduled, running, error).

STA_PSA_QUEUED_LOADER

PSA loaders that are queued for execution (status NOT SCHEDULED).

STA_PSA_ACTIVE_LOADER

PSA loaders in status SCHEDULED, RUNNING, or ERROR, with runtime statistics and alert evaluation as for the TSA loaders.

STA_PSA_LOADER_COUNT

Number of not scheduled, scheduled, running, and failed PSA loaders.

STA_PSA_PIPELINE_STATUS and STA_PSA_RUN_LINEAGE

For an object, BDomain, and business date: the expected deliveries and the state of each layer, and the executed chain of runs from the source via TSA to PSA.

9.2.4. Application Status (REDADM)

STA_APP_STATUS

Consolidated status of the applications: loaded, preliminary, final, data quality, and materialisation status, as well as the number of components that are waiting, loaded, and finished. See Applications.

STA_APP_OBJECT_STATUS

The object status of all components of an application. The object status is taken from STA_OBJECT_STATUS, the column LAST_CHANGE from LAST_PSA_FINISHED. Loads that are not planned by a load cycle are ignored, since they are not relevant for the application status.

STA_APP_UNFINISHED_OVERVIEW

Unfinished applications by alert level, with the oldest and newest affected business date.

STA_APP_ACTION_STATUS

Status of each application action per business date (latest run only).

STA_APP_ACTION_DETAIL

Details of each action run (full history).

STA_APP_ACTION_UNFINISHED_OVERVIEW

All pending or running application actions.

9.2.5. Jobs and Watchers (REDSYS)

STA_JOB_STATUS, STA_JOB_OVERVIEW, and STA_WATCHER_OVERVIEW

Status, errors, and alerts of the jobs and watchers of all schemas. If no job has been started for a longer time, RED-PSS (the job monitor) is most likely stopped.

9.2.6. Connector Logs

Detailed log information about the data import from the source into the TSA layer can be found in the views CFIADM.CFI_IMPLOG_DETAILS and DBIADM.DBI_CDC_LOG_DETAILS. An aggregated status is available in CFIADM.CFI_IMP_STATUS and DBIADM.DBI_CDC_STATUS.

9.3. Licences and Integrity

RED monitors its own licence, the Exasol licence, and the integrity of the framework objects:

RED licence

REDSYS.STA_RED_LICENSE_INFO shows the validity, the days until expiration, and the contracted limits of the RED licence. The licence is verified by the job monitor at start-up, daily, and on resume. If the verification fails, the job monitor suspends all DB jobs. Where to place the licence file is described in the RED-PSS Guide.

Exasol licence and capacity

EXASYS.STA_EXA_LICENSE_INFO projects the growth of the database volume with linear regressions over 5, 31, and 365 days and raises an alert before the licensed raw data size is reached.

Object integrity

As part of the licence verification, RED checks that the framework objects in the database match the delivered technical version. Deviations, e.g. manually changed framework scripts, make the verification fail.

On-demand check

redctl check checks the licence and the framework objects at any time and lists changed or missing framework objects. It is read-only: it does not suspend any jobs (see the RED-PSS Guide).

Licence status and storage prognosis are also shown in RED Vision.