1. Overview

The RED Process Stack (RED-PSS) is the automation stack for the RED Framework and runs on a dedicated RED server which is based on RHEL 8 (or compatible operating systems). The RED-PSS is completely self-contained which means that on top of a RHEL minimal installation all necessary libraries and tools are installed in /app independent of Python and Openssl versions on the machine itself.

RED-PSS is part of the RED Automation Module (see Automation Module)

2. RED-PSS Environment

2.1. Instances

The RED Server may contain multiple independent RED-PSS instances. Each instance controls one dedicated Exasol database with a corresponding RED Metadata Dictionary (RED-MDD). Different instances can have different versions.

2.2. Directory Structure

/app
├── exasol
│   ├── bin
│   ├── cli
│   ├── etc
│   └── lib
├── java
│   └── jdk-19.0.1
├── openssl
│   ├── 1.1.1s
│   └── conf
├── python
│   ├── 3.12.4
├── redftp
│   ├── bin
│   ├── env
│   │   └── <instance>
└── redsys
    ├── bin
    ├── env
    │   └── <instance>
    │      ├── bin
    │      ├── cmd
    │      ├── cust
    │      ├── data
    │      ├── etc
    │      ├── lib
    │      ├── log
    │      ├── src
    │      └── tmp
    ├── etc
    ├── lib
    ├── log
    └── tmp

2.2.1. redsys Directory Tree

This tree contains the complete installation of RED-PSS except for base libraries like Openssl or Python. All objects are owned by redsys user.

  • redsys/bin

  • redsys/etc

  • redsys/lib/

  • redsys/log

  • redsys/tmp: Tools and Logs for non-instance specific administration purposes

  • redsys/env: Contains a directory for each instance with exactly the same subtree and tools (depending on the technical version (TV).

  • redsys/env/<instance>/data: Landing zone for files

2.2.2. redftp Directory Tree

  • redftp/bin: Admin Tools

  • redftp/env/<instance>: Top level directory for FTP users for a specific instance owned by root

While FTP-users are named like redftp_<instance>_<user> the directory in the env-directory represents only the <user>-part.

2.2.3. Environment Variables

VARIABLE

DESCRIPTION

REDHOME

/app/redsys

REDENV

the current instance (upper case)

REDENV_HOME

$REDHOME/env/

2.3. Security

2.3.1. Application Management

For administration purposes the Application Manager must sudo to redsys and change into the target environment for the instance:

 [user@red01 ~]$ sudo su - redsys
 [redsys@red1 ~]$ tst1
 ---------------------------------
 current environment: TST1
 exit with CTRL-D
 ---------------------------------
 TST1 [redsys@red01 tst1]$ pwd
 
 /app/redsys/env/tst1
 TST1 [redsys@red01 tst1]$

2.3.2. sFTP Users

sFTP users reside in a change root environment (see 2.2.2) authenticated by public key. Uploaded data are made available for redsys via softlinks in $REDENV_HOME/data/input. redsys and the sFTP users share the same group redftp.

sFTP users are bound to the UID range 1200 – 1299.

In order to add or remove sFTP-users use the commands redftp_useradd or redftp_userdel, respectively. The commands may be executed as redsys or root.

2.3.3. REST API

The REST-API is available through https only. Certificate and Private Key must exist in $REDENV_HOME/etc/ssl/REST:

  • certificate.pem

  • private_key.pem

2.4. Start/Stop of RED-PSS

The complete stack is automatically started on machine startup by systemd (redsys.service). However, only instances are started where the (empty) file $REDENV_HOME/.autostart exists.

redsys.service calls $REDHOME/bin/redsysctl which may be used as well as redsys-User in order to start and stop all instances or just one.

DEV1 [redsys@red01 bin]$ redsysctl -h 
USAGE: 

Manage the current instance (uses REDENV): 
redsysctl {start | stop | status | restart} 

Manage a specific instance: 
redsysctl {start | stop | status | restart} 

instance Start/stop all instances: 
redsysctl {start-all | stop-all | restart-all | status-all} 

DEV1 [redsys@red01 bin]$

redsysctl in turn calls \(REDENV_HOME/bin/redctl and \)REDENV_HOME/bin/restctl.

2.5. Suspended Mode

redctl may be started in suspended mode by using the -S option. In this case no jobs and watchers will be started and, therefore, no load processes will be executed. All other services, however, are up and running.

The suspended mode can be ended with the resume command or by restarting redctl (restart command).

2.6. Cloning a DB into an existing RED-environment

Before starting the Exasol cloning process (i.e. restoring a backup from another DB) you must execute redctl preclone. By doing this, important data and metadata are backed up in $REDENV_HOME/tmp/preclone.

After the restore, redctl takes care of restoring these data to the clone and, furthermore, all logs (in the DB and on the filesystem) are cleaned up.

Note: The audit trail of the cloned DB will not be deleted by redctl. Only the SYS-user can do so.

3. Command Line Tools

3.1. redctl

redctl is the main commandline utility for the RED-PSS

$ redctl -h
usage: redctl [options] command [command-options]

options:
-h, --help show this help message and exit

commands:
{version,start,stop,abort,restart,start_trace,status,reload,suspend,
resume,show,watch,we_list_error,we_reset_error,start_db_job,list_db_
jobs,toggle_db_job,list_db_watcher,toggle_db_watcher,start_job,precl
one}

COMMAND

DESCRIPTION

version

show versions

start

start server

stop

stop server gracefully

abort

kill all server processes immediately

restart

restart server

start_trace

start server with output on STDOUT and STDERR

status

show server status

reload

reload job config

suspend

suspend all DB-jobs

resume

resume all DB-jobs

show

show current status of jobs and processes

watch

keep watching the status of jobs and processes

we_list_error

list errors of watcher executors

we_reset_error

reset errors of watcher executors (reschedule)

start_db_job

start a DB job immediately

list_db_jobs

list DB jobs

toggle_db_job

toggle DB job between active and inactive

list_db_watcher

list DB watcher

toggle_db_watcher

toggle DB watcher between active and inactive

start_job

start an already scheduled job immediately

preclone

save state before cloning from another instance

3.2. redinfo

See redinfo in the Concepts Guide.

3.3. redenv_info

Prints environment information of the current environment or for all environments on this machine.

3.4. restctl

restctl starts and stops the GUnicorn application server for the REST-API.

DEV1 [redsys@red01 bin]$ restctl -h
USAGE: restctl {start | stop | status | restart}
DEV1 [redsys@red01 bin]$

4. Process Architecture

4.1. JMON

The Job Monitor scans the views REDSYS.ALL_ADM_JOBS and REDSYS.ALL_ADM_WATCHER and schedules jobs and watchers, respectively, as defined therein. JMON monitors if they are running or scheduled and reschedules if necessary. Every 60 seconds JMON writes back statistics into REDSYS in the tables ADM_JMON_LOAD and ADM_JMON_JOBLOG.

When a job or watcher is scheduled JMON hands it over to the scheduler (SCHED) which executes the job then in a worker pool. JMON also recognizes if a job or watcher was activated or deactivated and sets the proper action. If the interval or cron attributes are changed redctl has to be restarted in order to bring the changes into effect.

4.2. SCHED

The scheduler master process starts workers for 3 worker pools:

  • L – loader queue (for all load processes)

  • A – action queue (for application actions)

  • D – the default queue for all jobs

Jobs and Watchers log information into ADM_JOB_LOG and ADM_WATCHER_LOG, respectively, of the target schema. Watcher executor errors are logged into ADM_WATCHER_EXEC_ERROR.

4.3. DBLS

The database log server receives log buffers from the DB and writes them into $REDENV_HOME/log/<target schema>. Errors and alerts are recorded in the database log-table (LOG_MESSAGES) as well.

4.4. JEXEC

The job executor runs external jobs as defined in an application action. Based on the action definition the proper script in $REDENV_HOME/cmd is executed.

4.5. SEXEC

The script executor runs LUA scripts as defined in an application action.

4.6. CFISRV

The CFI-server presents files in the data-tree to the RED framework in the DB. This server uses internally the FTPs protocol for communication with the DB. CFISRV is optional and can be deactivated.

4.7. JSON2CSV

The JSON-to-CSV process converts JSON input files into CSV format before they are recognized by CSVSRV. JSON2CSV is active only if CFISRV is active.

4.8. PARQ2CSV

The Parquet-to-CSV process converts Parquet input files into CSV format before they are recognized by CSVSRV. PARQ2CSV is active only if CFISRV is active.

4.9. SENDMAIL

The sendmail server sends buffers received from the DB to the defined mail relay.

4.10. SENDFILE

The sendfile server exports a CSV or Parquet file from a table or view in the DB and sends it via mail or sFTP to the defined destination.

4.11. MAIL2CMD

Mail-to-command interface. MAIL2CMD is optional and can be deactivated.

5. Configuration

The configuration of RED-PSS can be found in $REDENV_HOME/etc/red_params.conf.

5.1. [DEFAULT] Section

Default values for all other sections.

PARAMETER

DESCRIPTION

log_level

DEBUG, INFO, WARN, ERROR, ALERT

exa_connection

Exasol connect (DSN)

exa_dbname

DB name

5.2. [redctl] Section

PARAMETER

DESCRIPTION

service_base_port

base port for internal services

service_host

IP or hostname for internal services
0.0.0.0 or localhost will resolve to the default interface

sched_default_pool_size

number of workers in the default pool

sched_loader_pool_size

number of workers in the loader pool

sched_action_pool_size

number of workers in the action pool

sendmail_smtp_host

interface for the internal SMTP service
0.0.0.0 or localhost will resolve to the default interface

sendmail_smtp_port

port for the internal SMTP service

sendmail_smtp_user

SASL auth username for the internal SMTP service

sendmail_smtp_pass

SASL auth password for the internal SMTP service

sendmail_smtp_from

sender mail address for outgoing mails

sendmail_smtp_encr

transport encryption for outgoing mails: STARTTLS, PLAIN

cfi_enabled

enable or disable CFISRV: true/false

cfi_pasv_port_start

start of the passive port range

cfi_pasv_port_stop

end of the passive port range

json2csv_json_suffix

suffix of incoming JSON files

json2csv_csv_suffix

suffix of converted CSV files

json2csv_retention

retention (in days) of JSON input files

parq2csv_parq_suffix

suffix of incoming Parquet files

parq2csv_csv_suffix

suffix of converted CSV files

parq2csv_retention

retention (in days) of Parquet input files

mail2csv_enabled

enable or disable MAIL2CSV: true/false

mail2csv_host

interface for the internal mail2csv service
0.0.0.0 or localhost will resolve to the default interface

mail2csv_port

port for the internal mail2csv service

log_retention

log retention (in days) for all server, job and DB-logs

log_level

log level for all server processes

5.3. [redinfo] Section

PARAMETER

DESCRIPTION

log_level

log level for redinfo

5.4. [REST] Section

PARAMETER

DESCRIPTION

rest_enabled

start REST service on startup: true/false

gu_host

GUnicorn listener interface
0.0.0.0 or localhost will resolve to the default interface

gu_port

GUnicorn listener port

gu_workers

number of GUnicorn workers

log_level

log level for the REST service

6. Logging

All logs can be found in $REDENV_HOME/log.

6.1. Server Logs

The redctl and JMON logs go to redctl.log.

6.2. REST-API Logs

See REST subdirectory.

6.3. Database Logs

The top level directory for DB logs is the DB subdirectory. Within that subdirectories which are named like the respective target DB-schema can be found. ERROR and ALERT messages generated by DB-processes (i.e. LUA Scripts) are written into the respective LOG_MESSAGES tables as well.

All log messages of all schemas can be found in REDSYS.ALL_LOG_MESSAGES.

7. Monitoring Views

The status of RED-PSS can be queried directly from the DB.

  • ALL_ADM_JOBS

  • ALL_ADM_JOB_LOG

  • ALL_ADM_JOB_STATS

  • ALL_ADM_JOB_STATUS

  • ALL_ADM_WATCHER

  • ALL_ADM_WATCHER_EXEC_ERROR

  • ALL_ADM_WATCHER_EXEC_STATUS

  • ALL_ADM_WATCHER_LOG

  • ALL_ADM_WATCHER_STATUS

  • ALL_LOG_MESSAGES

  • STA_ADM_JOB_OVERVIEW

  • STA_ADM_WATCHER_OVERVIEW