1. RED TV Deployment Guide

This document describes the deployment, installation, upgrade, and patching processes for the Technical Version (TV) of the RED framework on target host environments. The TV represents the version of the RED framework itself, including its binary files, orchestrator utilities, system service configurations, and database schema structures.

The rollout process transitions the system from a prepackaged self-extracting installer (red_install_<version> or red_patch_<from_version>_to_<to_version>) up to the final execution of the database schema installer utility.

1.1. Overview of TV Deployment

The installation, upgrade, and patching of the TV is split into three sequential phases:

  1. Target Host Installation (Root Level): Deploying the self-extracting full installer executable (red_install_<version>) or self-extracting patch installer executable (red_patch_<from_version>_to_<to_version>, e.g. red_patch_2.0.0_to_2.0.1) on the target host to install system-wide configurations, systemd services, binary templates, or targeted hotfixes.

  2. Environment Instance Setup (Instance Level): Cloning the seed template to initialise a new environment instance, or upgrading an existing environment instance with redenv_upgrade to apply new or patched binaries and scripts.

  3. Database Schema Installation (Database Level): Executing the database schema setup (redtvctl install) to perform a fresh database installation, apply incremental database upgrades, or execute patch-specific schema updates.

1.2. Installation, Upgrade, and Patch Methods

The deployment process handles initial (fresh) full installations, incremental version upgrades, and targeted patch deployments. The system automatically detects the state of the target host and databases to apply the appropriate logic.

Deployment Stage

Fresh (Full) Installation

Incremental Upgrade

Patch Method

Delivery Artefact

Self-extracting full installer: red_install_<version>

Self-extracting full installer: red_install_<version>

Self-extracting patch installer: red_patch_<from_version>_to_<to_version> (e.g. red_patch_2.0.0_to_2.0.1)

Host Installation

Installs base binaries and host configurations from scratch. Sets global version in /etc/red.

Scans lib/upgrade/ and runs all incremental host upgrade scripts in sequence.

Applies targeted updates to framework binaries, libraries, and .seed templates, transitioning host version from <from_version> to <to_version>.

Instance Setup

Creates a new environment instance using redenv_create based on the .seed template.

Upgrades an existing environment instance using redenv_upgrade and applies instance-specific upgrade scripts.

Propagates patched binaries and scripts from the updated .seed template into existing instances via redenv_upgrade.

Database Schema

Initialises schemas, tables, and metadata repository from scratch via redtvctl install.

Executes incremental database SQL scripts (install.sql) sequentially via exaplus to upgrade the TV.

Executes patch SQL scripts and re-registers modified routines/views via redtvctl install if included in the patch.

1.3. Phase 1: Target Host Installation

The host installation is performed by the system administrator at the root level of the target server. It is recommended to perform this step directly as the root user (e.g. via sudo -i), which removes the need to prefix commands with sudo.

1.3.1. Executing the Full Host Installer

The self-extracting full installer script (typically named red_install_<version>) must be transferred to the target server (e.g. into the /tmp directory) and executed:

# Switch to root user
sudo -i

# Execute the full installer
/tmp/red_install_<version>

1.3.2. Executing the Self-Extracting Patch Installer

When applying a hotfix or targeted update between revisions, the administrator receives a self-extracting patch installer named according to the transition range: red_patch_<from_version>_to_<to_version> (for example, red_patch_2.0.0_to_2.0.1). Transfer the patch executable to the target server and execute it as root:

# Switch to root user
sudo -i

# Execute the patch installer
/tmp/red_patch_2.0.0_to_2.0.1

The patch installer verifies that the host environment has the exact <from_version> installed before extracting the updated binaries, libraries, and .seed templates to update the installation to <to_version>.

1.3.3. Host Installation Internal Process

When the installer runs (full installer or patch installer), it performs the following sequential actions:

  1. Version Verification:

    • The installer reads the current host version from /etc/red. If this file is missing, it assumes a default baseline version of 1.9.0.

    • For full installations and upgrades, it compares the current version with the target version packaged in the installer. If the current host version matches the target version, the installer exits safely without making changes. If the current host version is newer than the target version, the installation is aborted.

    • For patch installers (red_patch_<from_version>_to_<to_version>), it verifies that the host TV version matches <from_version> exactly before proceeding.

  2. Incremental Upgrade and Patch Script Execution:

    • The installer extracts its payload archive to a temporary directory.

    • In upgrade mode, it scans the extracted /app/redsys/lib/upgrade/ directory for upgrade subdirectories. These subdirectories are named after their target version (e.g. lib/upgrade/1.10.0/upgrade.sh). The installer identifies all upgrade scripts within the version range current_version < version <= target_version, sorts them in ascending order using natural version sorting (equivalent to sort -V), and executes each script sequentially.

    • In patch mode, it executes patch-specific host update scripts to update affected components without re-running full version migration sequences.

    • If any script returns a non-zero exit code, the installation aborts immediately.

  3. System Files Extraction:

    • For full installers, it extracts the core system directories (/app/redsys, /app/redftp, and /app/exasol) to the root / directory.

    • For patch installers, only the modified framework files, libraries, and .seed templates under /app/redsys are updated.

    • Specific user dotfiles (such as .bashrc and .bash_profile files under /app/redsys and /app/exasol) are explicitly excluded during extraction to prevent overwriting custom user modifications.

  4. Host Configuration Deployment:

    • The installer copies system-wide configuration files from /app/redsys/etc/root/ to the host system /etc directory. It updates file ownership to root:root and configures the following secure permissions:

      • Sudoers Rules: Copies files from /app/redsys/etc/root/etc/sudoers.d/ (such as 97-redapp and 98-red) to /etc/sudoers.d/ with 0440 permissions.

      • SSH Configurations: Copies files from /app/redsys/etc/root/etc/ssh/sshd_config.d/ (such as 98-red.conf) to /etc/ssh/sshd_config.d/ with 0600 permissions.

      • Systemd Unit Files: Copies /app/redsys/etc/root/etc/systemd/system/redsys.service to /etc/systemd/system/ with 0644 permissions.

  5. Systemd Service Registration & Version Update:

    • The installation automatically configures the systemctl service for the RED framework. It registers and enables the redsys.service systemd unit on the host to ensure the framework starts automatically on boot.

    • For full/upgrade installations, it updates the global version file /etc/red with the target version. For patch releases, it updates /etc/red to the target <to_version>.

  6. Exasol Language Container Deployment:

    • The installer inspects /app/exasol/etc/ for Exasol client configuration files (confd_*.cfg).

    • If exactly one configuration is found, it automatically executes clconfig as the exasol user to deploy and update the language containers. If multiple exist, the administrator is prompted to run the update manually.

1.3.4. System Users and Directory Structure

During a full (fresh) host installation, the installer creates dedicated system users with fixed UIDs and GIDs to isolate services and ensure consistent file ownership across the environment. The deployed users include:

  • redsys user: Owns the RED framework core installation, orchestration tools, and Python environments. (UID 1070, primary group redsys GID 1070, supplementary group redftp)

  • exasol user: Owns the complete Exasol client environment and the cluster automation framework. (UID 1050, primary group exasol GID 1050)

  • redftp group: Used to manage secure file transfers across environments. The redftp group uses GID 1200. Individual sFTP users are bound to the UID range 1200 – 1299 and share this group.

The installer extracts these components into a structured directory tree under /app:

  • /app/redsys/: Contains the RED framework executables, library seed templates, and environment instances.

  • /app/exasol/: Contains the Exasol client tools, connection libraries, and automation utilities.

  • /app/redftp/: Contains the sFTP environment directories where external data is ingested.

For a detailed breakdown of the internal RED directory tree, refer to the Directory Structure section in the RED-PSS Guide.

1.4. Phase 2: Environment Instance Setup

Once the host binaries and system-level configurations are deployed, the environment instances on the machine must be created, upgraded, or patched to incorporate the updates.

Important

Before running environment management commands, always switch to the redsys user context (e.g. via sudo -iu redsys). Since the redsys profile adds the management tools to the user’s PATH, absolute paths are not required.

Warning

Always run environment management commands from the host shell. Do not run these commands while inside an active environment subshell context (i.e. make sure you are not in a redenv context).

1.4.1. Upgrading or Patching an Existing Environment Instance

To update an existing environment instance following a version upgrade or a patch deployment, run the redenv_upgrade utility:

redenv_upgrade <env_name>

1.4.1.1. Existing Upgrade and Patch Process

The utility automates the upgrade or patch rollout for the target instance’s binary assets and scripts using the following flow:

  1. Safety Check: It verifies that the target environment is a compiled production environment and not a developer environment. It will refuse to upgrade a developer environment to prevent overwriting local uncompiled Python scripts or virtual environments.

  2. Process Stop: It stops all running processes inside the target environment instance using the redsysctl control tool.

  3. Upgrade and Patch Scripts:

    • In version upgrade mode, it runs the instance-level upgrade scripts (upgrade.sh) located under .seed/lib/upgrade/<version>/ for all versions greater than the instance’s current version up to the target version, in strictly increasing natural sequence.

    • In patch mode, it executes any patch migration scripts designated for the active TV version.

  4. Binary and Library Assets Update: It synchronises the instance’s bin/ and lib/ directories with the newly deployed .seed template directory, replacing updated executables and shared libraries.

  5. Version Tracking: It updates the version marker file at log/TVERSION within the environment instance directory with the new version or patch identifier.

1.4.2. Creating a New Environment Instance (Fresh Install)

To perform a fresh installation of an environment instance, run the redenv_create utility:

redenv_create <new_env_name>

1.4.2.1. New Instance Process

The utility sets up a new directory structure by cloning the .seed template:

  1. Port Allocation: It scans all existing environment instances on the host to determine their configured ports. It automatically assigns new, non-overlapping port numbers for the following parameters:

    • service_base_port

    • mail2cmd_port

    • gu_port

  2. Seed Replication: It copies the complete directory structure and default templates from /app/redsys/env/.seed to the new instance directory env/<new_env_name>.

1.4.3. Switching to the Environment Context

To work within a specific environment instance (for example, to verify configurations or run database setups), you must switch into its context. You can do this using the redenv utility:

redenv <env_name>

Important: The redsys profile automatically generates a shell alias for every active environment instance. This means you can simply type the environment name itself as a command to switch into its context:

<env_name>

Note

Fresh Installations: If you have just created a new environment instance using redenv_create, its alias will not be immediately available in your active shell session. You must reload your user profile to generate the new alias. This can be done by sourcing the profile directly (source ~/.bashrc), sourcing the aliases file (source /app/redsys/etc/aliases), or logging into the redsys user once more (e.g., su - redsys).

When you enter an environment context, your shell prompt will update to reflect the active environment, and all subsequent RED commands will operate against this specific instance.

1.4.4. Configuration Parameter Customisation

Following the upgrade, patch, or creation of the environment instance, verify the configuration settings:

# Switch to the environment first
<env_name>

vi etc/red_params.conf

Verify that all environment-specific values, such as database connection strings, are set correctly.

1.5. Phase 3: Database Schema Installation

The database schema installation, upgrade, or patch is the final step of the TV deployment process. It applies the necessary changes to the target database schema associated with the environment instance.

1.5.1. Running the Database Installer

To execute the database schema installer, you must switch to the target environment context and run the redtvctl install utility. This utility requires the database SYS password.

# Switch to the environment context
<env_name>

# Run the installer
redtvctl install

1.5.2. Database Installer Internal Process

Under the hood, the database installer performs the following actions:

  1. Establish SYS Connection: Connects to the Exasol database using SYS privileges and verifies that the database parameters and connection limits are sufficient.

  2. Installation Mode Detection:

    • Full Install (Fresh): Triggers when the database does not contain any existing RED user schemas.

      • The installer creates all RED system schemas, tables, views, and metadata repository tables from scratch.

    • Incremental Upgrade: Triggers when existing RED schemas are detected with an earlier TV version.

      • The installer queries the redsys.adm_current_tversion view to identify the current database TV version.

      • It verifies that the active database version is marked with a status of CLOSED.

      • It scans the /app/redsys/env/<env_name>/src/tv/install/incr/ directory to identify database upgrade scripts (install.sql) for all versions in the range current_version < version <= target_version.

      • It sequentially executes these SQL scripts via the exaplus database client using a secure, temporary profile.

    • Patch Application: Triggers when the database TV version matches the patch prerequisite.

      • It executes patch-level SQL updates, re-compiles updated stored procedures/scripts, and updates metadata views without requiring a full version closure cycle.

  3. Log Spooling: The installer spools all execution details and compilation messages to a dedicated log file:

    • log/tv.install.full.<version>.<timestamp>.log (for fresh installs)

    • log/tv.install.incr.<version>.<timestamp>.log (for upgrades)

    • log/tv.install.patch.<version>.<timestamp>.log (for patch updates)

  4. Verification: The installer checks the database status after the scripts have completed. It verifies that the database has successfully transitioned to the target TV version or patch revision and that its status is set to CLOSED.

1.5.3. Service Resumption

Once the database installer exits successfully, leave the environment subshell context and start the instance services:

exit
redsysctl start <env_name>