3.2. red_params.conf Configuration

The red_params.conf file contains the primary configuration for the RED Lineage module. It is divided into sections such as [DEFAULT] and [REDLIN] to manage general settings and module-specific configurations.

3.2.1. [DEFAULT] Section

This section defines global default settings for the application.

log_level

Sets the global logging level for the application (e.g., INFO, DEBUG, WARNING).

exa_connection

The connection string, IP address, and port used to connect to the Exasol database.

exa_dbname

The name of the Exasol database to connect to.

3.2.2. [REDLIN] Section

This section contains parameters specific to the harvesting and processing behavior of the REDLIN module.

log_level

Sets the logging level specifically for the REDLIN module. Overrides the global setting if provided.

num_processes

Defines the number of parallel processes to use for harvesting Exasol lineage to improve performance. (e.g., 6).

max_task_per_child

Sets the maximum number of views a worker process parses before it is replaced by a new one, helping to release memory in large runs (default: 50).

max_worker_memory_mb

Limits the additional memory (in MB) a worker process may allocate (Linux only). A view exceeding the limit is reported as failed with MemoryError instead of exhausting the server memory; the run continues. Size it so that num_processes × max_worker_memory_mb plus the main process fits into the server memory (e.g., 8000). 0 = unlimited (default).

trim_selects

true enables the legacy sqlglot lineage mode trim_selects=True / copy=True, which deep-copies the whole SELECT (including the WITH clause) for every lineage node and therefore needs a lot of memory on large views. The CLI option --trim-selects / --no-trim-selects takes precedence. Default: false.

push

A list of target systems to push the data lineage to (e.g., ['dataspot_interface']).

dataspot_interface

Configuration definition for the Dataspot interface, containing target SQL configurations for different layers (e.g., [{"target_sql": "LIN_SOURCES"}, {"target_sql": "LIN_EXASOL"}, {"target_sql": "LIN_MSTR"}]).