3.2. red_params.conf Configuration¶
The red_params.conf file contains the primary configuration for the RED Lineage module. It is divided into sections such as [DEFAULT] and [REDLIN] to manage general settings and module-specific configurations.
3.2.1. [DEFAULT] Section¶
This section defines global default settings for the application.
- log_level¶
Sets the global logging level for the application (e.g.,
INFO,DEBUG,WARNING).
- exa_connection¶
The connection string, IP address, and port used to connect to the Exasol database.
- exa_dbname¶
The name of the Exasol database to connect to.
3.2.2. [REDLIN] Section¶
This section contains parameters specific to the harvesting and processing behavior of the REDLIN module.
- log_level¶
Sets the logging level specifically for the REDLIN module. Overrides the global setting if provided.
- num_processes¶
Defines the number of parallel processes to use for harvesting Exasol lineage to improve performance. (e.g.,
6).
- max_task_per_child¶
Sets the maximum number of views a worker process parses before it is replaced by a new one, helping to release memory in large runs (default:
50).
- max_worker_memory_mb¶
Limits the additional memory (in MB) a worker process may allocate (Linux only). A view exceeding the limit is reported as failed with
MemoryErrorinstead of exhausting the server memory; the run continues. Size it so thatnum_processes×max_worker_memory_mbplus the main process fits into the server memory (e.g.,8000).0= unlimited (default).
- trim_selects¶
trueenables the legacy sqlglot lineage modetrim_selects=True/copy=True, which deep-copies the whole SELECT (including the WITH clause) for every lineage node and therefore needs a lot of memory on large views. The CLI option--trim-selects/--no-trim-selectstakes precedence. Default:false.
- push¶
A list of target systems to push the data lineage to (e.g.,
['dataspot_interface']).
- dataspot_interface¶
Configuration definition for the Dataspot interface, containing target SQL configurations for different layers (e.g.,
[{"target_sql": "LIN_SOURCES"}, {"target_sql": "LIN_EXASOL"}, {"target_sql": "LIN_MSTR"}]).