1.1. Architecture

        flowchart BT
    classDef exasolBox fill:#e8f4ea,stroke:#c4e3c9,stroke-width:2px;
    classDef lineageBox fill:#ffff00,stroke:#333,stroke-width:2px,color:#000;
    
    subgraph Sources [Data Sources]
        direction LR
        DB[(Core Systems)]
        Ext[/External Sources/]
        Web([Call-Center / Web-Page])
        
        DB ~~~ Ext ~~~ Web
    end

    subgraph Exasol [Exasol]
        direction BT
        ELT["<div style='display: flex; align-items: center; justify-content: center;'><img src='../../_static/data_lineage_guide/concepts/red.png' style='width: 16px; height: 20px; margin-right: 8px;' />RED - ELT-View Layers</div>" ]
        Star[Star Schema<br>Business-Logic<br>Consolidation]
        LM["<div style='display: flex; align-items: center; justify-content: center;'><img src='../../_static/data_lineage_guide/redlin.png' style='width: 16px; height: 20px; margin-right: 8px;' />RED - Lineage Mart</div>" ]:::lineageBox
        
        ELT ==>|Data Flow| Star
    end

    BI["<div style='display: flex; align-items: center; justify-content: center;'><img src='../../_static/data_lineage_guide/concepts/ms.png' style='width: 16px; height: 20px; margin-right: 8px;' />BI-Tool: MicroStrategy<br>Controlling, Sales, Branches"]
    Gov{{"<div style='display: flex; align-items: center; justify-content: center;'><img src='../../_static/data_lineage_guide/concepts/dg.png' style='width: 16px; height: 20px; margin-right: 8px;' />DATA GOVERNANCE<br>Dataspot, Collibra, etc."}}
    Dashboards[Dashboards]

    Sources ==>|Data Ingestion| ELT
    Star ==>|Data Consumption| BI

    ELT -.->|Harvesting| LM
    Star -.->|Harvesting| LM
    BI -.->|Harvesting| LM

    LM ==>|Export| Gov
    LM ==>|Visualize| Dashboards
    
    class Exasol exasolBox;

1.1.1. Overview

The redlin module provides an automated, end-to-end data lineage extraction and consolidation framework. It is fully integrated into the RED-Framework. Actually supporting environments utilizing Exasol as the core Data Warehouse and MicroStrategy as the primary Business Intelligence platform, it bridges the gap between raw data ingestion and enterprise data governance.

By parsing SQL views, ELT processes, and BI metadata, redlin constructs a comprehensive map of data flows, ultimately storing this metadata in a dedicated Lineage Mart.

This data can be used to be propagated to a datagovernance tool. Dataspot is actually supported.

1.1.2. Architecture Layers

The architecture can be logically divided into four main pillars: Data Sources, the Exasol Data Warehouse (Integration & Modeling), the Lineage Mart (redlin Core), and Data Governance / Consumption.

1.1.2.1. Data Sources

Data originates from a variety of operational and external systems before being ingested into the warehouse. Typical sources include:

  • Core Systems: Transactional databases and legacy systems.

  • External Sources: Third-party data feeds and flat files.

1.1.2.2. Exasol Data Warehouse (Processing)

All data transformation and business logic consolidation occurs within the Exasol environment.

  • RED – ELT-View Layers: The foundational layer where raw data is extracted, loaded, and transformed (ELT). This layer handles the initial mapping and historization.

  • Applicationmarts: The semantic modeling layer. Data is aggregated and structured into dimensional models (facts and dimensions) optimized for querying and reporting.

1.1.2.3. Business Intelligence (MicroStrategy)

Sitting on top of the Star Schema is the BI-Tool (e.g., MicroStrategy).

  • Consumption: This layer delivers dashboards, PDF reports, and analytics to various business units such as Controlling, Sales, and Branch Management.

  • Lineage Contribution: The BI platform acts not just as a consumer, but also as a metadata source. The metrics, attributes, and dashboard definitions are crucial for end-to-end lineage.

1.1.2.4. Lineage Mart

The Lineage Mart is the central component of the redlin architecture. It is a dedicated schema/mart within Exasol that aggregates metadata from the entire pipeline:

  • Bottom-Up Extraction: redlin parses the SQL definitions from the ELT-View Layers to track column-level lineage from the source up to the Star Schema.

  • Top-Down Extraction: It extracts metadata from the Star Schema and cross-references it with the BI-Tool (MicroStrategy) definitions.

  • Consolidation: The resulting directed acyclic graph (DAG) of data movement is stored in the Lineage Mart, providing a unified view of “where data comes from” and “where it is used.”

Important

The Lineage Mart should be accessed via the function lin_gen_lineage or lin_live_lineage to ensure correct and performant data retrieval as it abstracts the internal complexity which is matter of change. brackets.

1.1.2.5. Data Governance & Export

The consolidated lineage data is not meant to stay siloed. The Lineage Mart serves as the authoritative source for enterprise Data Governance platforms. Supported target systems include:

  • dataspot.

While native integration is not currently supported, compatibility with the following platforms is actively being investigated for future releases

  • OpenLineage

  • Collibra

  • Informatica

  • watsonx.data

  • D-QUANTUM

By feeding these platforms, redlin ensures that data stewards, business users, and compliance teams have accurate, up-to-date visibility into the entire data lifecycle.

1.1.3. Data Flow Summary

        flowchart TD
    classDef lineageBox fill:green,stroke:#333,stroke-width:2px,stroke-dasharray:5 5;
    
    Sources[Sources] --> ELT[Exasol: ELT Views]
    ELT --> Star[Exasol: APV Schema]
    Star --> BI[BI-Tool: MicroStrategy]
    BI --> Reports(Dashboards / Reports)

    ELT --> | Harvesting| Lineage[Exasol: Lineage Mart]:::lineageBox
    Star --> | Harvesting | Lineage
    BI --> | Harvesting | Lineage

    Lineage --> Gov[Data Governance Tools]:::lineageBox

1.1.4. Workflow & Lineage Versioning

The redlin architecture is designed to support continuous integration and automated metadata harvesting. As business data models evolve and dashboards are updated, the Lineage Mart must reflect these changes systematically and historically.

1.1.4.1. The Harvesting Workflow

The end-to-end lineage generation follows a deterministic workflow, typically triggered after deployment windows or on a scheduled basis:

  1. Deployment & Modification: Data engineers deploy changes to the Exasol data model (e.g., modifying ELT views, business logic, or Star Schema structures), or BI developers publish updates to MicroStrategy dashboards.

  2. Harvesting Trigger: The redlin pipeline is executed. This process is usually integrated into a CI/CD pipeline (e.g., GitLab CI) or orchestrated as a scheduled batch job.

  3. Extraction & Parsing: * Exasol: redlin extracts the latest view definitions and system metadata. It parses the raw SQL queries into an Abstract Syntax Tree (AST) to resolve column-level dependencies. * MicroStrategy: redlin interfaces with the BI platform to extract the latest dataset, report, and dashboard definitions.

  4. Consolidation & Optimization: The raw extracted data is consolidated into standardized structures. Duplicate dependencies are removed to ensure a clean directed acyclic graph (DAG).

  5. Lineage Mart Update: The finalized, deduplicated lineage graph is written to the Exasol Lineage Mart, making the latest metadata immediately available for downstream consumption.

1.1.4.2. Lineage Versioning Strategy

To maintain a historical audit trail of data lineage and support point-in-time analysis, redlin implements a snapshot-based versioning system:

  • Version ID Stamping: Every harvesting run is assigned a unique version_id (typically a timestamp or a deployment pipeline ID). All extracted entities (items) and their relationships (bases) are stamped with this identifier during the enrichment phase.

  • Version ID Aliasing: Allows to allias the version_id with business related meaning full names.

  • Expression Hashing: To prevent database bloat across multiple runs, raw SQL expressions are hashed to create unique EXPRESSION_ID values. If an expression has not changed between versions, redlin links to the existing record and strips the redundant expression data to optimize storage.

  • Historical Traceability: Because every node and edge in the lineage graph is versioned, Data Governance platforms (like dataspot) can query the Lineage Mart to track the lifecycle of a data point, comparing the current production state against previous model iterations.