1.2. Workflow & Lineage Versioning

The redlin architecture is designed to support continuous integration and automated metadata harvesting. As business data models evolve and dashboards are updated, the Lineage Mart must reflect these changes systematically and historically.

1.2.1. The Harvesting Workflow

The end-to-end lineage generation follows a deterministic workflow, typically triggered after deployment windows or on a scheduled basis:

  1. Deployment & Modification: Data engineers deploy changes to the Exasol data model (e.g., modifying ELT views, business logic, or Star Schema structures), or BI developers publish updates to MicroStrategy dashboards.

  2. Harvesting Trigger: The redlin pipeline is executed. This process is usually integrated into a CI/CD pipeline (e.g., GitLab CI) or orchestrated as a scheduled batch job.

  3. Extraction & Parsing: * Exasol: redlin extracts the latest view definitions and system metadata. It parses the raw SQL queries into an Abstract Syntax Tree (AST) to resolve column-level dependencies. * MicroStrategy: redlin interfaces with the BI platform to extract the latest dataset, report, and dashboard definitions.

  4. Consolidation & Optimization: The raw extracted data is consolidated into standardized structures. Duplicate dependencies are removed to ensure a clean directed acyclic graph (DAG).

  5. Lineage Mart Update: The finalized, deduplicated lineage graph is written to the Exasol Lineage Mart, making the latest metadata immediately available for downstream consumption.

  6. Pushing to Targets The lineage is pushed to downstream systems to provide them with the latest lineage information.

           graph TD
       subgraph InfoContainer [ ]
        direction LR
        C -.-B[Harvesting Trigger]
       end

       A[ETL Deployment] -->|Changes deployed| C[Exasol Extraction & Parsing]
       C --> E[MicroStrategy API Extracting]
       E --> F[Consolidation & Cleansing]
       F -->|Deduplicate & Create DAG| G[(Lineage Mart Update)]
       G -->|Final Lineage Data| H[Pushing to Targets]

       %% Den Subgraph komplett unsichtbar machen
       style InfoContainer fill:none,stroke:none

       %% Post-it Styling für die Info-Box
       style B fill:#fff9c4,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5

1.2.2. Lineage Versioning Strategy

To maintain a historical audit trail of data lineage and support point-in-time analysis, redlin implements a snapshot-based versioning system:

  • Version ID Stamping: Every harvesting run is assigned a unique version_id (typically a timestamp or a deployment pipeline ID). All extracted entities (items) and their relationships (bases) are stamped with this identifier during the enrichment phase.

  • Expression Hashing: To prevent database bloat across multiple runs, raw SQL expressions are hashed to create unique EXPRESSION_ID values. If an expression has not changed between versions, redlin links to the existing record and strips the redundant expression data to optimize storage.

  • Historical Traceability: Because every node and edge in the lineage graph is versioned, Data Governance platforms (like dataspot) can query the Lineage Mart to track the lifecycle of a data point, comparing the current production state against previous model iterations.