1. Overview and Main Features

The RED framework was built for Exasol as an Auto-ELT framework. RED is 100% matadata driven and manages all tasks around loading, historising and securing data. Loading data from heterogenous sources (like databases, files or REST-APIs) is accomplished through asynchronous, highly parallelised processes. However, no manual configuration or tuning is needed, no external schedulers are involved and there is no need for building manual transformation pipelines.

Furthermore, this RED framework contains prefabricated building blocks, which allows for individualised projects to be realised in a very short time. Another advantage of this framework is that, by merging the data of different databases and other files, a secure and high-performance analysis of these large amounts of data is possible, without having to make special adjustments or using tuning mechanisms. Even complex queries can be made due to this framework.

The next paragraphs deal with the main features of the RED framework in more detail.

1.1. Shortest Project Turnaround Times

Loading a new source with complete historisation into the analytical database is simple and only takes filling out the following metadata in the relevant tables:

  • Define the location of the source

  • Define a unique key (Business Surrogate Key (BSK)) and the type of historisation (event or period data)

  • Define load cycles (calendar logic)

  • Define security (on object, row and column level)

  • Activate the loading

Each of the options mentioned above, will be explained in more detail in the following chapters.

This declarative approach with its built-in calendar control and implicit dependency analysis eliminates the need for elaborate job designs and custom code writing. Furthermore, usually there is no core model, as the extreme performance of the database does not require it. Instead, a virtual layer can be built in the form of views, if, for example, a BI tool requires such. However, the data, which is automatically transferred 1:1 with unchanged structures, but completely and traceably historized, is stored in tables and is directly available to the department via such a view layer.

1.2. Automated Operation

In operation, the extraction of data sources takes place near real time via direct connections to the data source or via file load. The loading is highly parallel and asynchronous. No jobs need to be defined and dependencies within the data are recognised implicitly.

After the automated, complete historisation of the data, it is stored in the Persistent Staging Area. Conversions or transformations of the data are usually not necessary. However, if these are needed in special cases, they can be integrated via another view layer.

1.3. Complete Historisation

Historisation allows the reconstruction of data as it was valid at any point in the past. In other words, the data status that existed at a certain point in time, e.g. when a report was executed, can be queried. Historisation also makes it possible to analyse the complete historical development of individual data records.

To meet these requirements, all data is subject to strict bi-temporal, two-dimensional historisation. There is a functional and a technical validity period for each data set. The functional period results from the functional requirements, while the technical period reflects the time of storage. Another benefit of this type of historisation is that it enables a partial restore on table level.

1.4. Fine-Granular Security

Security is a central component: Personalised user accounts receive access rights on object, row and column level.

  • On the first level, classic access rights are granted to groups of objects.

  • On the second level, it can be decided for each object on the row level which data should be visible for which users.

  • On the third level, sensitive columns and the respective user access can be defined. The declarative assignment of permissions ensures that this high level of detail remains maintainable. Administration is kept to a minimum as automated processes take over complex activities in the background.

The business department has access via virtual layers (view layers), which are placed on top of the data tables. The business department does not have direct access to the base tables. The view layer also maps the access security at object, row and column level.

1.5. Full Compliance with Regulatory Frameworks

The RED framework is fully compliant with

  • NIS-2

  • GDPR

  • DORA

  • BSBS-239

1.6. Summary

Central features are:

  • Shortest project lead time and easy handling in operation

  • Highly parallelised, asynchronous loading without the need to define jobs

  • Complete historisation of data - traceability of all data changes

  • Fine-granular authorisation concept with personalised access

  • Use of existing well-known data structures without adaptation

  • High performance out-of-the-box

  • Fully compliant to regulatory frameworks