The augmented medallion architecture - Gold layer

The Gold layer - Where all the business rules are implemented

Purpose of the layer

The main purpose of this layer is to provide accurate business oriented datasets, enabling decision-making.

The secondary purposes of this layer are to :

• Centralized data governance : Enable the enforcement of a data governance on business rules and KPIs
• Centralized reference : Act as a centralized repository of standard company data

Characteristics of the layer

• Storage should be optimized for analytical use
  • The data should be stored in a query-friendly structures, including primary key definition, partitionning and indexing
• Allowed Data Structures
  • Structured
• Granularity of data
  • The data can be detailed and/or aggregated
• Applies business logic
  • The data is transformed according to the business logic, such as grouping, summarizing and KPI calculation

Good practices

The gold layer contains usually a good part of the intelligence of the data platform with all the business rules and KPIs. This means that there can be a high level of complexity in the data pipelines. To ensure easy root cause analysis and to facilitate maintenance, it is advised to split all complex pipelines into smaller processes.

Standard transformations applied

• Referential Data Management
  • Creation of complete referential with attributes and hierarchies
• Business logic rules
  • KPI Calculation
  • Currency & Unit Conversion
  • Segmentation & Categorization
• Aggregation & Summarization
  • Aggregate on business relevant axis

Examples

Creation of complete referential

The creation of complete referential is the fact to merge all the data related to a particular axis of analysis. In most cases, this is not mandatory, but it is useful to have the entire data of a referential available in a single location, e.g. a table.

Silver Table - Customer

Silver Table - Customer Group Hierarchy

Gold Table - Customer Complete referential

All the information is stored inside a single table for easier access to the data.

Conversion of amounts and quantities

The conversion of amounts and quantities can be useful to provide comparable data.

In this example, on a list of purchase orders, we'll convert the billed amount into a company currency (EUR), using a currency conversion table parametrized in the parametrization layer.

Silver Table - Orders

Parameter Table - Currency Conversion

Gold Table - Orders

3 purchase orders were billed in a different currency as the company currency, which is EUR. A conversion of the billed amount is applied using the currency conversion parametrization table considering the billing date as the date of reference for currency conversion.

Aggregation of data

The aggregation of data can be relevant to summarize data for analysis.

Usually, the aggregation is not the first step in the gold layer and thus the source tables for this logic are usually also in the gold layer.

In this example, the goal is to provide the quantity of cost linked to purchase orders by month (based on billing date), customer and product.

Gold Table - Orders

Gold Table - Orders summary

This article is part of a series of article describing the augmented medallion architecture.