AWS datalake

About the project

Migration of an ETL pipeline into AWS
Batch pipelines to fill a datalake
- Using different zones: landing, structured, processed
- Delta updates
- Using AWS Step Functions
Import from
- Oracle
- CSV files
Export to
- Parquet files on AWS S3
- PostgreSQL, managed by AWS RDS
- CSV files
Metadata configuration using the Dhall configuration language
Generate case classes to represent records
- To get type safety
- To provide test data as code
- To simplify comparison in unit tests
- To provide quick information (type, nullable, comment, precision/scale, …) for attributes
- To get constants for attribute names instead of hard-coded values
Generate SQL scripts
- To create tables in PostgreSQL
Infrastructure-as-code using Terraform
Using AWS Athena for interactive queries
Create charts using Matplotlib and Leaflet/Folium to visualize reporting data
Create charts using Mermaid to visualize data flows