Incremental Ingestion and Processing of PostgreSQL data using Databricks Spark Declarative Pipelines
The process of (incremental) ingestion and processing of data using the medallion architecture can be hard.... Luckily, Databricks offers a lot of functionality to make your life easier. In this blog series I will show you how to incrementally load PostgreSQL data and process it through the medallion architecture with Databricks spark declarative pipelines. The full code is available here. Databricks medallion architecture PoC Tools In this PoC I use several tools: Data source: PostgreSQL database from Render. You can get this database for free for a period of one month, without entering your creditcard details. pgAdmin4: Free PostgreSQL UI...
Read more