Additional Projects / Data Engineering
COVID-19 Data Platform
An Azure Data Factory–focused data engineering solution for analysing and predicting COVID-19 across the EU/EEA & UK — built to support reporting, analysis, and data-science scenarios.
Developer
Anthony Calek
Platform
Microsoft Azure
Focus
Azure Data Factory
Date
January 2024
Solution Architecture
Raw data is ingested over HTTP and from Blob storage, orchestrated by Azure Data Factory into a Gen2 data lake, transformed across multiple engines, then served to Azure SQL and Synapse for Power BI reporting.
HTTP + Blob
ECDC COVID-19 datasets over HTTP; EuroStat population from Blob storage.
Azure Data Factory
Pipelines, triggers & monitoring drive ingestion and integration.
ADLS Gen2
Raw and presentation data land in a Gen2 data lake.
Multi-engine
ADF Data Flows, HDInsight (Hive) and Databricks reshape each dataset.
SQL → Synapse → Power BI
Presentation data feeds Azure SQL and Synapse for Power BI reports.
Data-science branch — presentation datasets also land in a Gen2 lake, ML-ready (modelling left for future work).
How it works
ADF Data Flows enriched the Cases & Deaths data with country lookups and UK-specific records, and split hospital admissions into daily and weekly grains using a date-dimension lookup.
HDInsight (Hadoop + Hive) transformed the Testing dataset and Databricks reshaped the EuroStat population data — chosen to show ADF orchestrating across engines rather than to lean on any one. Presentation datasets then loaded into Azure SQL for Synapse analysis and the Power BI reports.
CI/CD — Azure DevOps
An advanced CI/CD setup mirrors a production team: Git integration, branch policies, approval workflows and a build pipeline spanning three environments.
Power BI Reports
Three interactive report pages built on the presentation datasets — switch between them below.
Azure Data Factory’s integration and orchestration make it a strong backbone for complex public-health datasets — scalable, secure, and ready for real-time, data-driven response.
From the Original Documentation
The original Azure-styled diagrams produced for the project, preserved here as source artifacts.
Original solution architecture
The production-style ADF architecture diagram from the original project documentation.
Original CI/CD architecture
The Dev → Test → Prod CI/CD design on Azure DevOps, from the original documentation.