Data Platform Architecture
A full-stack data engineering pipeline — from raw sources to analytics-ready outputs.
🖱️ Hover a card on desktop · 👆 Tap on mobile to see how each tool is used
Data Sources
3 tools
PostgreSQL
Operational DB source for transactional ingestion pipelines
APIs
REST & GraphQL endpoints as real-time data sources
Files / Storage
ADLS Gen2 & GCS blobs as raw data landing zones
Streaming & Event Systems
3 tools
Kafka
High-throughput distributed event streaming for real-time pipelines
Google Pub/Sub
GCP-native async messaging & event delivery at scale
Azure Event Hub
Azure-native event streaming hub for high-volume telemetry
Data Ingestion
3 tools
Apache Airflow
DAG-based orchestration for complex batch pipeline scheduling
Azure Data Factory
Cloud ETL/ELT for scalable Azure data movement & transformation
Databricks Auto Loader
Incremental file ingestion with automatic schema evolution
Data Processing
4 tools
Apache Spark
Distributed processing engine for large-scale dataset transformations
Databricks
Unified analytics platform — used at Rolls-Royce & Boots for lakehouse builds
Python
Primary language for data engineering, automation & ML workflows
SQL
Core language for data modelling, transformation & analytical queries
Lakehouse & Warehousing
4 tools
Delta Lake
ACID-compliant lakehouse storage — foundation of medallion architecture
Snowflake
Cloud data warehouse for high-performance analytical workloads
dbt
SQL-based data transformation with lineage, testing & documentation
Microsoft Fabric
Unified SaaS analytics platform combining data engineering, warehousing & BI
AI & Generative AI
6 toolsAzure OpenAI
GPT-4 & embeddings via Azure-native OpenAI service for enterprise AI apps
LangChain
Agent orchestration & RAG pipeline framework for LLM-powered workflows
Hugging Face
Pre-trained transformer models & open-source model hub for NLP & vision
MLflow
ML experiment tracking, model registry & deployment lifecycle management
Vector DB
Pinecone & ChromaDB for embeddings storage powering semantic search & RAG
Machine Learning
TensorFlow & Scikit-learn for predictive modelling & feature engineering
Serving & Analytics
2 tools
Power BI
Enterprise BI dashboards & self-service analytics for stakeholders
Knowledge Graph
Neo4j graph modelling for entity relationships & connected data queries
Platform Engineering
4 tools
Docker
Containerisation for reproducible, portable data pipeline environments
Kubernetes
Container orchestration for scalable, resilient data platform workloads
Azure
Primary cloud — ADF, ADLS Gen2, Synapse, Fabric & Azure DevOps
Google Cloud
GCP for BigQuery, Pub/Sub, Dataflow & Cloud Composer