Chapter 1: Data Engineering Described - Study Notes
Core Definition and Purpose
Data Engineering is the discipline of developing, implementing, and
maintaining systems that transform raw data into high-quality, consistent
information for downstream use cases. It serves as the critical bridge between
data sources and data consumers, sitting at the intersection of:
Security and data governance
Data management and architecture
DataOps and orchestration
Software engineering practices
Key Insight: Data engineering exists upstream from data science and
analytics, providing the essential foundation that enables these downstream
activities to succeed.
The Data Engineering Lifecycle Framework
Five Core Stages
Data Engineering Lifecycle
1. Generation
Origin point of all data
Includes source systems, applications, IoT devices
Understanding data generation patterns is crucial for downstream
design
2. Storage
Persistent infrastructure for data retention
Involves choosing appropriate storage solutions (data lakes,
warehouses, databases)
Must balance cost, performance, and accessibility
3. Ingestion
Movement of data from sources to storage systems
Can be batch, micro-batch, or real-time streaming
Requires handling various formats, protocols, and reliability concerns
4. Transformation
Converting raw data into analytically useful formats
Includes cleaning, validation, aggregation, and enrichment
Where business logic gets applied to data
5. Serving
Delivering processed data to end consumers
Supporting analytics, ML models, reporting, and applications
Ensuring appropriate access patterns and performance
Critical Undercurrents
These foundational elements flow through every stage of the lifecycle:
Security: Data protection, encryption, access control, compliance
Data Management: Governance, quality, lineage, cataloging
DataOps: Automation, monitoring, incident response
Data Architecture: System design, technology selection, scalability
planning
Orchestration: Workflow management, dependency handling, scheduling
Software Engineering: Code quality, version control, testing, deployment
The Data Science Hierarchy of Needs
Monica Rogati's influential 2017 framework illustrates why data engineering
must precede advanced analytics:
Data Science Hierarchy of Needs
Hierarchy Breakdown (Bottom to Top)
1. Data Collection (Foundation)
Identifying required data sources
Implementing logging and instrumentation
Ensuring data capture completeness
2. Data Flow & Storage (Infrastructure)
Building reliable ETL/ELT pipelines
Establishing accessible storage systems
Ensuring data availability and durability
3. Exploration & Transformation (Preparation)
Data cleaning and quality assurance
Discovering data issues and gaps
Creating reusable transformation logic
4. Analytics & Metrics (Insights)
Defining KPIs and business metrics
Building dashboards and reports
Creating features for ML models
5. Machine Learning & AI (Advanced Analytics)
Model development and training
Deployment and monitoring
Continuous improvement
Critical Statistic: Data scientists typically spend 70%+ of their time on the
bottom three layers. Effective data engineering can flip this ratio, enabling
90%+ focus on advanced analytics.
Evolution and Modern Context
Historical Development
Emerged from the pain points of data scientists doing manual data work
Evolved from traditional ETL to modern cloud-native architectures
Shifted from custom infrastructure to managed services
2025 Landscape Characteristics
Cloud-Native Design: Default architecture for new systems
Managed Services: Reduced operational overhead through PaaS/SaaS
Real-Time Processing: Streaming as standard, not exceptional
DataOps Maturity: CI/CD, IaC, and automation as baseline practices
Decentralization: Data mesh principles gaining adoption
Essential Skills Portfolio
Technical Competencies
Programming & Development
Python for pipeline development and automation
SQL for data manipulation and analysis
Java/Scala for big data processing frameworks
Shell scripting for system automation
Data Systems
OLTP databases (PostgreSQL, MySQL)
OLAP systems (Snowflake, BigQuery, Redshift)
NoSQL stores (MongoDB, Cassandra)
Message queues (Kafka, RabbitMQ)
Cloud & Infrastructure
Major platforms (AWS, GCP, Azure)
Container orchestration (Kubernetes, Docker)
Infrastructure as Code (Terraform, CloudFormation)
Monitoring and observability tools
Data Processing
Apache Spark for distributed computing
Stream processing (Flink, Kafka Streams)
Orchestration frameworks (Airflow, Dagster)
Data integration tools
Business & Soft Skills
Communication
Translating technical concepts for non-technical stakeholders
Writing clear documentation
Presenting data strategies to leadership
Strategic Thinking
Aligning data initiatives with business objectives
Cost-benefit analysis for technology choices
Risk assessment and mitigation planning
Collaboration
Cross-functional teamwork
Stakeholder management
Mentoring and knowledge sharing
Stakeholder Ecosystem
Primary Collaborators
Data Scientists & ML Engineers
Provide clean, feature-ready datasets
Support model deployment infrastructure
Ensure data quality and consistency
Business Analysts
Enable self-service analytics
Maintain reliable reporting pipelines
Provide data documentation and lineage
Software Engineers
Integrate with application systems
Collaborate on API design
Share engineering best practices
Leadership & Management
C-suite executives for strategic initiatives
Product managers for requirement gathering
Project managers for delivery coordination
Data Maturity Framework
Three Maturity Stages
1. Starting with Data
Focus: Basic data collection and organization
Challenges: Lack of infrastructure, unclear requirements
DE Role: Building foundational systems, educating stakeholders
2. Scaling with Data
Focus: Standardization and efficiency
Challenges: Growing complexity, performance issues
DE Role: Architecting scalable solutions, establishing best practices
3. Leading with Data
Focus: Innovation and competitive advantage
Challenges: Advanced use cases, real-time requirements
DE Role: Enabling cutting-edge capabilities, optimizing costs
Guiding Principles for Success
Core Philosophy
Principles Over Tools: Technologies are temporary; fundamentals endure
Trade-off Awareness: Every decision involves compromises
Reversibility: Design for flexibility and avoid vendor lock-in
Security-First: Build protection into every layer
Cost Consciousness: Apply FinOps principles throughout
Decision Framework
1. Understand the business problem before choosing technology
2. Evaluate build vs. buy vs. managed service options
3. Consider long-term maintenance and scalability
4. Plan for failure and design for recovery
5. Document decisions and rationale
Key Success Factors
Technical Excellence
Master fundamentals before chasing trends
Build robust, maintainable systems
Automate repetitive tasks
Monitor and measure everything
Business Alignment
Understand organizational goals and constraints
Communicate value in business terms
Prioritize based on impact
Build trust through reliable delivery
Continuous Learning
Stay current with evolving technologies
Learn from failures and incidents
Share knowledge with the community
Seek feedback and iterate
Chapter Summary
Data engineering has evolved into a critical discipline that enables
organizations to extract value from their data assets. Success requires
mastering both technical skills and business acumen, while maintaining focus
on fundamental principles rather than specific tools.
The data engineering lifecycle provides a robust framework for thinking about
data systems, while the hierarchy of needs illustrates why strong data
foundations must precede advanced analytics. As the field continues to evolve
rapidly, data engineers who focus on principles, collaboration, and continuous
learning will be best positioned to deliver lasting value.
Essential Takeaways
✅ Data engineering transforms raw data into valuable information through a
systematic lifecycle ✅ The five stages (Generation → Storage → Ingestion
→ Transformation → Serving) provide a tool-agnostic framework ✅ Strong
data foundations enable data scientists to focus on high-value work ✅
Success requires balancing technical expertise with business understanding
✅ Focus on enduring principles rather than temporary technologies ✅
Collaboration across the organization is essential for impact ✅
Organizational data maturity determines the data engineer's role and priorities
Chapter 2: The Data Engineering Lifecycle - Study Notes
Core Concept: From Tools to Principles
The data engineering lifecycle represents a fundamental shift in how we
approach data engineering—moving from a tool-centric mindset to a
principle-centric philosophy. This framework describes the "cradle to grave"
journey of data through an organization, providing a holistic perspective that
remains relevant regardless of technological changes.
Key Insight: Data engineers are increasingly becoming data lifecycle
engineers, thinking and operating in terms of lifecycle management principles
rather than specific technologies.
Understanding the Framework
Data Lifecycle vs. Data Engineering Lifecycle
Data Engineering Lifecycle
Important Distinction:
Data Lifecycle: Encompasses the entire lifespan of data from creation to
deletion, including all business interactions
Data Engineering Lifecycle: Focuses specifically on stages where data
engineers have direct control and responsibility
This subset relationship is crucial—data engineers don't control everything
about data but must understand the broader context while focusing on their
domain of responsibility.
The Five Core Stages
The lifecycle divides into five interconnected stages that transform raw data
into valuable information ready for consumption:
1. Generation: Understanding Source Systems
Definition: Source systems are the origins of data used in the data
engineering lifecycle. Data engineers consume from these systems but
typically don't control them.
Key Characteristics:
Variety of Sources:
Transactional databases (OLTP systems)
IoT devices and sensors
File systems (CSV, XML, JSON)
APIs and web services
Message queues and event streams
SaaS applications
Critical Considerations:
Data Generation Patterns:
Continuous vs. batch generation
Event-driven vs. scheduled
Push vs. pull mechanisms
Quality at Source: Understanding inherent data quality issues
Schema and Format: Structured, semi-structured, or unstructured
Access Constraints: Authentication, authorization, rate limits
Upstream Dependencies: SLAs, maintenance windows, change
management
Engineering Questions:
How does the source system handle state changes?
What's the frequency and velocity of data generation?
What happens when the source system fails?
How do we handle schema evolution at the source?
2. Storage: The Foundational Layer
Unique Position: Storage is the only stage that underpins the entire lifecycle,
occurring multiple times throughout a pipeline.
Modern Storage Reality:
Cloud architectures typically leverage multiple storage solutions
Few solutions function purely as storage—many support complex queries
Storage choices impact every other stage of the lifecycle
Storage Types and Use Cases:
Object Storage (S3, GCS, Azure Blob):
Raw data landing zones
Archive and backup
Unstructured data
Data Warehouses (Snowflake, BigQuery, Redshift):
Structured analytical workloads
Business intelligence
Historical analysis
Data Lakes/Lakehouses (Delta Lake, Iceberg):
Multi-format data
Schema evolution support
ACID transactions on files
Operational Databases:
Application data
Transactional workloads
Real-time serving
Critical Engineering Questions:
Is the storage compatible with required read/write patterns?
How will it scale with data volume growth?
What are the cost implications at scale?
How does it handle different data formats and schema evolution?
What are the durability and availability guarantees?
Can it support both batch and streaming workloads?
3. Ingestion: The Critical Bridge
Reality Check: Ingestion and source systems are often organizational
bottlenecks because:
Source systems are externally managed
Ingestion can fail unpredictably
Failures cascade throughout the pipeline
Ingestion Patterns:
Batch Ingestion:
Characteristics: Data moved in discrete chunks
Triggers: Time-based, size-based, event-based
Use Cases: Daily reports, periodic syncs, large historical loads
Benefits: Simpler error handling, easier replay, cost-effective
Challenges: Latency, handling incremental updates
Streaming Ingestion:
Characteristics: Continuous, near real-time data flow
Technologies: Kafka, Kinesis, Pub/Sub
Use Cases: Real-time analytics, fraud detection, monitoring
Benefits: Low latency, immediate insights
Challenges: Complexity, ordering guarantees, exactly-once processing
Key Ingestion Questions:
What are the downstream use cases for ingested data?
Can we reuse data instead of creating duplicates?
Is the ingestion reliable and predictable?
What volume and velocity of data are expected?
Can downstream systems handle the incoming format?
How do we handle late-arriving or out-of-order data?
4. Transformation: Creating Value from Raw Data
Purpose: Convert raw data into analytically useful information through
refinement operations.
Transformation Activities:
Data Cleaning: Handling nulls, duplicates, inconsistencies
Validation: Ensuring data meets quality standards
Aggregation: Summarizing detailed data
Joining: Combining data from multiple sources
Feature Engineering: Creating ML-ready features
Business Logic: Applying domain-specific rules
Normalization/Denormalization: Optimizing for use cases
Modern Transformation Approaches:
SQL-First (dbt, SQLMesh):
Declarative transformations
Version control friendly
Accessible to analysts
Code-Based (Spark, Pandas):
Complex logic support
Custom algorithms
Performance optimization
Streaming Transformations:
In-flight processing
Windowing and aggregations
Stateful operations
Critical Decisions:
Batch vs. streaming transformation?
Where should transformations occur (ELT vs. ETL)?
How to handle data quality issues?
What level of testing and validation is needed?
How to manage transformation dependencies?
5. Serving: Delivering Value
The Payoff Stage: Where stakeholders realize value from data engineering
efforts.
Primary Serving Patterns:
Analytics:
Business Intelligence: Dashboards, KPIs, executive reports
Self-Service Analytics: Empowering business users
Operational Analytics: Real-time business metrics
Embedded Analytics: In-application insights
Machine Learning:
Training Datasets: Historical data for model development
Feature Stores: Consistent features for training/serving
Real-time Inference: Low-latency predictions
Batch Scoring: Periodic predictions at scale
Reverse ETL:
Definition: Syncing processed data back to operational systems
Use Cases: CRM enrichment, marketing automation, operational
workflows
Technologies: Census, Hightouch, custom solutions
Data Products:
APIs: RESTful or GraphQL endpoints
Data Shares: Snowflake shares, BigQuery datasets
Files/Reports: Scheduled deliveries, exports
Streaming: Real-time data feeds
The Undercurrents: Essential Foundations
Data Engineering Lifecycle with Undercurrents
Security: Non-Negotiable Protection
Access Control: RBAC, ABAC, least privilege, API keys
Encryption: At-rest, in-transit, field-level
Compliance & Privacy: GDPR, HIPAA, anonymization
Audit & Monitoring: logging, anomaly detection, incident response
Data Management: Ensuring Trust and Usability
Governance frameworks, ownership, data policies
Quality dimensions: accuracy, completeness, consistency, timeliness,
validity
Metadata & Cataloging: technical, business, lineage
DataOps: Operational Excellence
CI/CD, automated testing, IaC, self-healing
Observability: metrics, logging, tracing, alerting
Reliability: SLA/SLOs, incident management, chaos engineering
Data Architecture: The Blueprint
Lambda: batch + streaming
Kappa: streaming-only
Data Mesh: domain-oriented
Lakehouse: unify lake & warehouse
Design considerations: scalability, cost, integration, performance
Orchestration: Coordination and Control
DAGs, scheduling, backfilling, branching
Resource management, error handling
Modern tools: Airflow, Dagster, Prefect, Temporal
Software Engineering: Professional Practices
Standards: version control, code quality, docs, reviews
Testing pyramid: unit, integration, end-to-end, data quality
Deployment: blue-green, canary, feature flags, rollback
Practical Implementation Example
E-Commerce Recommendation System
1. Generation: clickstream, purchases, catalog, reviews
2. Storage: S3 raw → Delta Lake processed → Feature store serving
3. Ingestion: Kafka (real-time), Airbyte (batch), quality checks
4. Transformation: cleaning, joins, metrics, ML feature vectors
5. Serving: ML training, real-time API, dashboards, reverse ETL
# Simplified transformation logic
- Clean and validate customer events
- Join clickstream with purchase history
- Calculate customer lifetime value
- Create product affinity scores
- Generate ML feature vectors
- Aggregate business metrics
Throughout - Undercurrents:
Security (encrypt PII), DataOps (testing/monitoring), Architecture
(microservices), Orchestration (Airflow/Flink).
Key Trade-offs and Decisions
Build vs. Buy: custom, commercial, OSS, managed services
Batch vs. Streaming: simple/cheap vs. complex/real-time
Centralized vs. Decentralized: governance vs. agility, federated hybrid
Modern Evolution and Trends (2025)
Live Data Stack: streaming-first, embedded analytics, ML integration, selfservice
Tech Shifts: serverless, unified batch/stream, cloud-native, AI-assisted
Org Changes: Data Mesh adoption, platform teams, FinOps, DataOps
maturity
Key Success Factors
Technical Excellence
✅ Master fundamentals
✅ Design for failure & recovery
✅ Automate tasks
✅ Monitor everything
Business Alignment
✅ Understand value chain
✅ Communicate in business terms
✅ Focus on outcomes
✅ Deliver incrementally
Continuous Improvement
✅ Learn from incidents
✅ Stay current
✅ Share knowledge
✅ Measure & optimize
Chapter Summary
The lifecycle (Generation → Storage → Ingestion → Transformation →
Serving) + undercurrents (Security, Data Management, DataOps, Architecture,
Orchestration, Software Engineering) = robust, scalable, valuable systems.
It’s a mental model, not a rigid prescription—guides decision-making despite
changing tools.
Essential Takeaways
1. Principles > tools
2. Storage underpins all stages
3. Bottlenecks cluster early (source/ingestion)
4. Undercurrents are first-class concerns
5. Stages are interconnected
6. Value realized at serving
7. Evolution is constant
Chapter 3: Designing Good Data Architecture — Refined Notes
Why Architecture Matters
Architecture is the backbone of the data lifecycle; it enables reliable
ingestion, transformation, governance, and serving.
“Good” means: flexible, reversible, cost-aware, secure, and aligned to
business outcomes—not just trendy tooling.
1) What Is Data Architecture?
Enterprise Architecture in Context
Enterprise architecture = designing systems to support change via thoughtful,
reversible decisions. Typical domains:
Business Architecture: Value creation, processes, capabilities
Application Architecture: Applications and their interactions
Technical Architecture: Platforms, networks, infrastructure
Data Architecture: How data flows, is modeled, governed, and served
Definition (working)
Data architecture is the design of systems that meet evolving enterprise data
needs through flexible and reversible decisions made with explicit trade-offs.
Two lenses:
Operational Architecture: People, process, responsibilities, governance,
SLAs/SLOs
Technical Architecture: How data is ingested, stored, transformed,
secured, cataloged, and served
What makes it “good”:
Serves real business use cases with reusable building blocks
Enables change (e.g., migrations, new workloads) without massive rework
Balances standardization and autonomy; documents trade-offs
2) Nine Principles of Good Data Architecture
1. Choose Common Components Wisely
Shared building blocks: storage, orchestration, observability, processing
engines.
Standardize where helpful; avoid tool sprawl. How to apply: Curate a
golden path, publish contracts/templates, review quarterly.
2. Plan for Failure
Reliability, availability, RTO, RPO.
Redundancy, autoscaling, retries, DLQs. How to apply: Define SLOs, run
chaos tests, document runbooks.
3. Architect for Scalability
Scale on volume, concurrency, geography, workload.
Prefer horizontal scale-out. How to apply: Separate compute/storage,
shard, cache reads, queue bursty writes.
4. Architecture Is Leadership
Choices shape culture, speed, and delivery. How to apply: Run
lightweight RFC/ADR process, office hours, review designs for reuse.
5. Always Be Architecting
Continuous evolution, never final. How to apply: Living roadmap, budget
for tech debt, run fitness checks.
6. Build Loosely Coupled Systems
Decouple teams/components; async-first. How to apply: Event-driven
patterns, schema versioning, distributed tracing.
7. Make Reversible Decisions
Favor open formats, interfaces, modularity. How to apply:
Iceberg/Delta/Hudi, abstract compute, write migration runbooks.
8. Prioritize Security
Zero-trust, defense-in-depth. How to apply: IAM centralization, secret
rotation, pen tests, compliance mapping.
9. Embrace FinOps
Cost as a first-class metric. How to apply: Cost SLOs,
showback/chargeback, reservations/spot, track unit economics.
3) Core Architecture Concepts
Domains & Services
Organize around business domains; data products as APIs with contracts.
Distributed Systems & Failure
CAP trade-offs, eventual consistency, observability. Coupling Spectrum
Monolith vs SOA/microservices: trade-offs depend on team/org scale.
Event-Driven Architecture
Real-time decoupling, schema registry, retries, DLQs.
4) Architecture Patterns
Batch-Oriented
Data Warehouse → governance & analytics
Data Lake → flexible & ML-friendly
Lakehouse → unify BI + ML with ACID Streaming-Oriented
Lambda → batch + speed layers (dual code paths)
Kappa → single stream, replay logs
Medallion (Bronze/Silver/Gold) → unified batch/stream on lakehouse
Modern
Data Mesh → domain ownership, data-as-product
Unified Batch + Streaming → single API (Beam/Flink/Spark)
5) Cloud Architecture Considerations
Separation of Compute/Storage: elasticity, open formats, caching.
Tenancy Models: multi-tenant, single-tenant, hybrid.
Serverless/Managed Services: less ops toil, but beware cold starts,
egress, lock-in.
Cloud Tips: Standardize IAM, networking, secrets; build guardrails.
6) Architecture in Practice
Technology Selection Framework
Use-case fit, team capability, TCO, ecosystem, exit strategy. Migration
Strategies
Lift & shift, re-platform, re-architect, strangler/hybrid. Organizational
Alignment
Conway’s Law, Team Topologies (platform + domain teams + enabling
teams).
7) Modern Context (2025)
Trends
AI-Native: auto-quality & governance
Real-Time Everything: freshness as edge
Zero-Trust: strong identity, micro-perimeters
Green Data: carbon-aware costs
Edge Integration: local processing, sync patterns
Platform Consolidation
Fewer, broader platforms with governance + observability.
8) Key Takeaways
Architecture = informed trade-offs.
Design for change: loose coupling, reversibility, open formats.
Security & FinOps must be foundational.
Lakehouse + event-driven unify batch/stream.
Architecture is continuous leadership.
9) Architecture Decision Record (ADR) Template
# ADR-[Number]: [Title]
## Status
[Proposed | Accepted | Deprecated]
## Context
What problem or decision point are we addressing? What
constraints matter?
## Decision
What did we decide and why (trade-offs)?
## Consequences
Impacts, risks, costs, and how we’ll measure success.
## Alternatives Considered
Options, pros/cons, reasons for not choosing.
Chapter 4: Choosing Technologies Across the Data Engineering
Lifecycle
Big idea
Architecture is strategic; tools are tactical. Start with principles and
constraints, then choose tools that fit your context and can evolve with you.
Technology Stack Overview
1) Start with Context: Team Size and Capabilities
Assess before you select:
Team size and skills
Org maturity and support (SRE, security, platform)
Compliance posture and data sensitivity
Operational bandwidth for self-hosting
Team patterns
Small teams (1–5) → Managed services, paved paths, SQL-first,
serverless.
Medium teams (5–20) → Mix of managed + self-hosted; start
standardizing; add specialization.
Large teams (20+) → Platform teams, governance, internal golden paths,
custom solutions where strategic.
Quick defaults
Small: Managed ingestion + lakehouse table format + dbt + managed
warehouse/lake compute + hosted orchestration.
Medium: Add streaming, data quality, catalog, cost guardrails.
Large: Federated domains, semantic layer, lineage, fine-grained security.
2) Optimize for Speed to Value
Speed considerations
Time to first insight (TTFI), iteration velocity, debt trade-offs, migration
costs. Accelerators
Managed services, prebuilt connectors, golden path reference
architectures, IaC modules. Guardrails
Define SLOs (freshness, latency, cost), version everything, keep exit paths
(open formats, export APIs, runbooks).
3) Interoperability First
Challenges
Format compatibility, schema evolution, metadata sharing, cross-system
security.
Helpful standards
Table: Iceberg, Delta, Hudi
Files: Parquet, ORC, Avro
Query: SQL, Trino/Presto
Metadata/telemetry: OpenLineage, OpenTelemetry
Schema evolution: Avro/Protobuf + schema registry
Practices
API-first design, schema checks at boundaries, compatibility tests, central
catalog + lineage.
4) Total Cost of Ownership (TCO)
Cost categories
Direct (licenses, compute, storage, network), Operational (people,
training), Hidden (egress, idle, lock-in). Unit economics
Cost per row, query, dashboard, model training, GB stored/month.
Optimization playbook
FinOps (budgets, alerts, showback), right-sizing, autoscaling, tiered
storage, caching, reduce cross-region traffic.
5) Build vs Buy
Build when
Differentiating capability, strict constraints, roadmap control. Buy when
Commodity capability, need speed, limited ops bandwidth. Reality
Build + buy: buy platform, build domain logic; managed infra + open
formats. Risk mitigations
Require export APIs, PoC with success criteria, document exit strategy.
6) Immutable vs Transitory Technologies
Invest deeply in
SQL, data modeling, distributed systems basics, security/privacy
principles, testing/observability. Treat as replaceable
Vendor products, proprietary formats, framework variants, hosted vs selfhosted.
7) The Modern Data Stack (2025)
Core components
Ingestion: Fivetran, Airbyte, Stitch; Debezium; Kafka/Pulsar/Kinesis.
Storage: Object store + Iceberg/Delta/Hudi; warehouses when needed.
Processing: SQL engines (warehouse SQL, Trino); Spark/Flink/Beam;
dbt.
Orchestration: Airflow, Dagster, Prefect; cloud schedulers.
Serving: BI tools (Looker, Tableau, Power BI), semantic layers, reverse
ETL, feature stores, vector DBs.
Governance/Security/Observability: DataHub/OpenMetadata, Great
Expectations/Soda, IAM, secrets, policy-as-code.
Note: Some modern platforms consolidate ingestion → transform → delivery.
Validate maturity + exit paths first.
8) Technology Decision Framework
Evaluation criteria
Functionality, performance, scalability, operability, cost/TCO, skills,
ecosystem, interoperability, security/compliance, reliability, vendor risk.
PoC checklist
Realistic dataset + concurrency, freshness/latency/cost targets, failure
injection, migration/export tests, operability drill.
Decision doc template
# Technology Selection: [Tool/Platform]
## Context
Constraints, non-functionals, current pain.
## Options Evaluated
1) [Option A] – Pros/Cons
2) [Option B] – Pros/Cons
3) [Option C] – Pros/Cons
## Decision
Selected + rationale.
## PoC Results
Performance, cost, operability, risks.
## Migration Plan
Phases, backfill, rollback.
## Success Metrics
Adoption, SLOs, cost per unit.
9) Lifecycle-Stage Recommendations
Generation: Data contracts, CDC, audit logging. Tools: schema registry,
Debezium.
Ingestion: Batch + stream, idempotent. Tools: ELT connectors,
Kafka/Pulsar/Kinesis.
Storage: Lakehouse on object store, tiered. Tools: Iceberg/Delta/Hudi.
Processing: SQL-first + tests, modular. Tools: dbt, Spark/Flink, Trino.
Serving: Semantic layer, governed access, caching. Tools: BI, reverse
ETL, feature stores.
Cross-cutting: Orchestration (DAGs), observability, zero-trust security.
10) Case Studies
E-Commerce (small team, fast growth)
Managed stack: Snowflake/BigQuery + dbt Cloud + hosted orchestration.
Trade-off: Higher SaaS cost, low ops, fast TTFI. Financial Services
(compliance, real-time)
Hybrid: Kafka on-prem, cloud analytics, open formats, strict KMS.
Trade-off: Higher ops, better control/compliance. Tech Startup (costsensitive)
OSS-first: S3 + Iceberg, Spark/Flink, Airflow, Metabase.
Trade-off: Lower license cost, more ops toil. Lessons: Match architecture
to constraints; keep stack minimal; measure cost/reliability early.
11) Future-Proofing
Bet on standards/open formats.
Abstract via interfaces; keep logic portable.
Document ADRs and exit strategies.
Invest in transferable skills.
Re-evaluate periodically with metrics.
Warning signs: Pricing shocks, weak community, performance cliffs, ops toil,
integration/security gaps, hiring difficulty.
Key Takeaways
Architecture > hype; constraints drive tooling.
Speed to value with guardrails.
Interoperability + open standards preserve options.
TCO = direct + ops + hidden costs.
Build + buy hybrid is the norm.
Learn concepts, not tools.
Validate choices with real PoCs.
Chapter 5: Data Generation in Source Systems — Refined Notes
Why this matters
You can’t engineer reliable data if you don’t understand its origins. Robust
pipelines start with deep knowledge of source system behavior: how data is
created, changed, exposed, and constrained.
1) What is a Source System?
A source system is any origin of data your platform ingests: apps and
services, databases, files, devices, or third-party APIs.
Traits
Ownership: usually product/ops teams, not DE
Diversity: from CSVs to microservices
Variability: formats, latencies, SLAs
Autonomy: independent release cycles
Why you must understand
Generation patterns, frequencies
Consistency & availability guarantees
Extraction impact on prod
Schema evolution model
Security & access requirements
2) Major Source Categories
A) Files & Unstructured Data
Practices: atomic writes, size checks, checksums; columnar formats;
event notifications; incremental loading.
Pitfalls: partial files, encoding issues, implicit schemas, filename drift.
B) APIs (SaaS, endpoints)
Practices: respect rate limits; pagination/deltas; pin versions; verify
webhooks; cap GraphQL queries.
Pitfalls: silent changes, long pulls, over/under-fetching.
C) Application Databases (OLTP)
Practices: use replicas; consistent snapshots; CDC for near-real-time;
minimize locks; indexed queries.
Pitfalls: replica lag, bloated WAL/binlogs, ignoring soft deletes.
D) OLAP / Warehouses
Practices: snapshot metrics/defs; manage egress cost; avoid double
aggregation.
Pitfalls: treating derived data as ground truth, metric drift.
E) Change Data Capture (CDC)
Methods: log-based (preferred), query-based, trigger-based.
Practices: initial load + stream, preserve order, capture schema changes,
idempotent upserts, support replay.
Pitfalls: unbounded log retention, missing deletes, out-of-order changes.
Tools: Debezium, Maxwell, cloud CDC services.
F) Queues & Streaming
Practices: partition keys, schema registry, DLQs, commit offsets postwrite.
Pitfalls: hot partitions, assuming global ordering, neglecting schema
governance.
G) IoT Devices
Practices: MQTT/CoAP; gateways; capture event time; edge filtering;
time-series DBs.
Pitfalls: duplicates, clock skew, weak security (certs, OTA updates).
3) Time Semantics
Event time = when event occurred (truth)
Ingestion time = when entered platform
Processing time = when processed
Practices: capture event time in UTC, watermarking + lateness handling,
event-time partitioning, aligned windows.
Pitfalls: time zone surprises, clock skew, using ingestion as event time.
4) Data Quality at Source
Dimensions: completeness, accuracy, consistency, timeliness, validity.
Practices: validate at ingest; row-level provenance; sample & profile;
automated tests (freshness, nulls, anomalies).
Tools: Great Expectations, Soda, DataHub, OpenMetadata.
5) Schema Management & Contracts
Models: backward, forward, full compatibility.
Practices: enforce via registry; explicit versioning; contract tests in CI; use
JSON Schema or Avro/Protobuf.
Pitfalls: silent repurposing, dropped fields without deprecation.
6) Security & Access
Principles: least privilege, secrets management, network controls,
encryption, audit logging.
Special: webhook sig verification, IoT device identity + cert rotation.
7) Performance Impact on Sources
Mitigations: read replicas, off-peak extracts, CDC, monitor DB internals.
Signals: CPU/IO, locks, lag, error rates.
8) Modern Patterns (2025)
Event-sourced microservices (Outbox).
Real-time by default (CDC + streaming).
Streaming-capable tables (MERGE in Iceberg/Delta/Hudi).
Unstructured data surge → OCR/NLP → structured annotations.
AI/LLM telemetry: treat prompts/responses as first-class, with PII controls.
Cloud-native APIs: cost/egress constraints.
9) Best Practices
1. Understand before building (patterns, SLAs, owners).
2. Design for change (schemas evolve).
3. Monitor freshness, volumes, schema drift.
4. Collaborate with owners (SLAs, dashboards).
5. Optimize for source (CDC > heavy queries).
6. Idempotent delivery (event IDs, merges).
7. Plan backfills & replays (retain logs, raw events).
10) Case Studies
E-Commerce: orders (CDC), events (stream), APIs (pull). Challenge:
timing alignment.
IoT Manufacturing: sensors via MQTT + edge DBs. Challenge:
connectivity, drift.
Financial Services: banking CDC, market feeds, batch filings. Challenge:
cross-source consistency.
11) Decision Framework
Source
Type
When to Use
Method
Key Considerations
Files
Periodic
exports/logs
Event loaders
Atomicity, schema
validation
APIs
SaaS/3rd party
Pulls, webhooks
Rate limits, versions,
dedupe
OLTP
Transactional truth
CDC, replicas
Consistency, lag, impact
OLAP
Derived data
Scheduled queries
Metric defs, egress cost
Streams
Real-time events
Continuous
consumption
Partitioning, DLQs,
schema
IoT
Telemetry
Edge + stream
Connectivity, event time
12) Key Takeaways
1. Sources aren’t generic plumbing—understand them.
2. Each type has unique extraction trade-offs.
3. CDC is the gold standard for DBs.
4. Time semantics matter—event time is king.
5. Schema evolution is inevitable—contract it.
6. Security/least privilege are non-negotiable.
7. Monitor everything, from freshness to replica lag.
Chapter 6: Storage — Refined Notes
Why this matters
Storage is the substrate of the entire data lifecycle. Every choice you make
about how data is stored impacts ingestion, transformation, serving, cost,
security, and reliability. In practice, “storage” is not a single stage—it’s
everywhere.
1) Storage Fundamentals
Where storage shows up
Source: OLTP/OLAP systems, SaaS buckets, device buffers
Ingestion: staging areas, queues/streams, spill files
Processing: checkpoints, shuffle/caches, intermediate tables
Serving: materialized views, dimensional/feature tables, indexes
Access patterns
OLTP: small, random reads/writes; high IOPS
OLAP: large scans; column pruning; predicate pushdown
Streaming: append-heavy; ordered logs; replay
Time series: sequential writes; range queries; downsampling
Vector search: numeric arrays; approximate indexes; low latency
2) Raw Ingredients: Media, I/O, and Encoding
Storage media
RAM: nanoseconds; caches/state stores
SSD/NVMe: microseconds; OLTP, shuffle, temp storage
HDD: milliseconds; cold tiers, backups
Object storage: durable, elastic, high latency; lakes/backups
Tape: archival/compliance
I/O path
Latency = media + network + protocol + engine
Cross-AZ/region hops add latency + cost
Serialization & formats
Row: CSV/TSV (simple, inefficient), JSON/NDJSON (flexible, verbose),
Avro (compact, schema evolution)
Column: Parquet/ORC (compression, pushdown), Arrow (in-memory IPC)
Choose by workload:
Streaming/logs → Avro/Protobuf
Analytics at rest → Parquet/ORC
In-memory → Arrow
Compression
Snappy/LZ4 = fast, lower ratio
ZSTD/Gzip = slower, higher ratio
Avoid double compression (Parquet already compresses)
3) Storage System Types
File: POSIX, rename atomic (NFS/HDFS)
Block: volumes for DBs/VMs (EBS/disks)
Object: flat namespace; rename = copy+delete (non-atomic)
Single machine: simple, limited
Distributed: replication, erasure coding, trade-offs
Consistency models
Strong (read-your-writes)
Eventual (high availability, eventual convergence)
Causal/prefix (causality preserved)
Note: cloud object stores → strong read-after-write per object, not
directories
4) Modern Storage Architectures
Warehouses (OLAP)
Columnar, ACID, separation of compute/storage, governance
Fit: curated analytics, BI, reporting
Data lakes
Raw/unstructured, schema-on-read
Pros: cheap, flexible
Cons: governance, small files, consistency issues
Lakehouse
Object store + table format + engine + governance
Formats: Iceberg, Delta, Hudi, Paimon
Benefits: ACID on object store, schema/partition evolution, time travel,
concurrent reads/writes
Format highlights
Iceberg: hidden partitions, partition evolution, REST catalogs
Delta: transaction log, Z-ORDER, Change Data Feed, UniForm
Hudi: copy-on-write + merge-on-read; incremental queries; streamingfriendly
Paimon: streaming-first, Flink-oriented
Governance & catalogs
Hive Metastore/Glue, Iceberg REST/Nessie, Unity
Enforce RBAC/ABAC, masking, lineage
Streaming storage
Kafka/Pulsar/Kinesis = durable logs, retention, compaction
Tiered storage = older → object store
Exactly-once = idempotence + transactions + checkpointing
Time-series/logs
Timescale, Influx, ClickHouse, OpenSearch
Retain in lakehouse for history
Vector
pgvector, Milvus, Weaviate, Pinecone
Indexes: HNSW, IVF/PQ
Store raw docs separately; manage drift
5) Cloud Patterns
Separation of compute/storage: independent scaling, multi-engine reads
Tiered storage: hot → warm → cold → archive
Multi-region/edge: replication for DR; sovereignty constraints
AI-assisted ops: anomaly detection, auto-tiering, partitioning suggestions
6) Performance Engineering
Partition by event time + ≤2 business keys
File size: 128–1024 MB; compact often
Sort/clustering: range sort, Z-order
Use stats (min/max, bloom) for pruning
Append-only preferred; MERGE only if needed
Streaming: checkpoint SSD/RAM → object storage
Throughput: multi-part transfers, parallel I/O, co-locate compute + storage
7) Cost Engineering & Sustainability
Hidden costs
Egress, request charges, small files, hot storage for cold data
Playbook
FinOps guardrails
Retention & lifecycle policies
Format choice: Parquet+ZSTD (cold), Snappy (interactive)
Compact/cluster on cadence
Push-down queries vs exports
Factor in carbon impact (region, scheduling)
8) Reliability & Durability
Replication vs erasure coding
Backups: versioning, Object Lock (WORM)
PITR and tested restores
End-to-end checksums
DR: replication + tested RPO/RTO
Idempotence: change IDs, dedupe, 2PC/checkpointing
9) Security, Privacy & Governance
Encryption at rest/in-transit; KMS/HSM
IAM least privilege; short-lived creds
Row/column-level security; masking/tokenization
DLP scanning, tags/policies
Immutable audit logs
Tips
Prefer bucket policies/IAM over object ACLs
Private endpoints > public egress
Separate prod/non-prod buckets
10) DataOps for Storage
IaC for buckets/catalogs/policies
CI/CD for schemas + table changes
Scheduled housekeeping: compaction, VACUUM, retention
Observability: errors, request rates, egress, file sizes
Lineage: catalog all tables, attach SLOs
11) Decision Framework
Factor
Key Questions
Why It Matters
Volume & Growth
Current + 24m growth?
Capacity, tiering
Access Patterns
OLTP/OLAP/stream/time-series/vector?
Media + format choice
Latency/Throughput
SLOs, concurrency?
Engine, partitioning
Consistency
Strong/eventual? Snapshots?
Correctness
Cost/TCO
Storage + requests + ops
Budget + FinOps
Compliance
Retention, residency, WORM?
Region/legal
Skills & Ops
Managed vs self-hosted?
Reliability, velocity
Interop
Open formats, multi-engine?
Future-proofing
Patterns
Lambda: accurate, heavy ops
Kappa: stream-first, replayable
Lakehouse: one copy for all
Data Mesh: domain-owned storage + federated governance
Heuristics
Default = lakehouse + open formats
Raw zone (append-only), curated zone (ACID), one serving path per use
case
12) Case Studies
E-Commerce
Redis cache + Postgres OLTP + S3 Parquet + Kafka
Sub-second Black Friday responses; cheap long-term analytics
Financial Services
Warehouse recent + Parquet historic with Object Lock
70% cost down, 7-year retention with restores
IoT Manufacturing
Edge → MQTT → time-series DB + lakehouse; ClickHouse real-time
Real-time anomaly detection + cheap multi-year history
13) Future Trends
GPU-direct object storage for AI/analytics
Serverless object classes (low latency)
Persistent memory/CXL, NVMe-oF
Pushdown near data
Zero-copy analytics (Arrow/Flight)
Multi-cloud abstractions + portable catalogs
Key Takeaways
1. Storage is everywhere—optimize for access, not just capacity.
2. Open table formats add ACID + evolution + multi-engine access.
3. File hygiene (partition, sort, compact) drives performance + cost.
4. Cost, security, reliability = first-class concerns.
5. Automate lifecycle + housekeeping; document SLOs/ADRs.
6. Choose reversible, interoperable options—future needs will change.
Chapter 7: Ingestion — Refined Notes
Why this matters
Ingestion is the bridge between data generation and data value. If it’s fragile or
leaky, every downstream stage inherits pain: missing data, late data, corrupted
data, and cascading failures. Get ingestion right, and everything else is easier.
1) What is Data Ingestion?
Ingestion: reliable movement of data from source → destination
Integration: combining/alignment from multiple sources
Transformation: modifying structure/content with business logic
Why separate?
Clear responsibilities
Better reliability (less side-effects)
Easier scaling + testing
2) Core Engineering Considerations
Bounded vs unbounded: finite batch vs continuous streams
Frequency: daily/weekly (batch) vs sub-second/minute (stream) vs hybrid
Sync vs async: caller waits (simple but bottlenecks) vs fire-and-forget
(scales but needs retries/dedup)
Push vs pull: push (real-time, must buffer) vs pull (controlled, higher
latency) vs hybrid (notify + fetch)
Idempotency: sinks must accept replays safely (keys, sequence,
MERGE)
Contracts: enforce schema + compatibility at ingress; quarantine bad data
3) Batch Ingestion Patterns
Snapshot vs incremental
Snapshot → simple but heavy; use for small tables or rare backfills
Incremental → efficient but requires change tracking (timestamps,
audit tables, hashes, CDC)
Windows/micro-batching
Tumbling, sliding, session windows
Micro-batching balances latency vs efficiency
Large-scale tactics
Pagination/chunking
Parallel reads across partitions
Checkpoints for resumability
Protect sources (replicas/export jobs)
File hygiene
Write temp → rename or use manifests
Checksums, counts, schema validation
Idempotent handling of duplicates/late files
4) Streaming Ingestion Patterns
Key concepts: event vs processing time, watermarks, windowing, durable
state stores
Backpressure: bounded queues, rate limiters, adaptive consumers
Exactly-once: idempotent I/O + transactional sinks + checkpointing
Frameworks: Kafka, Pulsar, Kinesis, Pub/Sub + Flink, Spark Streaming,
Beam
Schema/contracts: Avro/Protobuf + schema registry; version events, no
field repurposing
Outbox pattern: avoid dual-write by persisting domain events alongside
business transactions
5) Modern Ingestion Technologies (2025)
Managed ELT: Fivetran, Airbyte, Stitch, Meltano → quick but check pricing
+ lock-in
CDC: Debezium, Qlik, AWS DMS → prefer log-based
Streaming services: Kafka, Kinesis, Pub/Sub → mind SLAs, cross-region
costs
Unified batch + stream: Flink, Beam, Materialize → single semantics
AI-enhanced ingestion: schema mapping, anomaly detection, predictive
scaling
6) Best Practices
Quality at edge: schema/type checks, DLQs for bad data
Error handling: retries with backoff+jitter, circuit breakers, frequent
checkpoints
Monitoring: throughput, freshness, quality, system health, traces
Security: least privilege, short-lived creds, TLS, field masking, audit logs
Cost controls: API budgets, co-location, compression, batching
Governance: contracts, SLAs, lineage, ownership
7) Practical Ingestion Patterns
CDC pipeline
Source: PostgreSQL
Method: Log-based CDC (Debezium)
Steps:
1) Snapshot tables
2) Stream WAL changes → Kafka
3) Enrich + mask PII
4) Sink to lakehouse + warehouse
API ingestion
Source: SaaS API
Method: Incremental pull + pagination
Steps:
1) Scheduler runs
2) Calls with cursors/backoff
3) Validate + dedupe
4) Stage to object store
5) MERGE curated tables
IoT streaming
Source: Devices (MQTT)
Method: MQTT → Bridge → Kafka → Storage
Steps:
1) Devices publish with auth
2) Bridge buffers
3) Stream processor downsamples
4) Hot tier: time-series DB; Cold tier: Parquet
File drop zone
Source: S3 drop
Method: Event-driven manifest load
Steps:
1) Producer uploads + checksum + _SUCCESS
2) Loader triggers on manifest
3) Validate schema + dedupe
4) Commit curated partition
8) Case Studies (condensed)
E-commerce: Kafka (clickstream) + CDC (orders) + batch API (inventory)
→ ~5 min latency, 99.9% reliability
Financial services: streaming + exactly-once + encrypted/audited
pipelines → sub-second fraud detection
Healthcare IoT: MQTT + Kafka + edge filtering → real-time alerts, 7-year
HIPAA retention
9) Decision Framework
Use Case
Approach
Tech
Notes
Daily reports
Batch
Orchestrator + SQL
Cheap, throughput
User analytics
Micro-batch
Spark/Flink streaming
Balance latency
Fraud
detection
Real-time
Kafka + Flink/Beam
Event-time, exactly-once
Migration
Snapshot +
CDC
Debezium/DMS
Cutover smoothly
IoT telemetry
Edge+streaming
MQTT + Kafka
Handle bursts,
downsample
SaaS API
Pull
Airbyte/Fivetran/custom
Respect quotas
Heuristics
Choose simplest pattern that meets SLOs
Prefer CDC for DBs, webhooks for events, manifests for files
Keep ingestion “just movement + minimal validation”
10) Future Trends
Serverless ingestion (pay-per-event, auto-scale)
Semantic-aware ingestion (contracts enforced at edge)
Multi-cloud neutral connectors
Real-time contracts + SLA monitoring
Edge intelligence (WASM/agents for local validation + compression)
Key Takeaways
1. Match ingestion pattern to requirements.
2. Build idempotency + replay into every pipeline.
3. Validate at edge; quarantine failures early.
4. Monitor freshness, lag, quality, cost continuously.
5. Prefer managed services for speed; open formats for flexibility.
6. Expect change: schemas drift, rates spike—design for it.
Ingestion Readiness Checklist
SLOs (freshness, completeness, reliability, cost)
Source contract (schema, auth, rate limits)
Idempotent writes + dedup keys; replay strategy
Schema validation + DLQ/quarantine
Checkpointing + backoff/retry + circuit breakers
Observability (lag, throughput, errors, traces)
Security (least privilege, TLS, secrets rotation, audit logs)
Cost guardrails (budgets, egress, batching/compression)
Runbooks (failures, backfills, replays)
DR tested from snapshot/checkpoint
Chapter 8: Queries, Modeling, and Transformation — Refined
Notes
Why this matters
Storage and ingestion set the stage, but queries, modeling, and
transformation create the value. This is where you encode business logic,
ensure quality, and produce reusable, trustworthy data products. Good
designs enable self-service and speed; bad ones accrue technical debt that
slows everyone down
1) The Power of Queries
Good queries are:
Correct → accurate results
Fast enough → meet latency/cost SLOs
Readable → clear naming, comments, CTEs
Reusable → parameterized, modular, tested
Optimization basics
1. Query plans → inspect with EXPLAIN ; watch for full scans, large shuffles
2. Index/layout → B-tree, bitmap, covering, partial indexes
3. Joins → filter early, order by selectivity, hash for large sets
4. Partitioning/pruning → critical in warehouses/lakehouses
EXPLAIN (ANALYZE, BUFFERS)
SELECT c.customer_id,
COUNT(DISTINCT o.order_id) AS order_count,
SUM(o.total_amount)
AS lifetime_value
FROM customers c
LEFT JOIN orders o ON c.customer_id = o.customer_id
WHERE c.created_date >= '2024-01-01'
GROUP BY c.customer_id;
Modern SQL
Window functions (totals, ranks, gaps)
CTEs for multi-step logic
Grouping sets/rollups for multidimensional analysis
Arrays/maps/JSON functions
Approximate aggregations (HLL, approx_percentile)
Anti-patterns
SELECT * in production
Non-sargable filters ( WHERE YEAR(date)=... )
Accidental cross joins
Blind DISTINCT on huge data
Nested CTEs with no materialization
Warehouse tips
BigQuery: partition+cluster; avoid SELECT * ; slot mgmt
Snowflake: micro-partition pruning, clustering, tasks/streams
Delta/Databricks: OPTIMIZE+Z-ORDER; VACUUM; maintain stats
Redshift: dist/sort keys; concurrency scaling
2) Data Modeling: From Concept to Delivery
Three levels
Conceptual → business entities
Logical → attributes, relationships
Physical → types, partitions, indexes
Dimensional modeling
Star schema: fact + dimensions
Snowflake: normalized dims (space-efficient, more joins)
SCD types: Type 1 (overwrite), 2 (versioned), 3 (limited history)
CREATE TABLE fact_sales (
sale_id
BIGINT PRIMARY KEY,
date_key
INT REFERENCES dim_date(date_key),
product_key INT REFERENCES dim_product(product_key),
customer_key INT REFERENCES dim_customer(customer_key),
store_key
INT REFERENCES dim_store(store_key),
quantity
INT,
amount
DECIMAL(10,2),
discount
DECIMAL(10,2)
);
Modern approaches
Data Vault 2.0: hubs+links+satellites → auditability, agility
Activity schema: single event fact table; good for product analytics
One Big Table: denormalized, simple queries; duplication risk
Semantic/metrics layers
Define metrics once; expose everywhere
Tools: dbt metrics, MetricFlow, Cube, LookML
metrics:
- name: monthly_recurring_revenue
label: MRR
model: ref('subscriptions')
type: sum
sql: amount
timestamp: subscription_date
time_grains: [day, week, month, quarter, year]
dimensions: [plan_type, region, segment]
Principles
Stable surrogate keys, keep natural keys
Explicit null handling
Conform common dimensions (customer/product)
Document grains + SLOs
3) Transformation: Patterns and Paradigms
ETL vs ELT
ETL = old, transform before load
ELT = dominant, transform in warehouse/lakehouse
Layered approach
Raw → Staging → Intermediate → Marts/Semantics
Raw immutable, staging cleans, marts join/enrich
Batch patterns
Incremental (high-water marks)
SCD handling
Idempotent upserts
MERGE INTO target t
USING (SELECT * FROM source WHERE updated_at > (SELECT MAX(ts)
FROM watermark)) s
ON t.id = s.id
WHEN MATCHED THEN UPDATE SET ...
WHEN NOT MATCHED THEN INSERT (...);
Streaming patterns
Event-time + watermarks
Windowed aggregations
Stateful ops (Flink/Kafka Streams)
Exactly-once sinks
Outbox pattern
Write events with transactions
Relay → Kafka/stream; avoid dual-write bugs
4) Modern Transformation with dbt
dbt core
Models as code, refs for deps, tests/docs, snapshots
Staging → intermediate → marts
{{ config(materialized='incremental', unique_key='order_id') }}
SELECT order_id, order_date, total_amount
FROM {{ ref('stg_orders') }}
{% if is_incremental() %}
WHERE order_date > (SELECT MAX(order_date) FROM {{ this }})
{% endif %}
Tests/docs
models:
- name: fct_orders
columns:
- name: order_id
tests: [unique, not_null]
- name: total_amount
tests:
- not_null
- dbt_utils.accepted_range: {min_value: 0, max_value:
1000000}
Other tools
SQLMesh, Malloy
Spark/Flink for large scale/stream
DuckDB/Polars for fast local dev
Dagster/Airflow/Prefect for orchestration
Great Expectations, Soda, Monte Carlo for quality
5) Data Quality and Contracts
Dimensions: completeness, accuracy, consistency, timeliness, validity,
uniqueness
Implementations
Tests at boundaries (dbt, GE)
Profiling + drift detection
Quarantine bad data (DLQs)
Contracts: schemas + SLAs + compatibility rules
name: orders_contract
sla: {freshness: 1 hour, quality: 99.9%}
schema:
- {name: order_id, type: string, unique: true, nullable:
false}
- {name: amount,
type: decimal, min: 0, nullable: false}
Privacy
Mask/tokenize PII early
Apply row/column security
Audit sensitive transforms
6) Performance, Cost, Operability
Partition by event time, cluster on hot predicates
Columnar file sizes: 128–1024 MB; compact often
Filter pushdown; use correct types
Pre-aggregate hot queries
Approx functions where acceptable
Cache hot datasets; avoid cross-region scans
Incremental models; right-size compute
Reliability
Idempotent writes, dedupe by keys
Checkpoints, backfills, replay plans
CI/CD, lineage, contracts, SLOs
Document business logic in-line
7) AI-Enhanced Transformation (2025)
AI helps propose joins/mappings
Detect schema drift/anomalies
Natural language → SQL (review before prod)
Treat outputs as drafts; enforce tests/contracts
8) Case Studies
E-Commerce
Raw Parquet → dbt → Flink real-time → Snowflake
Sessionization, Customer 360, real-time inventory
Result: 10× faster queries, reports 2h → 5m
Financial Risk
Kafka ingestion → Flink (real-time risk) → Spark (history) → lakehouse
Positions, VaR, anomaly detection
Result: sub-second risk updates, 99.99% accuracy
9) Decision Framework
Scenario
Approach
Tools
BI analytics
SQL-first ELT + marts
dbt + Snowflake/BigQuery/Trino
Real-time metrics
Event-time streaming
Flink/Beam + Kafka
ML features
Batch+stream + registry
Spark/Flink + Feast
Scenario
Approach
Tools
Large joins/history
Lakehouse + clustering
Delta/Iceberg/Hudi + Spark
Quality gates
Contracts + tests
dbt tests, GE, Soda
Ad-hoc dev
Local OLAP
DuckDB, Polars
Heuristics
Default to ELT, layered models
Open table formats for interop
Simplest materialization that meets SLOs
10) Future Trends (2025–2026)
Lakehouse-native transforms
Declarative pipelines
Real-time default (materialized views)
AI-assisted development
Key Takeaways
1. SQL = workhorse for analytics
2. ELT dominates for cost/flexibility
3. Model with intent (grain, keys, marts)
4. Quality built-in (tests, contracts, quarantine)
5. Idempotent + incremental transforms; plan replays
6. Optimize for access patterns; maintain hygiene
7. Treat models as prod code (CI/CD, docs)
Chapter 9: Serving Data for Analytics, Machine Learning, and
Reverse ETL — Refined Notes
Why this matters
Serving is where the lifecycle delivers value. If data isn’t trustworthy,
discoverable, and fast to consume, even perfect upstream pipelines won’t
move the needle. Treat served datasets as products with owners, SLAs,
documentation, and telemetry.
Data Serving Architecture
Data Serving Architecture
1) Principles of Effective Data Serving
Trust as the foundation
SLAs/SLOs: freshness, availability, correctness
Lineage and explainability
Quality gates + canary/shadow releases
Know your consumers
Analysts, operators, DS/ML, external partners
Access patterns: SQL, API, file share, stream
Self-service: semantic/metrics layer, docs
Data as a product
Ownership, versioning, catalog discoverability
Adoption/impact metrics
Performance/scale
Align layout to queries
Pre-compute/caches for hot paths
Concurrency + cost control
Security/governance
RLS/CLS, masking, ABAC
Audit logs, data classification
Latency modes
Batch (mins–days), On-demand (sub-sec–sec), Real-time (ms–low sec)
2) Serving for Analytics and BI
Use cases
Business analytics, operational analytics, embedded dashboards
Core patterns
-- Semantic view
CREATE OR REPLACE VIEW analytics.v_customer_metrics AS
SELECT c.customer_id,
COUNT(DISTINCT o.order_id) AS lifetime_orders,
SUM(o.amount)
AS lifetime_value,
MAX(o.order_date)
AS last_order
FROM dim_customer c
LEFT JOIN fact_orders o USING (customer_id)
GROUP BY 1;
-- Materialized view
CREATE MATERIALIZED VIEW analytics.mv_daily_sales AS
SELECT DATE(order_date) AS sale_date,
product_category,
SUM(amount) AS total_sales
FROM fact_orders
GROUP BY 1,2;
Semantic/metrics layer: dbt Semantic Layer, MetricFlow, Cube, LookML
Performance playbook: partition/prune, cache hierarchy, pre-aggregate
common grains
Anti-patterns: SELECT * , ad-hoc joins on raw, unbounded highcardinality in BI tools
BI tools 2025: Looker, Tableau, Power BI, Mode/Hex, ThoughtSpot (+ AI
copilots)
3) Serving for Machine Learning
ML needs
Training (historical), real-time inference, feature engineering
Patterns
Feature stores: offline+online, point-in-time correctness
Training/serving consistency: same transformations, time travel for
reproducibility
Model registry/serving: track versions, metrics, lineage
from feast import FeatureStore
fs = FeatureStore(".")
features = fs.get_online_features(
entity_rows=[{"user_id": "123"}],
features=["user:age","user:region","txn:avg_amount_30d"]
).to_dict()
Observability
Latency (P50/P95), errors, drift, logging with correlation IDs
4) Reverse ETL (Operational Activation)
Definition: Sync warehouse insights → CRMs, CDPs, ops tools
Patterns
Scheduled syncs, event-triggered syncs, API upserts with idempotency
def sync_segments():
rows = warehouse.query("SELECT ... WHERE updated_at > {{
last_sync }}")
for batch in chunk(rows,1000):
crm_api.upsert_customers(batch)
update_checkpoint()
# idempotent
Tooling: Hightouch, Census, RudderStack, Airbyte OSS
Gotchas: API limits, PII, idempotency, contract-driven mappings
5) Serving Methods and Technologies
File exchange (CSV/Parquet, SFTP, signed URLs)
Direct DB/warehouse access (governed, RLS, reader accounts)
APIs (REST/GraphQL/gRPC) → versioning, rate limiting, persisted queries
Streaming feeds (Kafka/Pulsar, SSE/WebSockets)
Data sharing (Snowflake Secure Share, Delta Sharing, BigQuery sharing)
Federation (Trino, external tables)
Performance choices: materialize heavy joins, semantic caches, pre-warm
caches
6) Undercurrents Applied to Serving
Security: RBAC/ABAC, masking, encryption, private endpoints
Data mgmt: catalog, contracts, publish-time checks
Reliability: CI/CD for views/APIs, canary releases, usage telemetry
Architecture: centralized vs mesh, dependency-aware DAGs
Software eng: code reviews, docs, runbooks
7) Modern Context (2025)
AI-augmented serving: NLQ copilots, auto-dashboards, narrative BI
Unified serving: lakehouse-native, multi-modal access, zero-copy shares
Observability/governance: per-query cost attribution, SLAs, policy-ascode
8) Best Practices
Self-service: semantic layer + docs
Performance: profile queries, precompute hot paths, cache
Quality gates: validate before publish, canary, monitor drift
Scale: plan for 10× users, quotas, autoscaling
Optionality: open formats, zero-copy
Snippets
# Reverse ETL with DLQ
for batch in batches:
resp = crm_api.bulk_upsert(batch,
idempotency_key="customer_id")
if not resp.ok: dlq.write(batch, reason=resp.error)
# Feature serving API
@app.get("/features/{user_id}")
def get_features(user_id:str):
return feature_store.get_online_features(
entity={"user_id": user_id},
features=["user.age","user.region"]
)
9) Case Studies
Retail analytics: Snowflake + Databricks + Hightouch + Tableau → fewer
stockouts, 500+ self-service users
Healthcare ML: Tecton + Vertex + Census → ms-latency scores, HIPAA
compliance, earlier interventions
10) Decision Framework
Use Case
Method
Tools
Notes
BI dashboards
Queries + semantic layer
Looker, Tableau,
dbt/Cube
Precompute; RLS
Ops analytics
Real-time
materializations
Flink, Materialize
Low latency
ML training
Offline feature sets
Lakehouse +
Feature Store
Time travel,
versioning
Inference
Online features + model
serving
Feast, Tecton,
Vertex
P95 latency, drift
Reverse ETL
Batch/Event sync
Hightouch, Census
Idempotency, PII
Data sharing
Zero-copy
Snowflake, Delta
Sharing
Governance
Partner
exchange
Files/API
Parquet + signed
URLs
Contracts,
cadence
Heuristics
Prefer zero-copy over export
Materialize heavy queries
Isolate analytical vs operational loads
11) Serving Readiness Checklist
Consumers + SLAs documented
Semantic/metrics layer versioned
RBAC/ABAC, RLS/CLS, masking enforced
Quality gates + canaries
Partition/MV/cache plan in place
Monitor freshness, usage, latency, cost
Reverse ETL: idempotency, retries, DLQ
ML: offline/online parity, drift monitoring
Sharing: contracts, lineage, revocation
Runbooks + rollback tested
Key Takeaways
1. Serving = product; own, document, measure impact.
2. Trust = SLAs + lineage + semantic consistency.
3. Optimize for access patterns with precompute/caches.
4. ML serving needs point-in-time + parity.
5. Reverse ETL closes loop → handle idempotency/PII/limits.
6. Prefer zero-copy/open formats.
7. Observe usage, latency, errors, drift.
8. Evolve with canaries + telemetry.
Summary
Serving is where data turns into decisions and actions. Prioritize trust, selfservice, semantic consistency, and performance. Use the right patterns for
analytics (semantic layers/materializations), ML (feature stores), and activation
(reverse ETL). In 2025, AI-augmented and lakehouse-native serving make it
easier to deliver governed, real-time data products—if designed with reliability,
security, and cost in mind.
Chapter 10: Security and Privacy — Refined Notes
Why this matters
Security and privacy aren’t bolt-ons—they’re design constraints across the
entire data lifecycle. In a zero-trust, cloud-native world, data engineers share
direct responsibility for confidentiality, integrity, availability, and compliance.
Build systems that are secure by default, observable, and resilient to failure
and abuse.
Data Security Architecture
Data Security Architecture
1) Culture First: People and Mindset
Assume breach: every boundary is untrusted; design detection +
containment.
Make security habitual: audits, least-privilege reviews, key rotations, IR
drills.
Train by role: engineers → secure coding, secrets hygiene, encryption;
org-wide → phishing sims.
Safe reporting: blameless IR, clear escalation, celebrate early detection.
Security champions: embed in each team to bridge policy ↔ practice.
2) Processes and Governance
Least privilege + separation of duties: JIT/JEA access, break-glass with
audit.
Shared responsibility: cloud sec = provider infra + your data/IAM/configs.
Vulnerability mgmt: scans (containers, IaC, libs), patch SLAs,
CSPM/DSPM.
Incident response: detect → triage → contain → eradicate → recover →
learn.
Backup & recovery: 3-2-1 rule, immutable WORM, cross-region, test
restores.
Policies
Classification: Public, Internal, Sensitive, Restricted → map to storage,
access, retention.
Privacy by design: minimization, purpose limitation, consent,
retention/erasure.
Compliance: GDPR/CCPA/HIPAA/PCI, SOC2/ISO 27001, DPIAs.
Change mgmt: reviews for schema/pipeline/access; ADRs capture tradeoffs.
3) Technical Controls (Practical)
Identity & access
SSO + MFA, short-lived creds, OIDC federation for CI/CD.
RBAC/ABAC, policy-as-code (OPA/Rego).
Secrets managers (Vault, AWS Secrets Manager); rotate on leak.
Network & compute
Zero-trust networking: mTLS, private endpoints, service auth.
Egress controls, WAF, rate limiting.
Hardened images, SBOMs, SLSA; scan IaC + containers.
Data security
Encryption: at rest (KMS/HSM), in transit (TLS 1.2/1.3), in use (TEE).
Key mgmt: rotate, separate “use” vs “admin,” BYOK/HYOK if required.
Mask/tokenize/anonymize: prod vs dev/test, k-anonymity vs DP.
DLP: classify in motion/rest, quarantine leaks, tag datasets.
Platform specifics
Warehouse/lakehouse: RLS/CLS, secure catalogs, WORM, GDPR
deletes.
Streaming: ACLs on topics, TLS/SASL, schema registry auth.
Observability
Centralized immutable logs, SIEM/SOAR correlation, UEBA anomaly
detection.
Monitor failed logins, privilege escalations, egress anomalies, query
spikes.
API & edge
OAuth2/OIDC, scopes, rate limiting, schema-first APIs.
Dev endpoints/laptops: MDM, disk encryption, patched agents.
4) Security Across the Data Lifecycle
Stage
Top controls
Ingestion
TLS + auth, schema/contract checks, DLQ/quarantine
Storage
KMS, RLS/CLS/policy tags, lifecycle + WORM
Transform
Mask/tokenize, sanitized dev data, CI contract tests
Serving
Semantic layer + RLS, API auth/rate limits, zero-copy sharing
Reverse ETL
Idempotency, consent/DPA, field minimization
ML
Feature ACLs, point-in-time data, drift/attack monitoring
5) Privacy Engineering
Minimization/purpose: don’t collect “just in case.”
Subject rights: automate access/erasure/portability; handle backups with
grace windows.
PETs: differential privacy, federated learning, HE/MPC, TEEs.
De-identification caveats: re-ID risk rises with linkage; DP = strong
guarantee.
6) Modern Context (2025)
Zero-trust maturity: passwordless (WebAuthn), device posture checks.
AI + security: SOAR playbooks, anomaly detection; but also AI-phishing,
prompt injection → apply I/O filtering and secret redaction.
DSPM + governance-as-code: auto-classify + enforce with OPA.
Post-quantum crypto: hybrid TLS (X25519+Kyber), track NIST PQC.
Supply chain sec: SBOMs, Sigstore/Cosign, SLSA provenance.
7) Best Practices and Checklists
Baseline
SSO + MFA, short-lived creds
IAM least privilege, access reviews, ABAC tags
Encrypt at rest/in transit; CMK for sensitive zones
Secrets in manager, rotation, secret scanning in CI
Deny public buckets, enforce private endpoints
Central logs → SIEM alerts, UEBA
Data classification + lifecycle policies
Sanitized dev/test data, masking in prod mirrors
3-2-1 backups, DR tested, object lock/WORM
IR runbooks + tabletop drills
Vendor security reviews, DPAs, exit/export paths
Contracts include classification, consent, retention
Dashboards to monitor
Auth anomalies, data egress spikes, schema drift, DLP hits, encryption
failures
Pitfalls
Over-permissive roles, public buckets, hardcoded secrets, retention vs
GDPR conflicts, logging PII, untested restores
8) Example Policies and Snippets
IAM (YAML)
policy:
name: data_engineer_role
permissions:
- resource: warehouse
actions: [read, write]
conditions:
- time: business_hours
- ip: corporate_network
- resource: sensitive_data
actions: [read]
conditions: mfa_required
S3 hardening (Terraform)
resource "aws_s3_bucket_public_access_block" "data" {
bucket
= aws_s3_bucket.data.id
block_public_acls
= true
block_public_policy
= true
ignore_public_acls
= true
restrict_public_buckets = true
}
OPA (Rego)
deny[msg] {
input.resource.type == "s3_bucket"
not input.resource.encryption_enabled
msg := sprintf("Bucket %s must enable encryption",
[input.resource.name])
}
Dynamic masking (SQL)
CREATE MASKING POLICY mask_email AS
(val STRING) RETURNS STRING ->
CASE WHEN CURRENT_ROLE() IN ('ANALYST_SECURED') THEN val
ELSE REGEXP_REPLACE(val,'(^.).*(@.*$)','\\1***\\2')
END;
9) Decision Framework
Scenario
Primary risks
Key controls
Tools
Cloud
migration
Misconfig, broad
IAM, egress
CSPM, least privilege,
KMS, private endpoints
AWS/Azure/GCP
IAM, KMS
ML pipelines
Leakage,
poisoning, drift
Feature ACLs, point-intime joins, DP, registry
Feast/Tecton,
MLflow
API serving
Abuse,
exfiltration
OAuth2/OIDC, scopes,
rate limits, WAF
API Gateway, Kong,
Okta
Batch
processing
Secrets, overreads
Secret mgr, scoped roles,
contract tests
Vault, dbt tests
Real-time
streaming
Topic snooping,
weak auth
TLS/SASL, ACLs, tenant
isolation
Kafka/Pulsar ACLs
Data sharing
Over-exposure
Zero-copy shares,
RLS/CLS, contracts
Snowflake, Delta
Sharing
Key Takeaways
1. Security is continuous: people + process + tech.
2. Defaults: least privilege, encrypt everywhere, short-lived creds.
3. Privacy: minimize, mask, tokenize, automate subject rights.
4. Observe everything: logs, SIEM/SOAR, anomaly detection.
5. Recovery: immutable backups, tested DR.
6. Govern as code: classification, ABAC, reproducible policies.
7. Stay current: zero-trust, DSPM, PQC, supply-chain hardening.
Summary
Security/privacy = undercurrents, not bolt-ons. Programs need culture,
governance, and controls (IAM, data, network, observability) across every
lifecycle stage. In 2025, zero-trust, PETs, DSPM, and supply-chain security
are table stakes. Design for least privilege and resilience, automate
compliance, and continuously test assumptions.
Chapter 11: The Future of Data Engineering — Refined Notes
Why this matters
Tools change fast; principles don’t. The data engineering lifecycle remains
your north star. The next wave is about simplification, live (real-time) systems,
tighter fusion with applications, and governing “dark matter” data—all while
keeping security, privacy, and cost as first-class concerns.
Future of Data Engineering
Future of Data Engineering
1) What Won’t Change: The Enduring Lifecycle
Lifecycle map: generation → storage → ingestion → transformation →
serving.
Undercurrents that persist: security/privacy, governance/metadata,
DataOps, FinOps, reliability/observability.
Principle-centric thinking:
Trade-offs: latency vs cost vs correctness.
Reversibility: avoid one-way doors; prefer open standards.
Loosely coupled architectures: smaller blast radius, faster iteration.
Practice tip: map every new idea back to the lifecycle; document decisions
with ADRs + explicit non-functionals.
2) Macro Trends (3–5 Years)
Simplicity & consolidation: fewer, broader platforms; managed services.
Interoperability: open formats, zero-copy sharing, federation.
Live data stack: unified batch/stream; materialized views; feedback loops.
Fusion with apps: data as APIs/metrics/semantics embedded in apps.
Decentralization with governance: pragmatic data mesh.
AI-augmented engineering: copilots for modeling/queries/tests; anomaly
detection; guardrails essential.
FinOps + sustainability: unit economics, carbon-aware scheduling.
Security & sovereignty: zero-trust, DSPM, PETs, PQ crypto readiness.
Edge & multi-cloud: near-source processing; standards for portability.
3) Architectures Likely to Endure
Lakehouse on open formats (Delta/Iceberg/Hudi).
Event-driven backbone + table changelogs (CDC/outbox).
Semantic/metrics layers (one definition of joins + KPIs).
Feature platforms for ML (offline/online parity).
Secure sharing & federation (zero-copy, governed).
4) Adoption Framework for New Tech
Score 1–5; weight by importance:
Problem fit, interoperability, operability, security/compliance, exit path,
cost/TCO, maturity, team fit.
PoC checklist
Realistic volume + concurrency
Freshness/latency/cost SLOs
Failure injection (network, schema drift, backlog bursts)
Interop tests (multi-engine read/write, export demo)
Operability drill (logs, tracing, runbook)
ADR template: Context → Options → Decision → Trade-offs →
Consequences → Revisit date.
5) Capability Maturity Models
A) Live Data Maturity
0: Batch only
1: Micro-batch; limited SLAs
2: Streaming for key domains; CDC
3: Unified batch/stream; materialized views; SLAs
4: Closed-loop, self-healing, autoscaling
B) Data Product Maturity
0: Tables, no owners
1: Owners + docs
2: Contracts + SLAs + lineage
3: Semantic APIs, versioning, telemetry
4: Marketplace, automated policies
6) Dark Matter Data (Spreadsheets, Docs, Email)
Catalog connectors; contracts (schemas, owners, cadence).
Sandboxes before promotion; guardrails: PII scan, masking, DLP.
Graduation path: popular sheets → managed data products.
7) People & Skills
Shift: from pipeline builders → lifecycle/product engineers.
T-shaped skills: data systems + literacy in software, security, ML, finance.
Business fluency: tie models to KPIs; co-design metrics.
AI as co-pilot: draft, then verify with contracts/tests.
Team topologies: stream-aligned domains, platform team, enabling team.
90-day plan
M1: Semantic layer + contracts
M2: CDC basics; incremental models; cost dashboards
M3: Security + FinOps guardrails; lineage + incident runbooks
8) Risks & Anti-Patterns
Platform sprawl (“best of breed”) → fix: curate golden paths.
“Real-time everything” → fix: align latency to value.
Semantic drift → fix: central definitions + contract tests.
AI over-trust → fix: treat as drafts, enforce reviews.
Proprietary lock-in → fix: open formats, portable catalogs.
Unbounded costs → fix: unit economics, budgets, egress controls.
9) Signals vs Counter-Signals
Streaming: adopt if freshness < minutes matters; counter = batch good
enough.
Semantic layer: adopt if KPI disputes and repeated logic; counter = few
consumers.
Lakehouse: adopt if multi-engine + time travel needed; counter = single
warehouse fits all.
Feature store: adopt if online inference needs parity; counter = offline
batch only.
10) Future-Proofing Heuristics
Keep data in open formats; compute is replaceable.
Prefer APIs/contracts over raw tables.
Design for idempotency, replay, time travel.
Track cost + carbon as SLOs; automate tiering.
Security by default: zero-trust, short creds, data classification.
Document logic/decisions; schedule “fitness checks.”
11) Execution Playbooks
A) 30/60/90 Plan
30: Inventory products; owners/SLOs; lineage + telemetry
60: Add semantic layer (top KPIs); CDC pilot
90: Add cost/quality gates; canary process; publish catalog
B) Live Data Starter Kit
CDC for OLTP, event-time windowing, MV feeding dashboard, DR plan
C) Dark Matter Checklist
Owner, schema/contract, PII scan, validation rules, cadence, retirement
criteria
12) Decision Framework (Quick Table)
Scenario
Direction
Notes
KPI drift
Semantic/metrics layer
Versioned metrics; contracts
Freshness <5m
Live stack pilot
CDC + streams + MVs; cost
guardrails
Cross-engine
analytics
Lakehouse on open
tables
One copy; zero-copy shares
Online ML
Feature store + registry
Point-in-time joins; drift monitoring
Heavy spreadsheets
Governed pipeline
Contracts, DLP, promotion path
Rising costs
FinOps + lifecycle
Unit economics;
autosuspend/tiering
Key Takeaways
1. Principles outlast tools—anchor to lifecycle & undercurrents.
2. The future is simpler + more real-time—but only where it adds value.
3. Interoperability + open formats = optionality.
4. Data products need owners, SLAs, semantics, telemetry.
5. AI is a co-pilot, not autopilot.
6. Invest in people + paved paths as much as platforms.
Summary
The future of DE = live, integrated, governed. Data and apps fuse; platforms
provide paved roads. Stick to principles, adopt open standards, evolve with
SLOs/contracts, and enforce cost/security guardrails. That mindset lets you
survive hype cycles while delivering durable value.