Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Data Engineering Fundamentals

ETL vs ELT

AspectETLELT
OrderExtract → Transform → LoadExtract → Load → Transform
TransformBefore loading (middleware)After loading (in warehouse)
Best forTraditional data warehousesCloud data warehouses
ToolsInformatica, Talend, SSISdbt, Spark, BigQuery SQL
FlexibilitySchema-on-writeSchema-on-read

Data Warehouse vs Data Lake vs Lakehouse

AspectData WarehouseData LakeLakehouse
DataStructuredRaw (all types)All types
SchemaSchema-on-writeSchema-on-readBoth
FormatSQL tablesFiles (Parquet, JSON)Tables + files
UsersAnalystsData scientistsBoth
CostExpensive (storage + compute)Cheap storageOptimized
ExamplesSnowflake, RedshiftS3, ADLSDelta Lake, Iceberg

OLTP vs OLAP

AspectOLTPOLAP
PurposeTransaction processingAnalytics
QueriesSimple, frequentComplex, aggregations
DataCurrent, operationalHistorical, analytical
SchemaNormalized (3NF)Denormalized (star/snowflake)
Row countThousands-millionsMillions-billions
ResponseMillisecondsSeconds-minutes

Star Schema

       ┌──────────┐
       │  Fact    │
       │ Table    │
       │(sales)   │
       └──┬──┬──┬─┘
          │  │  │
    ┌─────┘  │  └─────┐
    ▼        ▼        ▼
┌──────┐ ┌──────┐ ┌──────┐
│ Dim  │ │ Dim  │ │ Dim  │
│ Time │ │Prod  │ │Store │
└──────┘ └──────┘ └──────┘
  • Fact table: Measurements (sales amount, quantity)
  • Dimension tables: Context (time, product, store, customer)
  • Star: One level of dimensions around fact
  • Snowflake: Dimensions normalized into sub-dimensions

References