Delta Lake with Azure Databricks: Deep Dive

Delta Lake with Azure Databricks: Deep Dive

Delta Lake with Azure Databricks: Deep Dive

Delta Lake is an open-source storage layer that brings reliability to Data Lakes, by providing data warehouse-like features, on top of Data Lake. It has a big ecosystem, and there are various tools and architectures based on that - Delta Lake Storage, Delta Engine, Delta Architecture, Delta Live Tables,

How Does Delta Lake Work?, ACID Guarantees on Delta Lake, Understanding Delta Ecosystem, Setting up Azure Databricks Environment

3.35

Horas


Horas dedicadas al curso.
8

Clases


Clases totales
12

Estudio


Horas dedicadas de estudio
5

Nivel


Intermedio

Getting Started with Delta Lake

  • How Does Delta Lake Work?
  • ACID Guarantees on Delta Lake
  • Understanding Delta Ecosystem
  • Setting up Azure Databricks Environment

Working with Delta Lake Storage

  • Storing Data in Delta Format
  • Creating Delta Table
  • Inserting Data to Delta Table
  • Performing DML Operations: Update / Delete / Merge
  • Schema Enforcement and Evolution
  • Applying Table Constraints
  • Accessing Data with Time Travel

Handling Streaming Data on Delta Lake

  • Understanding Streaming on Delta Lake
  • Setting up Streaming Environment
  • Building Streaming Pipeline with Delta Lake
  • Updating Delta Table with Streaming Data
  • Transaction Log Checkpointing
  • Handling Concurrency

Optimizing with Delta Engine in Databricks

  • Understanding Delta Engine
  • Improving Performance with Statistics and Data Skipping
  • Optimizing & Z-ordering Delta Table
  • Cleaning Files with Vacuum
  • Working with Photon Execution Engine
  • Enabling Auto Optimization on Delta Table
  • Using Delta Cache

Building a Lakehouse Architecture

  • Understanding Lakehouse Architecture
  • Inside Lakehouse: Understanding Delta Architecture
  • Using Change Data Feed
  • Delta Architecture: With Batch Data & Change Data Feed

Building ETL Pipelines with Delta Live Tables

  • Understanding Delta Live Tables
  • Building Simple ETL Pipeline with Delta Live Tables
  • Building Complex & Incremental ETL Pipeline

Implementing Common Use Cases

  • Querying Delta Tables with Azure Synapse Analytics
  • Consuming Delta Tables in Power BI

Satisfacción

¿Qué aprendí?

  • Almacenamiento de datos en formato Delta
  • Creación de tabla Delta
  • Insertar datos en la tabla Delta
  • Realización de operaciones DML: Actualizar/Eliminar/Fusionar
  • Aplicación y evolución del esquema
  • Aplicación de restricciones de tabla
  • Acceso a datos con viajes en el tiempo