Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 35 hours
Course Outline
Introduction, Learning Objectives, and Migration Strategy
- Defining course goals, aligning with participant profiles, and establishing success metrics
- Overviews of high-level migration approaches and associated risk factors
- Initial setup of workspaces, repositories, and laboratory datasets
Day 1 — Migration Fundamentals and Architectural Concepts
- Core Lakehouse concepts, Delta Lake overview, and Databricks architectural components
- Differences between SMP and MPP architectures and their impact on migration planning
- Designing the Medallion (Bronze→Silver→Gold) architecture and introducing Unity Catalog
Day 1 Lab — Translating Stored Procedures
- Hands-on migration of a sample stored procedure into a Databricks notebook
- Converting temporary tables and cursor logic into DataFrame transformations
- Validating results by comparing them against the original output
Day 2 — Advanced Delta Lake Features & Incremental Loading
- ACID transactions, commit logs, versioning capabilities, and time travel functionality
- Utilizing Auto Loader, MERGE INTO patterns, upsert operations, and handling schema evolution
- Optimizing storage and performance using OPTIMIZE, VACUUM, Z-ORDER, and partitioning strategies
Day 2 Lab — Incremental Ingestion & Performance Optimization
- Implementing Auto Loader-based ingestion and MERGE workflow logic
- Applying OPTIMIZE, Z-ORDER, and VACUUM commands; verifying outcome accuracy
- Evaluating and measuring improvements in read/write performance
Day 3 — SQL in Databricks, Performance Analysis & Debugging
- Leveraging advanced analytical SQL features: window functions, higher-order functions, and JSON/array processing
- Interpreting Spark UI metrics: DAGs, shuffles, stages, tasks, and diagnosing bottlenecks
- Applying query tuning techniques: broadcast joins, hints, data caching, and reducing spill operations
Day 3 Lab — SQL Refactoring & Performance Tuning
- Refactoring complex SQL processes into optimized Spark SQL statements
- Using Spark UI traces to pinpoint and resolve data skew and shuffle issues
- Conducting before/after benchmarks and documenting the tuning methodology
Day 4 — Tactical PySpark: Replacing Procedural Logic
- Understanding the Spark execution model: driver, executors, lazy evaluation, and partitioning strategies
- Replacing loops and cursors with efficient, vectorized DataFrame operations
- Implementing modularization, UDFs/pandas UDFs, UI widgets, and creating reusable libraries
Day 4 Lab — Refactoring Procedural Scripts
- Converting procedural ETL scripts into modular, well-structured PySpark notebooks
- Incorporating parametrization, unit-style testing, and reusable functional components
- Conducting code reviews and applying best-practice checklists
Day 5 — Orchestration, End-to-End Pipelines & Best Practices
- Designing Databricks Workflows: job structure, task dependencies, triggers, and error handling
- Building incremental Medallion pipelines with integrated quality rules and schema validation
- Integrating with Git (GitHub/Azure DevOps), CI pipelines, and establishing testing strategies for PySpark logic
Day 5 Lab — Building a Complete End-to-End Pipeline
- Assembling a Bronze→Silver→Gold pipeline orchestrated via Databricks Workflows
- Implementing robust logging, auditing, retry mechanisms, and automated validation checks
- Executing the full pipeline, validating final outputs, and preparing deployment documentation
Operationalization, Governance, and Production Readiness
- Best practices for Unity Catalog governance, data lineage, and access control management
- Managing costs, cluster sizing, autoscaling, and optimizing job concurrency patterns
- Creating deployment checklists, defining rollback strategies, and drafting operational runbooks
Final Review, Knowledge Transfer, and Next Steps
- Participant presentations showcasing migration work and key lessons learned
- Conducting gap analysis, recommending follow-up activities, and handing over training materials
- Providing references, suggested further learning paths, and available support options
Requirements
- Foundational understanding of data engineering principles
- Practical experience with SQL and stored procedures (e.g., Synapse / SQL Server)
- Familiarity with ETL orchestration concepts (such as ADF or similar tools)
Target Audience
- Technology managers possessing a data engineering background
- Data engineers seeking to shift procedural OLAP logic toward Lakehouse patterns
- Platform engineers tasked with overseeing Databricks adoption and implementation
