IN_Senior Associate_Azure Databricks_D&A_Advisory_PAN India

Job Description & Summary: We are seeking an experienced Databricks Developer to design, develop, and optimize data engineering solutions on the Databricks Lakehouse Platform. The role involves building scalable data pipelines, implementing Delta Lake architectures, and ensuring high-performance, governed, and cost-efficient data platforms on Azure. Job Position Title: IN_Senior Associate_Azure Databricks_D&A_Advisory_PAN India Responsibilities: · Design and develop data pipelines using Apache Spark, PySpark, and Spark SQL within Databricks Workspace. · Build and manage Delta Live Tables (DLT) pipelines for declarative, reliable ETL processing. · Implement Structured Streaming and Auto Loader for incremental and real-time data ingestion. · Develop and orchestrate workflows using Databricks Workflows/Jobs and Azure Data Factory (ADF). · Design and maintain Delta Lake & Lakehouse Architecture — Bronze/Silver/Gold layered patterns. · Configure and manage Unity Catalog for centralized data governance, lineage, and access control. · Build analytical layers using Databricks SQL Warehouses for BI and reporting consumption. · Manage data storage and movement across ADLS Gen2 and Azure SQL environments. · Write production-grade Python scripts for data transformations, utilities, and automation. · Optimize Spark jobs for performance — partitioning, caching, broadcast joins, shuffle optimization. Mandatory skill sets: · Databricks (Must-Have): 3+ years — Workspace, Unity Catalog, Delta Live Tables (DLT), Workflows/Jobs, SQL Warehouses · Delta Lake & Lakehouse: Lakehouse architecture design, Bronze/Silver/Gold patterns, ACID transactions, schema evolution · Spark & PySpark (Must-Have): DataFrame API, Spark SQL, transformations, actions, performance tuning · Streaming (Must-Have): Structured Streaming, Auto Loader for incremental ingestion · Python (Must-Have): Production-grade scripting for data engineering and automation · Azure Stack (Must-Have) ADF (orchestration), ADLS Gen2 (storage), Azure SQL (relational) · Performance Tuning: Partitioning, Z-ordering, caching, shuffle optimization, job cluster sizing Preferred skill sets: · Experience with Databricks Asset Bundles or CI/CD for Databricks (GitHub Actions, Azure DevOps). · Familiarity with data modeling concepts (Star Schema, Data Vault 2.0). · Exposure to Power BI / Tableau connecting via Databricks SQL. · Knowledge of Databricks cost optimization (cluster policies, spot instances, photon engine). · Databricks Certified Data Engineer Associate/Professional is a plus. · Exposure to Agile/Scrum delivery methodologies. Years of experience required: 4-8 Years Education qualification: <B.Tech> /MBA