IN_Senior Associate_Azure Databricks Engineer_Digital Integration_Advisory_Kolkata

Senior Associate - Azure Databricks Data Engineer Experience Required: 5+ Years Job Location: Kolkata Job Summary We are seeking an experienced Azure Databricks Data Engineer to design, develop, and optimize scalable data solutions on the Microsoft Azure platform. The ideal candidate will have strong expertise in Azure Databricks, PySpark, Delta Lake, and modern data engineering practices to build high-performance data pipelines and support enterprise analytics initiatives. The role involves working closely with data architects, business analysts, and stakeholders to translate business requirements into robust technical solutions while ensuring data quality, governance, scalability, and performance. Key Responsibilities Design, develop, and maintain scalable data pipelines using Azure Databricks, PySpark, and Delta Lake. Build and optimize ETL/ELT workflows for structured, semi-structured, and unstructured data. Develop data ingestion frameworks using Azure Data Factory (ADF) and Azure-native services. Implement and manage data lakehouse architectures using Azure Data Lake Storage Gen2 (ADLS) and Delta Lake. Optimize Spark workloads, cluster configurations, and query performance. Collaborate with business and technology teams to convert business requirements into technical solutions. Design and implement data models for analytics, reporting, and downstream consumption. Ensure data quality, reliability, security, and governance across the platform. Develop reusable data engineering components and automation frameworks. Maintain technical documentation, data dictionaries, and solution designs. Participate in Agile delivery practices including Scrum, Kanban, or SAFe methodologies. Support deployment, monitoring, troubleshooting, and performance tuning of data solutions. Mandatory Skill Sets Azure Databricks & Data Engineering Strong expertise in Azure Databricks. Hands-on experience with PySpark, Spark SQL, and Delta Lake. Experience designing and implementing Lakehouse architectures. Knowledge of Spark optimization techniques, partitioning, caching, and performance tuning. Azure Data Platform Experience with: Azure Data Factory (ADF) Azure Data Lake Storage Gen2 (ADLS) Azure SQL Database Azure Key Vault Data Integration & Databases Strong experience in ETL/ELT development and data warehousing. Advanced SQL and T-SQL proficiency. Hands-on experience with PostgreSQL. Experience working with Graph Databases. Strong understanding of data modeling, schema design, and architecture patterns. Programming Strong proficiency in Python and PySpark. Experience developing reusable data engineering frameworks and utilities. Methodologies Experience working in Agile environments (Scrum, Kanban, or SAFe). Good-to-Have Skills Unity Catalog Microsoft Purview Azure Fabric Snowflake Power BI Azure DevOps / CI-CD implementation Data governance and metadata management Streaming solutions using Spark Structured Streaming Preferred Skills Strong hands-on expertise in Azure Databricks, PySpark, Python, and SQL. Experience building enterprise-scale Azure data platforms. Strong understanding of ETL, ELT, and Data Warehousing concepts. Experience with performance tuning and cost optimization in Azure Databricks. Knowledge of data governance, lineage, and security best practices. Certifications (Mandatory - Any One) Databricks Certified Data Engineer Associate Databricks Certified Data Engineer Professional Microsoft Certified: Azure Data Engineer Associate (DP-203) Microsoft Certified: Azure Solutions Architect Expert Educational Qualification Bachelor's Degree in Computer Science, Information Technology, or a related field.