IN_Senior Associate_Data Engineer + AI_Digital Integration_Advisory_Kolkata

We are seeking an Azure Databricks Data Engineer to design, develop, and optimize scalable data platforms and analytics solutions on Microsoft Azure. The ideal candidate will have strong expertise in Azure Databricks, PySpark, Delta Lake, and data engineering best practices to build efficient, reliable, and high-performance data pipelines. The role involves collaborating with business stakeholders, data architects, and analytics teams to deliver enterprise-grade data solutions while ensuring data quality, governance, scalability, and performance. Key Responsibilities Design, develop, and maintain scalable data pipelines using Azure Databricks. Develop ETL/ELT workflows using PySpark, Spark SQL, and Azure Data Factory. Build and manage Lakehouse architectures using Delta Lake and Azure Data Lake Storage (ADLS Gen2). Implement data ingestion frameworks for structured and unstructured data sources. Optimize Spark jobs, cluster performance, and query execution. Work with business and technical teams to translate requirements into data solutions. Design and maintain data models, schemas, and data integration processes. Ensure data quality, governance, security, and compliance standards. Troubleshoot production issues and optimize data processing performance. Create and maintain technical documentation and operational procedures. Participate in Agile methodologies including Scrum, Kanban, or SAFe. Mandatory Skill Sets Azure Databricks & Big Data Strong experience with Azure Databricks . Hands-on expertise in PySpark , Spark SQL , and Delta Lake . Experience with Lakehouse architecture and large-scale data processing. Understanding of Spark performance tuning and optimization techniques. Azure Data Platform Experience with: Azure Data Factory (ADF) Azure Data Lake Storage Gen2 (ADLS) Azure SQL Database Azure Key Vault Data Engineering & Databases Strong experience in ETL/ELT development. Advanced SQL and T-SQL proficiency. Experience with PostgreSQL databases. Knowledge of data warehousing concepts and dimensional modeling. Experience in data architecture and data modeling. Programming Strong proficiency in Python and PySpark . Experience building reusable data engineering frameworks. Development Methodology Experience working in Agile environments (Scrum, Kanban, or SAFe). Good to Have Skill Sets Unity Catalog Microsoft Purview Azure Fabric Snowflake Power BI Azure DevOps Structured Streaming CI/CD implementation for Databricks deployments Preferred Skill Sets Strong hands-on experience with Azure Databricks, PySpark, Python, and SQL. Experience building enterprise-scale Azure data platforms. Expertise in ETL, ELT, Data Warehousing, and Lakehouse architectures. Experience with data governance, metadata management, and lineage. Knowledge of performance optimization and cost management within Azure Databricks. Years of Experience Required 4-7 years of exp. Certifications / Credentials (Any One Mandatory) Databricks Certified Data Engineer Associate Databricks Certified Data Engineer Professional Microsoft Certified: Azure Data Engineer Associate (DP-203) Microsoft Certified: Azure Solutions Architect Expert Educational Qualification Bachelor's Degree in Computer Science, Information Technology, Data Engineering, or a related field .