IN_Senior Associate_Databricks\SAP Analytics_Advisory_Hyderabad (India)

Job Description & Summary: PwC India is seeking an experienced and results-driven SAP Databricks Consultant to design, development, and implementation of scalable data and analytics solutions leveraging SAP and Databricks platforms. This role requires deep expertise in SAP data ecosystems, cloud-based data engineering, advanced analytics, and modern data architectures. The candidate will collaborate with business stakeholders, solution architects, data engineers, and technology teams to integrate SAP data sources with Databricks, enabling real-time insights, AI/ML-driven innovation, and enterprise-wide data transformation initiatives. The role demands strong technical leadership, strategic thinking, and the ability to deliver high-impact, data-driven solutions that generate measurable business value across diverse industry sectors. Job Position Title: IN_Senior Associate_Databricks\SAP Analytics_Advisory_Hyderabad (India) Responsibilities: • Design, develop, and optimize scalable batch and streaming data pipelines using Databricks, Apache Spark (PySpark/Scala), and Delta Lake to process large volumes of structured, semi-structured, and unstructured data. • Build and maintain data ingestion frameworks leveraging Databricks Auto Loader, Structured Streaming, and Delta Live Tables (DLT) for reliable and efficient data processing. • Implement end-to-end data engineering solutions on Azure Databricks, integrating with Azure Data Lake Storage (ADLS Gen2), Azure Synapse Analytics, Azure Event Hubs, and Azure SQL Database. • Demonstrate a solid understanding of SAP data and the SAP ecosystem. • Develop and optimize Lakehouse architectures, including Bronze, Silver, and Gold data layers, ensuring high data quality, governance, and performance. • Create and manage Delta Lake tables, implement partitioning, indexing, data compaction, and performance tuning techniques to improve workload efficiency. • Design and maintain reusable data models, metadata frameworks, and data quality controls to support analytics, reporting, and machine learning workloads. • Monitor, troubleshoot, and optimize Spark jobs, Databricks Workflows, and cluster performance to ensure scalability, reliability, and cost efficiency. • Implement enterprise-grade data governance, security, and access controls using Unity Catalog, ensuring compliance with organizational and regulatory standards. • Collaborate with data architects, analysts, and business stakeholders to define and implement scalable data solutions aligned with business requirements. • Automate deployment processes using CI/CD pipelines and Infrastructure as Code (IaC) tools for Databricks environments. Mandatory skill sets: • Proven experience as a Databricks Data Engineer with strong expertise in Apache Spark, PySpark, Databricks Lakehouse Platform, and Delta Lake. • Hands-on experience building scalable data pipelines using Databricks Workflows, Delta Live Tables (DLT), Auto Loader, Structured Streaming, and Delta Lake. • Strong experience with Azure Databricks and related Azure services including ADLS Gen2, Azure Synapse Analytics, Azure Data Factory, Azure Event Hubs, and Azure SQL Database. • Advanced knowledge of Spark optimization techniques, including partitioning, caching, query tuning, adaptive query execution, and cluster resource management. • Experience implementing data governance and security using Unity Catalog, role-based access control (RBAC), data lineage, and data quality frameworks. • Strong programming skills in Python and/or Scala, along with experience in Git, branching strategies, code reviews, and CI/CD implementation. • Proficiency in SQL and data modeling concepts, including dimensional modeling, star/snowflake schemas, and data warehousing best practices. • Experience with orchestration and automation tools such as Azure Data Factory, Databricks Workflows, Airflow, or similar platforms. • Exposure to real-time data processing, event-driven architectures, and messaging platforms such as Kafka or Azure Event Hubs is highly desirable. • Relevant certifications such as Databricks Certified Data Engineer Associate/Professional or DP 700 or DP 750 are preferred. Preferred skill sets: • Experience with MLflow, Feature Store, or MLOps workflows in Databricks. • Knowledge of data mesh, medallion architecture, and modern lakehouse design patterns. • Experience migrating legacy Hadoop/Hive workloads to Databricks Lakehouse Platform Years of experience required: 4 to 7 Years Education qualification: BE/B.Tech/MBA/MCA/M.Tech