Python QA - (Pyspark with Databricks)

About the Role

We are looking for an experienced

QA Automation Engineer

with strong expertise in

Python, PySpark, and Databricks

to support enterprise data engineering and analytics initiatives. The ideal candidate should have hands-on experience in validating large-scale data pipelines, ETL processes, and data platforms while building robust automation frameworks to ensure data quality and reliability.

Key Responsibilities

Design, develop, and maintain automated test frameworks using

Python

• .
• Validate ETL pipelines, data transformations, and data ingestion processes built using

PySpark

and

Databricks

• .
• Perform data validation, reconciliation, and quality checks across structured and semi-structured datasets.
• Develop automated test scripts for functional, integration, regression, and end-to-end testing of data platforms.
• Validate large datasets using SQL and PySpark.
• Build and maintain automated data quality checks and monitoring processes.
• Collaborate with Data Engineers, Developers, and Business Analysts to understand data requirements and ensure high-quality deliveries.
• Analyse defects, perform root cause analysis, and support issue resolution.
• Integrate automated tests into CI/CD pipelines and support Agile delivery.
• Contribute to QA best practices, automation strategy, and continuous improvement initiatives.

Skills & Qualifications

• 8+ years
• of experience in Software Testing or Test Automation.
• Strong hands-on experience with

Python

• .
• Solid expertise in

PySpark

and

Databricks

• .
• Strong experience in

SQL

• and ETL/data validation.
• Experience testing large-scale data pipelines and data engineering workflows.
• Hands-on experience with
• data quality frameworks or libraries
• such as
• Great Expectations, Soda, Deequ, dbt Tests, or similar
• .
• Strong understanding of data validation, reconciliation, and quality assurance methodologies.
• Experience working in Agile/Scrum environments.
• Excellent analytical, troubleshooting, and problem-solving skills.