Functional AI Tester - GenAI

Functional AI Tester - GenAI

• - - - - - - - - - - -

We are seeking a Quality Assurance (QA) Engineer focused on testing Generative AI (GenAI) applications with a strong emphasis on Python-based test automation, GenAI evaluation, and ETL/data quality validation. You will design and execute end-to-end test strategies that ensure our AI solutions are accurate, reliable, safe, and compliant.

About the Role

You will be involved in QA for GenAI features including Retrieval-Augmented Generation (RAG), conversational AI and Agentic evaluations. The role centers on:

-

Systematic GenAI evaluation (qualitative and quantitative metrics)

-

ETL and data quality testing for the data flows that feed AI systems

-

Python-driven automated testing

This position is hands-on and collaborative, partnering with AI engineers, data engineers, and product teams to define measurable acceptance criteria and ship high-quality AI features.

Key Responsibilities

• -
• Test strategy and planning
• -
• Define risk-based test strategies and detailed test plans for GenAI features.
• -
• Establish clear acceptance criteria with stakeholders for functional, safety, and data quality aspects.
• -
• Python test automation
• -
• Build and maintain automated test suites using Python (e.g., PyTest, requests).
• -
• Implement reusable utilities for prompt/response validation, dataset management, and result scoring.
• -
• Create regression baselines and golden test sets to detect quality drift.
• -
• GenAI evaluation
• -
• Develop evaluation harnesses covering factuality, coherence, helpfulness, safety, bias, and toxicity etc.
• -
• Design prompt suites, scenario-based tests, and golden datasets for reproducible measurements.
• -
• Implement guardrail tests including prompt-injection resilience, unsafe content detection, and PII redaction checks.
• -
• Track quality metrics over time.
• -
• RAG and semantic retrieval testing
• -
• Verify alignment between retrieved sources and generated answers.
• -
• Verify adversarial tests.
• -
• Measure retrieval relevance, precision/recall, grounding quality, and hallucination reduction.
• -
• API and application testing
• -
• Test REST endpoints supporting GenAI features (request/response contracts, error handling, timeouts).
• -
• ETL and data quality validation
• -
• Test ingestion and transformation logic; validate schema, constraints, and field-level rules.
• -
• Implement data profiling, reconciliation between sources and targets, and lineage checks.
• -
• Verify data privacy controls, masking, and retention policies across pipelines.
• -
• Non-functional testing
• -
• Performance and load testing focused on latency, throughput, concurrency, and rate limits for LLM calls.
• -
• Cost-aware testing (token usage, caching effectiveness) and timeout/retry behavior validation.
• -
• Reliability and resilience checks including error recovery and fallback behavior.
• -
• Share results and insights; recommend remediation and preventive actions.

Required Qualifications

-

Experience

• -
• 5+ years in software QA, including test strategy, automation, and defect management.
• -
• 2+ years testing AI/ML or GenAI features, with hands-on evaluation design.
• -
• 4+ years testing ETL/data pipelines and data quality.
• -
• Technical skills
• -
• Python: Strong proficiency building automated tests and tooling (PyTest, requests, pydantic or similar).
• -
• API testing: REST contract testing, schema validation, negative testing.
• -
• GenAI evaluation: crafting prompt suites, golden datasets, rubric-based scoring, and automated evaluation pipelines.
• -
• RAG testing: retrieval relevance, grounding validation, chunking/indexing verification, and embedding checks.
• -
• ETL/data quality: schema and constraint validation, reconciliation, lineage awareness, data profiling.
• -
• Quality and governance
• -
• Understanding of LLM limitations and methods to detect/reduce hallucinations.
• -
• Safety and compliance testing including PII handling and prompt-injection resilience.
• -
• Strong analytical and debugging skills across services and data flows.
• -
• Soft skills
• -
• Excellent written and verbal communication; ability to translate quality goals into measurable criteria.
• -
• Collaboration with AI engineers, data engineers, and product stakeholders.
• -
• Organized, detail-oriented, and outcomes-focused.

Nice to Have

• -
• Experience with evaluation frameworks or tooling for LLMs and RAG quality measurement.
• Experience creating synthetic datasets to stress specific behaviors.