Senior Test Analyst

The Senior QA Engineer will drive the quality strategy for agentic AI systems, including applications powered by LLMs, autonomous agents, RAG pipelines, tool-calling, memory modules, and complex AI decisioning flows.

The role ensures that AI-driven capabilities meet strict enterprise-grade standards of accuracy, safety, reliability, and compliance, while collaborating closely with engineering, product, and AI/ML teams globally.

Job Responsibilities

• Write automation alongside development to test unit level, API and e2e UI.
• Implement backend services supporting agent runtimes, tool execution, and memory layers
• Integrate LLMs into business workflows and internal systems
• Contribute to orchestration logic connecting tools, APIs, and external services
• Write clean, maintainable, well-tested code
• Participate in architectural discussions
• Collaborate with AI engineers to productionize experimental features
• Contribute to monitoring, logging, and performance optimization
• Strong hands-on experience testing LLM-based or agentic AI applications.
• Expertise in Python automation (PyTest, Playwright, Postman).
• Understanding of prompts, embeddings, vector stores, tokenization. • Experience testing RAG pipelines and grounding data workflows.
• Proficiency with APIs, microservices, and cloud environments (Azure/AWS/GCP).
• Ability to evaluate unstructured AI output using structured scoring techniques.
• Strong analytical and debugging skills in complex AI-driven workflows.

Desirable Skills

• Experience with AI evaluation tools: TruLens, PromptLayer, MLFlow, W&B.;
• Familiarity with agentic frameworks: LangChain, AutoGen, Semantic Kernel, CrewAI.
• Understanding of LLM safety concepts, adversarial testing, guardrails, red teaming.
• Knowledge of model fine-tuning, RLHF, and observability pipelines.
• Exposure to distributed architecture and enterprise platforms.
• Ability to mentor QA engineers on AI testing methods.

Qualifications

• 5+ years of professional software engineering experience
• Hands-on work with generative AI and agentic systems.
• Experience working in cross-functional product teams
• Degree in Computer Science, Engineering, or related field (or equivalent experience)
• Strong problem-solving skills and ability to work in fast-evolving technical domains
• Interest in AI-native software architectures and continuous learning mindset
• Certifications such as ISTQB Advanced, Azure/AWS/Google AI certifications (optional).

Equality Statement

SmartStream is an equal opportunities employer. We are committed to promoting equality of opportunity and following practices which are free from unfair and unlawful discrimination.