Senior Associate_Agentic AI Test Engineer_Digital Integration_Advisory_Kolkata

Job Description & Summary We are looking for an experienced Agentic AI QA/Validation Engineer who can validate the complete agentic lifecycle, including user intent, prompting, agent reasoning, routing, MCP tool execution, and end-to-end workflow testing. The role requires expertise in AI observability, automation, and AI quality assessment. Job Position Title Senior Associate_Agentic AI Test Engineer_Digital Integration_Advisory_Kolkata Responsibilities • Validate end-to-end Agentic AI workflows, including prompting, reasoning, routing, and tool execution • Perform MCP testing, API validation, and workflow orchestration testing • Monitor agent behavior using AI observability and tracing tools • Design and execute test scenarios for prompts, agents, and AI workflows • Assess AI quality metrics such as accuracy, groundedness, consistency, and latency • Collaborate with development teams to improve AI system quality and reliability Mandatory Skill Sets • Experience in Agentic AI testing and validation • Hands-on experience with MCP testing, tool invocation validation, and API testing • Experience with observability tools such as LangFuse, Phoenix, or similar platforms • Strong understanding of AI quality metrics, hallucination detection, and evaluation methods • Experience with prompt testing, guardrail validation, and multi-turn conversation testing • Understanding of agent routing, context management, memory, and reasoning validation • Experience creating test strategies, evaluation datasets, and quality frameworks • Experience with automation and testing tools such as Postman, Playwright, Katalon, or equivalent Preferred Skill Sets • Experience with multi-agent testing and orchestration validation • Knowledge of RAG testing, retrieval evaluation, and citation validation • Understanding of Responsible AI, AI Governance, and AI risk frameworks • Experience building automated AI evaluation pipelines • Familiarity with token analytics, AI monitoring, and performance optimization Years of Experience Required 4 - 8 Years Education Qualification Bachelor's degree in Computer Science, Information Technology, or a related field Location Kolkata