Job Title

Role Overview:We are seeking a detail-oriented and proactive Agentic QA Engineer to join our AI product team in Bangalore. You will take charge of testing efforts for goal-driven LLM workflows, focusing on system robustness, safety, and reliability. This includes multi-agent coordination, tool invocation validation, memory testing, and context consistency.You’ll work closely with engineers, prompt designers, and AI researchers to validate system behaviour across complex, autonomous AI tasks—ensuring agents perform reliably under real-world and edge conditions.Key Responsibilities:Design, execute, and automate test plans for agentic AI workflows (tool usage, planning, state transitions).Validate AI behaviors across prompts, subgoal decomposition, retries, and multi-step tool chains.Identify and troubleshoot issues such as:HallucinationsInconsistent responsesPlanning and memory failuresTool invocation errorsLatency or cost inefficienciesEvaluate model robustness, safety alignment, and adherence to defined business rules.Integrate evaluation metrics like factuality, coherence, completeness, and toxicity into test suites.Analyze prompt behavior, trace tool execution paths, and maintain test logs and audit trails.Run structured experiments to evaluate changes to prompts, tools, or workflow logic.Maintain regression pipelines and ensure CI/CD integration for agent QA workflows.Required Skills & Qualifications:3+ years of QA or test automation experience, preferably in AI, ML, or agent-driven systems.Proficient in Python or JavaScript testing frameworks (e.g., Pytest, Playwright).Strong grasp of REST APIs, JSON workflows, and debugging distributed systems.Familiarity with LLM-based systems and orchestration tools (LangChain, OpenAI tool calling, CrewAI).Experience evaluating AI output quality using precision, hallucination, and contextual correctness.Exposure to AI observability or evaluation platforms such as LangSmith, Trulens, or equivalent.Nice to Have:Experience testing LLM-integrated assistants or autonomous AI agents.Knowledge of Retrieval-Augmented Generation (RAG) testing methodologies.Familiarity with guardrails, compliance frameworks, and AI safety metrics.Understanding of prompt lifecycle, tool-use planning, and dynamic state management.What You’ll Gain:Play a key role in shaping the future of Agentic AI systems.Work with an interdisciplinary team at the cutting edge of autonomous workflows.Define QA best practices for intelligent, adaptive systems.Be part of a fast-moving and mission-driven team based in Bangalore.