Skip to Main Content

Job Title


AI Quality Engineer


Company : New York Technology Partners


Location : ottawa, Ontario


Created : 2025-12-10


Job Type : Full Time


Job Description

We are looking for a AI Quality Engineer with strong automation expertise and hands-on experience validating LLMs, GenAI workflows, and AI-driven applications. This role involves building automated test suites, validating AI outputs, ensuring system reliability, and contributing to the quality strategy for next-generation AI features. Core Experience4-6 years as a Software Engineer, SDET, or Automation Engineer.Strong coding skills in Python, TypeScript, or Java.Hands-on experience developing automation scripts, tools, or frameworks.Practical experience using LLMs, prompt engineering, and evaluating AI-generated outputs.Familiarity with agentic AI systems and exposure to tools like LangGraph, AutoGen, or CrewAI.Basic understanding of Model Context Protocol (MCP) or context-aware workflow automation (nice to have)AI / ML TechnologiesPractical experience with AI/ML frameworks:LangChain, Hugging Face, GPT models, vector databases, RAG pipelines.Experience with ML/DL libraries such as:Scikit-learn, PyTorch, TensorFlow, Keras, Transformers, OpenCV.Ability to work with embeddings, similarity search, and content evaluation metrics.GenAI & AI Agent DevelopmentAbility to integrate or build GenAI components, including RAG pipelines or agent-based workflows.Support model evaluation tasks such as: Output quality checks Hallucination detection, Prompt validation ,Regression checks for model updates &Automation & Quality EngineeringExperience building automation using Python, PyTest, Selenium, Playwright, or API testing libraries.Ability to design and execute automated tests for: Functional Integration ,APIBasic performance & reliability testingHands-on experience testing RESTful APIs and building automated API suites. Cloud, DevOps & CI/CDExposure to deploying AI or automation solutions on AWS.Working knowledge of CI/CD pipelines, including: Automated testing ,Model validation stepsVersioning & artifact management SDLC & Collaboration