Job Title

About The Role As an Engineering & Data Science Specialist in Applied AI Evaluations, you will design, implement, and optimize evaluation frameworks and pipelines that measure the quality, safety, and performance of advanced AI systems at BMO. Youll work closely with the Product Owner and collaborate with engineering, research, and business teams to translate evaluation standards into robust, scalable solutions. Your work will directly impact model development, deployment, and continuous improvement cyclesensuring that evaluation signals are deeply integrated into experimental and production systems. Youll contribute to the development of reusable tools, agents, and harnesses that support responsible AI practices and regulatory compliance. Hybrid role. Key Responsibilities Develop, implement, and maintain evaluation pipelines and harnesses for AI models, ensuring reliability, reproducibility, and scalability for banking applications. Collaborate with the Product Owner to translate evaluation standards and business requirements into technical solutions. Prototype and validate evaluation methods using real banking workflows, integrating feedback into model training and deployment. Integrate evaluation metrics and signals into research, training, and production systems to drive measurable improvements in model performance and customer outcomes. Partner with engineering, research, and product teams to shape model interaction paradigms and deployment strategies. Build reusable tools and systems that enable contributions from across BMO and raise the quality bar for AI solutions. Ensure all evaluation frameworks adhere to regulatory, compliance, and Responsible AI requirements. Document processes, share knowledge, and contribute to a culture of continuous learning and responsible innovation. Qualifications 7+ years of experience in engineering or data science roles, preferably most recent 3 years in AI, machine learning, or datadriven product environments. Strong programming skills (e.g., Python, Java, or similar) and experience with ML/AI frameworks (e.g., PyTorch, TensorFlow). Experience building and maintaining evaluation pipelines for large language models (LLMs) and/or other AI systems, including multiagent workflows, tool use, or longcontext scenarios. Solid understanding of deep learning concepts, model training, and deployment practices. Familiarity with MLOps, CI/CD, and cloudbased AI infrastructure is a plus. Experience driving initiatives as a leader. Strong analytical and problemsolving skills, with the ability to communicate technical concepts to both technical and nontechnical audiences. Motivated by highimpact collaboration and able to thrive in fastpaced, ambiguous environments. Commitment to BMOs values of inclusion, integrity, and responsible innovation. Salary $82,800.00 - $154,800.00 (Salaried). The above represents BMO Financial Groups pay range and type. Salaries will vary based on factors such as location, skills, experience, education, and qualifications for the role, and may include a commission structure. Salaries for parttime roles will be prorated based on number of hours regularly worked. For commission roles, the salary listed above represents BMO Financial Groups expected target for the first year in this position. BMO Financial Groups total compensation package will vary and may include performancebased incentives, discretionary bonuses, as well as other perks and rewards. BMO also offers health insurance, tuition reimbursement, accident and life insurance, and retirement savings plans. To view more details of our benefits, please visit: https://jobs.bmo.com/global/en/TotalRewards About Us At BMO we are driven by a shared Purpose: Boldly Grow the Good in business and life. It calls on us to create lasting, positive change for our customers, our communities and our people. By working together, innovating and pushing boundaries, we transform lives and businesses, and power economic growth around the world. As a member of the BMO team you are valued, respected and heard, and you have more ways to grow and make an impact. We strive to help you make an impact from day one for yourself and our customers. Well support you with the tools and resources you need to reach new milestones, as you help our customers reach theirs. From indepth training and coaching, to manager support and networkbuilding opportunities, well help you gain valuable experience, and broaden your skillset. To find out more visit us at https://jobs.bmo.com/ca/en Inclusion and Accommodations BMO is committed to an inclusive, equitable and accessible workplace. By learning from each others differences, we gain strength through our people and our perspectives. Accommodations are available on request for candidates taking part in all aspects of the selection process. To request accommodation, please contact your recruiter. Recruiter Notes Note to recruiters: BMO does not accept unsolicited resumes from any source other than directly from a candidate. Any unsolicited resumes sent to BMO, directly or indirectly, will be considered BMO property. BMO will not pay a fee for any placement resulting from the receipt of an unsolicited resume. A recruiting agency must first have a valid, written and fully executed agency agreement contract for service to submit resumes. #J-18808-Ljbffr