Job Title

Artificial Intelligence EngineerThis range is provided by Tribus. Your actual pay will be based on your skills and experience talk with your recruiter to learn more.Base pay rangeA$180,000.00/yr - A$200,000.00/yrDirect message the job poster from TribusFounder | Recruiter @ tribus - connecting leading technology experts with top financial services firms across APAC | part-time recruiter, full-time...Software Engineer - AILLM | Python | AWSWe''re partnering with a fast-growing software company building AIdriven products used in highstakes, realworld workflows.The focus is on productionquality AI: systems that must be reliable, measurable, and safe at scale.They''re looking for a Software Engineer with AI experience to join a team responsible for the core AI platform, with a particular emphasis on LLM evaluation, observability, and reliability.This is a handson engineering role, sitting close to product and domain experts, where your work directly influences how AI quality is defined, measured, and enforced in production.What you''ll work onBuilding and operating LLM evaluation pipelines that assess model quality, robustness, and safetyDefining test sets, metrics, and evaluation workflows, including humanintheloop processes where requiredTranslating product and domain constraints into concrete, testable evaluation criteriaRunning and orchestrating distributed evaluation workloads on AWS, including monitoring compute usageAnalysing evaluation results, identifying failure modes, and collaborating on mitigations (prompt changes, data updates, model selection or finetuning)Integrating and assessing opensource and vendor evaluation frameworks, writing glue code where neededContributing to the evolution of the AI evaluation and platform architectureWhat they''re looking forExperience monitoring and evaluating LLMbased applicationsHandson exposure to LLM evaluation tools, benchmarks, and metricsUnderstanding of common LLM failure modes (e.g. hallucination, bias, toxicity, prompt injection)Experience with cloud ML infrastructure, ideally AWSFamiliarity with distributed workloads (e.g. Ray, AWS Lambda, or similar)Comfort working with an evolving LLM observability and evaluation stackAbility to work with nonML stakeholders and convert qualitative requirements into quantitative testsWorking environment & benefitsFlexible hybrid setup, with twiceweekly collaboration in a modern CBD officeStrong learning and career development opportunities in a scaling businessWellness focus including additional leave and gym membershipCollaborative team culture with regular social eventsPool table, snacks, and a genuinely supportive environmentThis role is well suited to engineers who care about AI reliability and correctness, and who want to work on systems where evaluation and safeguards genuinely matter.Must be based in Sydney with full working rights. Remote working or sponsorship is not available for this role.Seniority levelMidSenior levelEmployment typeFulltimeJob functionEngineering and Information TechnologyIndustriesTechnology, Information and Media #J-18808-Ljbffr