AI Quality Engineer

About this role

Not quite right? These jobs in Bengaluru match the same skills — apply to a few to improve your chances.

Software Engineer (5+ Years in DotNet+C#+React+Typescript+Azure+SQL Server)

Assurant

In OfficeBengaluru

Apply

Specialist, Application Development- Azure Infrastructure, backend python, API development

AT&T

RemoteBengaluru

Apply

Sr Technology Automation Engineer (Azure Cloud + Azure Devops + Terraform + AKS + Powershell + CI/CD Automation)

Assurant

In OfficeBengaluru

Apply

Senior Software Engineer – .NET, React & Azure

Version 1

In OfficeBengaluru

Apply

Skills you'll use

What you'll do

• Testing Strategy & Design
• Define testing strategies for AI agents, conversational interfaces, and agentic workflows
• Design behavioral test suites for non-deterministic outputs—where "correct" isn't binary
• Build evaluation frameworks that measure groundedness, factuality, relevance, and task completion
• Identify failure modes specific to AI systems: hallucinations, prompt injection, context window limitations, drift
• Develop testing approaches for each architecture pattern: RAG, function calling, human-in-the-loop, autonomous workflows
• Test Automation & Infrastructure
• Build automated evaluation pipelines that run as part of CI/CD
• Create test harnesses for LLM-based systems—mocking, fixtures, and reproducible test scenarios
• Develop regression suites that detect quality degradation when prompts, models, or data change

Who can apply

• 3–7 years of software engineering or quality engineering experience
• Strong programming skills in Python and/or TypeScript—you write test code, not just test cases
• Experience designing and building automated test frameworks
• Understanding of AI/ML systems—you know why testing LLM outputs is different from testing deterministic code
• Experience with CI/CD pipelines and integrating automated tests into build processes
• Ability to reason about non-deterministic systems and design meaningful quality metrics
• Strong analytical skills—you can look at agent outputs and determine whether they're good enough
• Experience testing AI/ML applications, conversational interfaces, or chatbots
• Background in LLM evaluation: prompt testing, groundedness scoring, factuality checking
• Familiarity with evaluation frameworks (DeepEval, Ragas, custom evaluation pipelines)

About Allegis Global Solutions

Searching for “Allegis Global Solutions jobs” or “jobs at Allegis Global Solutions”? You’re in the right place.