Quality assurance jobs in Austin, TX

About the role

We are seeking a QA Engineer to help ensure the reliability, accuracy, and robustness of our AI Agents. This role will focus on data quality, model evaluation, and regression testing frameworks to identify and mitigate common LLM failure modes. You will be responsible for designing automated and scalable quality assurance systems while working in an AWS-based infrastructure. If you have a strong background in LLM testing, data validation, and automated QA frameworks, this role is an excellent opportunity to contribute to cutting-edge AI systems.

What you'll bring

• 3+ years in software testing and quality assurance
• 2+ years with a focus on ML evaluation, NLP, LLMs, VLMs, etc.
• Deep understanding of LLM data quality challenges and common failure modes.
• Experience designing automated tests for AI/ML models.
• Familiarity with Python and testing frameworks such as PyTest, Hypothesis, or similar.
• Knowledge of evaluation metrics for LLMs (DeepEval, MLflow, LangSmith, or similar).
• Hands-on experience with automated data validation techniques.
• Strong debugging and analytical skills.
• Experience creating or working with labeled evaluation datasets (“golden” sets) for model evaluation.
• Working knowledge of evaluation metrics for structured information extraction: field-level precision, recall, and F1
• exact vs. fuzzy matching
• numeric tolerance

Benefits