Jobiglo

No results.

This job is no longer available

This job expired on 23/09/2026. It no longer accepts applications.

LLM Evaluator (Model Response Analyst)

Odixcity Consulting

Remote
Remote 🇬🇧 English
A/B testing

Job description

About the role

We are looking for a detail‑oriented LLM Evaluator (Model Response Analyst) to assess and improve the performance of large language models. Working remotely with a global team, you will analyze AI‑generated text for factual accuracy, coherence, safety, bias and alignment with guidelines.

Key responsibilities

  • Evaluate and rank model‑generated responses using complex rubrics covering factuality, coherence, safety, instruction‑following and creativity.
  • Compare multiple outputs for the same prompt, select the preferred one and justify the choice.
  • Provide concise feedback to modeling and training teams about recurring failure patterns.
  • Design adversarial prompts to expose biased, harmful or insecure model behavior and help patch safety gaps.
  • Collaborate with QA to refine evaluation guidelines and resolve ambiguous edge cases.
  • Participate in cross‑checking sessions to calibrate scoring and ensure inter‑rater reliability.
  • Analyze underperforming outputs, hypothesize root causes and flag novel model behaviors for research.

Required profile

  • Minimum 2 years of professional experience in computational linguistics, data analysis, technical writing, NLP quality assurance or cognitive science.
  • Bachelor’s degree in Computer Science or a related field.
  • Proven ability to craft prompts that test model limits and to explain why a response is good or bad.
  • Experience with Reinforcement Learning from Human Feedback (RLHF) data collection and evaluation.
  • Familiarity with dataset sourcing, cleaning, annotation and A/B testing for AI models.

Required skills

  • Prompt engineering
  • RLHF data collection
  • Dataset annotation
  • A/B testing for AI models
  • Evaluation rubric development

Questions fréquentes

Le salaire n'est pas communiqué publiquement par le recruteur. Vous pouvez postuler et négocier directement avec Odixcity Consulting.
Cliquez sur "Postuler maintenant" en haut de la page. Vous pouvez importer votre CV en 1 clic — Jobiglo extrait automatiquement vos informations et postule pour vous.

Why are you reporting this job?

Thank you for your report. We will review this job.

A question about this job?

Ask it here: you will get the full job summary by e-mail, right away.

Published 3 months ago

61 views · 0 interested

Boost your chances

Upload your CV — we will match you with relevant openings.

Analyzing your CV...

Odixcity Consulting