This job is no longer available
This job expired on 23/09/2026. It no longer accepts applications.
LLM Evaluator (Model Response Analyst)
Odixcity Consulting
Job description
About the role
We are looking for a detail‑oriented LLM Evaluator (Model Response Analyst) to assess and improve the performance of large language models. Working remotely with a global team, you will analyze AI‑generated text for factual accuracy, coherence, safety, bias and alignment with guidelines.
Key responsibilities
- Evaluate and rank model‑generated responses using complex rubrics covering factuality, coherence, safety, instruction‑following and creativity.
- Compare multiple outputs for the same prompt, select the preferred one and justify the choice.
- Provide concise feedback to modeling and training teams about recurring failure patterns.
- Design adversarial prompts to expose biased, harmful or insecure model behavior and help patch safety gaps.
- Collaborate with QA to refine evaluation guidelines and resolve ambiguous edge cases.
- Participate in cross‑checking sessions to calibrate scoring and ensure inter‑rater reliability.
- Analyze underperforming outputs, hypothesize root causes and flag novel model behaviors for research.
Required profile
- Minimum 2 years of professional experience in computational linguistics, data analysis, technical writing, NLP quality assurance or cognitive science.
- Bachelor’s degree in Computer Science or a related field.
- Proven ability to craft prompts that test model limits and to explain why a response is good or bad.
- Experience with Reinforcement Learning from Human Feedback (RLHF) data collection and evaluation.
- Familiarity with dataset sourcing, cleaning, annotation and A/B testing for AI models.
Required skills
- Prompt engineering
- RLHF data collection
- Dataset annotation
- A/B testing for AI models
- Evaluation rubric development
Questions fréquentes
Why are you reporting this job?
Explore further
Salaries, guides and searches in Sudan.
A question about this job?
Ask it here: you will get the full job summary by e-mail, right away.
Boost your chances
Upload your CV — we will match you with relevant openings.
Analyzing your CV...
Odixcity Consulting
Related job offers
-
Senior Software Engineer, Native Mobile (International)
Coinbase -
Forward Deployed Engineer - ProcessOS
Camunda -
Programmeur·euse compétitif·ve – Benchmarks IA (Remote)
Vettedlance -
Enregistreur de flux de travail pour outils de développement (éditeurs de code, IDE, clients API)
Vettedlance -
Ingénieur DevOps, Cloud & Linux – Environnement d'Entraînement IA (Remote)
Vettedlance