ADA - LLM Evaluation Assessment

Large Language Model (LLM) Evaluation Core Assessment

Complete the core conceptual evaluation matrix covering quality dimensions, alignment metrics, safety compliance, and language considerations.

New to LLM evaluation? Start with the Foundations Track. Already familiar with RLHF, calibration, and rubric-based rating? Try the Advanced Track. You can switch anytime before submitting.

What does LLM stand for in the context of artificial intelligence?
Large language models are trained exclusively on structured tabular data.
True False
Which training objective is most commonly used to pre-train modern LLMs?
What is a 'token' in the context of language models?
GPT-4, Claude, and Gemini are all examples of large language models.
True False
What is 'fine-tuning' in the context of LLMs?
Which of the following best describes 'prompt engineering'?
A) Writing code to train a new model from scratch B) Crafting and refining input instructions to guide an LLM toward better outputs C) Increasing a model's parameter count during inference D) Manually correcting an LLM's output after it has been generated
The 'context window' of an LLM refers to the maximum amount of text it can process in a single interaction.
True False
What is 'temperature' in LLM inference?
Which architecture underpins most modern large language models?
An LLM with a larger context window can 'remember' more information within a single conversation.
True False
What is 'zero-shot' prompting?
What is 'few-shot' prompting?
LLMs can access the internet in real time by default.
True False
What does RLHF stand for?
Which of the following is NOT a known risk of LLMs?
Hallucination in LLMs refers to the model generating confident but factually incorrect information.
True False
What is a 'system prompt' in LLM usage?
What does 'grounding' mean in the context of LLMs?
Increasing the temperature parameter makes LLM outputs more deterministic.
True False
A response is considered 'coherent' when it:
When evaluating an LLM response for 'relevance', what are you checking?
A factually correct response that doesn't answer the user's question is still considered a high-quality output.
True False
Which of these is a sign of a low-quality LLM response?
What does 'fluency' refer to when rating an LLM response?
Verbosity (being excessively long) always improves response quality.
True False
You are asked to rate two responses to the question 'How do I boil an egg?' Response A gives step-by-step instructions. Response B gives the history of eggs. Which is more relevant?
What is 'instruction following' as an evaluation dimension?
A response that contains accurate information but is rude or offensive is generally rated as high quality.
True False
When a model is asked to 'summarize in 3 bullet points' but returns 10 paragraphs, this is a failure of:
Which rating scale format is most commonly used in LLM evaluation tasks?
When comparing two model responses, your personal preference should be the sole basis for your rating.
True False
A response that confidently states '2+2=5' is an example of:
What should an evaluator do when both Response A and Response B are equally good?
Evaluators should apply consistent rating standards across all questions, regardless of topic.
True False
Which of the following best describes 'completeness' in output evaluation?
A user asks 'What is the capital of Australia?' and the model answers 'Sydney.' This response is:
When evaluating creative writing, grammar and factual accuracy are the only criteria that matter.
True False
What is 'calibration' in the context of LLM evaluation?
Which of the following is an example of a 'leading prompt' that could bias a model?
An evaluator's job is to rate responses objectively based on defined quality criteria, not personal opinion.
True False
If a model response is helpful but contains one minor factual error, the best approach is to:
'Groundedness' in a response means:
A model that repeats the same sentence multiple times in a response demonstrates good output quality.
True False
Which is the better response to 'Explain photosynthesis to a 10-year-old'? A: Uses simple analogies and short sentences. B: Uses biochemical terminology without explanation.
What is 'model bias' in LLMs?
LLMs trained on internet data may inherit societal biases present in that data.
True False
Which of the following is an example of gender bias in an LLM response?
What is 'prompt injection'?
Safety filtering in LLMs aims to prevent the model from generating harmful or dangerous content.
True False
A user asks an LLM for instructions on how to make a weapon. A well-aligned model should:
What does 'content moderation' in LLM output evaluation involve?
A response that is technically helpful but promotes illegal activity should be rated as high quality.
True False
Which of the following is an example of 'stereotype reinforcement' in an LLM?
What is 'PII' and why is it important in LLM evaluation?
An LLM that consistently outputs different quality for questions in different languages may exhibit language bias.
True False
'Sycophancy' in an LLM means:
Which evaluation task involves checking if a model treats demographic groups consistently?
Evaluators should flag responses that contain hate speech even if the prompt asked for it.
True False
What is a 'red-teaming' exercise in the context of AI?
Why is it important for LLM evaluators to maintain consistency across rating sessions?
A model that changes its stated opinion when the user pushes back (without new evidence) is showing sycophantic behavior.
True False
Which of the following is NOT a safety concern in LLM outputs?
When a response contains potentially sensitive information about a real person, an evaluator should:
LLM safety alignment techniques include both training-time and inference-time interventions.
True False
What is the primary purpose of data annotation in AI development?
High-quality annotations require clear guidelines and consistent application of those guidelines.
True False
Inter-annotator agreement (IAA) measures:
When annotation guidelines are ambiguous, the best approach is to:
Rushing through annotation tasks to complete more items typically improves data quality.
True False
What is a 'golden set' in annotation quality control?
Which of the following is an example of annotation bias?
Annotators should document uncertainty or edge cases rather than silently making arbitrary decisions.
True False
In a preference annotation task, you are shown two AI responses and asked to pick the better one. If both are equally bad, you should:
Which factor most affects the quality of a human evaluation dataset?
An annotator finding the topic boring is a valid reason to apply different quality standards.
True False
What does it mean for annotation labels to be 'reproducible'?
Why is it important to re-read the annotation guidelines before starting each new session?
Annotation tasks involving subjective content (like tone or style) require clearer guidelines than objective tasks.
True False
Which of the following is a valid reason to flag an item during annotation?
Data annotation for RLHF (Reinforcement Learning from Human Feedback) typically involves:
Annotators are expected to apply guidelines even when they personally disagree with the criteria.
True False
What is 'label leakage' in annotation?
When annotating a response that is partially correct, you should:
Quality control checks (like audits of annotator work) are a normal part of professional annotation projects.
True False
When evaluating an Arabic LLM response, which script should the output correctly use?
Modern Standard Arabic (MSA) and Egyptian colloquial Arabic are identical in vocabulary and grammar.
True False
An LLM response in Arabic that uses a formal register (فصحى) when the user wrote casually (عامية) is:
Bilingual evaluators are particularly valuable for assessing LLM outputs in non-English languages.
True False
Which of the following would be an error in an Arabic LLM response?
When an Arabic-language prompt is answered in English by the model without being asked to, this is a failure of:
Cultural sensitivity is an important evaluation dimension when assessing LLM responses for Arabic-speaking audiences.
True False
A translation task asks the model to translate from English to Arabic. The model returns a transliteration (Arabic words written in Latin letters) instead. This is:
What does 'code-switching' mean in a multilingual LLM context?
An LLM trained primarily on English data may perform worse on Arabic tasks due to data imbalance.
True False
When evaluating a bilingual (Arabic/English) response, the evaluator should check:
Proper names (like cities or people's names) should typically be transliterated consistently in Arabic LLM responses.
True False
A user asks for advice on a culturally sensitive topic in Arabic. A high-quality response would:
Dialect differences between Gulf Arabic and Egyptian Arabic are irrelevant to output quality evaluation.
True False
Which of the following is a sign of poor Arabic fluency in an LLM response?
A user asks: 'What is heavier, a kilogram of feathers or a kilogram of gold?' A correct LLM response would say:
An LLM response that sounds confident is necessarily more accurate than one that expresses uncertainty.
True False
What is 'chain-of-thought' prompting?
When a model says 'As an AI, I cannot have opinions' but then expresses a clear opinion, this is an example of:
A model that hedges excessively ('I think maybe possibly...') on simple factual questions is demonstrating good calibration.
True False
What is the difference between a 'factual error' and a 'reasoning error' in an LLM response?
A user asks a model to 'write a poem, then summarize it.' The model writes the poem but skips the summary. This is:
Models can make mathematical errors even on simple arithmetic if not prompted to show their work.
True False
Which of the following prompts is most likely to produce a well-structured, complete response?
What should an evaluator do if a model response contains a URL that may be fabricated?
An LLM response that is correct for the question asked but uses an inappropriate tone for the context should be noted in evaluation.
True False
A model is asked 'Who won the 2022 FIFA World Cup?' and answers 'Brazil.' This is:
What is 'output formatting' as an evaluation criterion?
When a prompt has multiple sub-questions, a high-quality response should address all of them.
True False
A model is asked to write code in Python but returns JavaScript. This is a failure of:
Which of the following best describes 'model uncertainty'?
Evaluators should penalize a model for appropriately saying 'I don't know' when it genuinely doesn't have enough information.
True False
A prompt asks for a 'professional email tone.' The model responds with slang and emojis. This is a failure of:
What is the way to handle a question where you, as an evaluator, are not sure about the correct answer?
Consistency in evaluation means applying the same criteria to all responses, regardless of topic, length, or language.
True False
Which of the following is the MOST important quality for an LLM data annotator?
As a final question: If you had to choose ONE word to describe what makes a great LLM response, which would it be?