The best writing I've seen about this is from Hamel Husain - https://hamel.dev/blog/posts/llm-judge/ and https://hamel.dev/blog/posts/evals-faq/ are both excellent."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Context engineering"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://chrisloy.dev/post/2025/08/03/context-engineering"}},"_tags":["comment","author_simonw","story_45788842"],"author":"simonw","comment_text":"This is so hard! I don't yet have a great solution for this myself, but I've been collecting notes about this on my "evals" tag for a while: https://simonwillison.net/tags/evals/
The best writing I've seen about this is from Hamel Husain - https://hamel.dev/blog/posts/llm-judge/ and https://hamel.dev/blog/posts/evals-faq/ are both excellent.","created_at":"2025-11-02T19:44:35Z","created_at_i":1762112675,"objectID":"45792830","parent_id":45792518,"story_id":45788842,"story_title":"Context engineering","story_url":"https://chrisloy.dev/post/2025/08/03/context-engineering","updated_at":"2026-03-05T23:00:37Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"TheIronYuppie"},"title":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["evals"],"value":"About AI Evals"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["evals","faq","hamel"],"value":"https://hamel.dev/blog/posts/evals-faq/"}},"_tags":["story","author_TheIronYuppie","story_44430117"],"author":"TheIronYuppie","children":[44454094,44454273,44454889,44455227,44455971,44456935,44457032,44457557,44458524,44458811,44459047,44460878,44474563],"created_at":"2025-07-01T02:48:16Z","created_at_i":1751338096,"num_comments":43,"objectID":"44430117","points":189,"story_id":44430117,"title":"About AI Evals","updated_at":"2026-08-17T07:59:00Z","url":"https://hamel.dev/blog/posts/evals-faq/"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"hamelsmu"},"title":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["evals"],"value":"Frequently Asked Questions About AI Evals"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["evals","faq","hamel"],"value":"https://hamel.dev/blog/posts/evals-faq/"}},"_tags":["story","author_hamelsmu","story_44423661"],"author":"hamelsmu","children":[44423697,44423808],"created_at":"2025-06-30T14:21:43Z","created_at_i":1751293303,"num_comments":1,"objectID":"44423661","points":8,"story_id":44423661,"title":"Frequently Asked Questions About AI Evals","updated_at":"2025-07-01T23:53:49Z","url":"https://hamel.dev/blog/posts/evals-faq/"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"tosh"},"title":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["evals"],"value":"AI Evals: Everything You Need to Know"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["evals","faq","hamel"],"value":"https://hamel.dev/blog/posts/evals-faq/"}},"_tags":["story","author_tosh","story_49832346"],"author":"tosh","created_at":"2026-09-24T15:46:51Z","created_at_i":1790264811,"num_comments":0,"objectID":"49832346","points":2,"story_id":49832346,"title":"AI Evals: Everything You Need to Know","updated_at":"2026-09-24T16:10:24Z","url":"https://hamel.dev/blog/posts/evals-faq/"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"pamelafox"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["evals","faq","hamel"],"value":"Fantastic FAQ, thank you Hamel for writing it up. We had an open space on AI Evals at Pycon this year, and had lots of discussion around similar questions. I only wrote down the questions, however:
# Evaluation Metrics & Methodology
* What metrics do you use (e.g., BERTScore, ROUGE, F1)? Are similarity metrics still useful?
* Do you use step-by-step evaluations or evaluate full responses?
* How do you evaluate VLM (vision-language model) summarization? Do you sample outputs or extract named entities?
* How do you approach offline (ground truth) vs. online evaluation?
* How do you handle uncertainty or "don\u2019t know" cases? (Temperature settings?)
* How do you evaluate multi-turn conversations?
* A/B comparisons and discrete labels (e.g., good/bad) are easier to interpret.
* It\u2019s important to counteract bias toward your own favorite eval questions\u2014ensure a diverse dataset.
## Prompting & Models
* Do you modify prompts based on the specific app being evaluated?
* Where do you store prompts\u2014text files, Prompty, database, or in code?
* Do you have domain experts edit or review prompts?
* How do you choose which model to use?
## Evaluation Infrastructure
* How do you choose an evaluation framework?
* What platforms do you use to gather domain expert feedback or labels?
* Do domain experts label outputs or also help with prompt design?
## User Feedback & Observability
* Do you collect thumbs up / thumbs down feedback?
* How does observability help identify failure modes?
* Do models tend to favor their own outputs? (There's research on this.)
I personally work on adding evaluation to our most popular Azure RAG samples, and put a Textual CLI interface in this repo that I've found helpful for reviewing the eval results:\nhttps://github.com/Azure-Samples/ai-rag-chat-evaluator"},"story_title":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["evals"],"value":"About AI Evals"},"story_url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["evals","faq","hamel"],"value":"https://hamel.dev/blog/posts/evals-faq/"}},"_tags":["comment","author_pamelafox","story_44430117"],"author":"pamelafox","children":[44457577,44458600],"comment_text":"Fantastic FAQ, thank you Hamel for writing it up. We had an open space on AI Evals at Pycon this year, and had lots of discussion around similar questions. I only wrote down the questions, however:
# Evaluation Metrics & Methodology
* What metrics do you use (e.g., BERTScore, ROUGE, F1)? Are similarity metrics still useful?
* Do you use step-by-step evaluations or evaluate full responses?
* How do you evaluate VLM (vision-language model) summarization? Do you sample outputs or extract named entities?
* How do you approach offline (ground truth) vs. online evaluation?
* How do you handle uncertainty or "don\u2019t know" cases? (Temperature settings?)
* How do you evaluate multi-turn conversations?
* A/B comparisons and discrete labels (e.g., good/bad) are easier to interpret.
* It\u2019s important to counteract bias toward your own favorite eval questions\u2014ensure a diverse dataset.
## Prompting & Models
* Do you modify prompts based on the specific app being evaluated?
* Where do you store prompts\u2014text files, Prompty, database, or in code?
* Do you have domain experts edit or review prompts?
* How do you choose which model to use?
## Evaluation Infrastructure
* How do you choose an evaluation framework?
* What platforms do you use to gather domain expert feedback or labels?
* Do domain experts label outputs or also help with prompt design?
## User Feedback & Observability
* Do you collect thumbs up / thumbs down feedback?
* How does observability help identify failure modes?
* Do models tend to favor their own outputs? (There's research on this.)
I personally work on adding evaluation to our most popular Azure RAG samples, and put a Textual CLI interface in this repo that I've found helpful for reviewing the eval results:\nhttps://github.com/Azure-Samples/ai-rag-chat-evaluator","created_at":"2025-07-03T16:46:51Z","created_at_i":1751561211,"objectID":"44456935","parent_id":44430117,"story_id":44430117,"story_title":"About AI Evals","story_url":"https://hamel.dev/blog/posts/evals-faq/","updated_at":"2025-07-20T03:52:29Z"}],"hitsPerPage":20,"nbHits":5,"nbPages":1,"page":0,"params":"query=evals+faq+hamel&advancedSyntax=true&analyticsTags=backend","processingTimeMS":22,"processingTimingsMS":{"_request":{"queue":132,"roundTrip":15},"afterFetch":{"merge":{"total":1},"total":1},"fetch":{"query":13,"scanning":6,"total":20},"total":22},"query":"evals faq hamel","serverTimeMS":155}