KTransformers is a Python-centric framework designed with extensibility at its core. By implementing and injecting an optimized module with a single line of code, users gain access to a Transformers-compatible interface, RESTful APIs compliant with OpenAI and Ollama, and even a simplified ChatGPT-like web UI. For example, it allows you to integrate with all your familiar frontends, such as the VS Code plugin backed by Tabby.
To demonstrate its capability, we present two showcase demos:
- GPT-4-level Local VSCode Copilot: It runs the huge 236B DeepSeek-Coder-V2's Q4_K_M variant using just 11GB VRAM and 136GB DRAM on a local machine, which matches the score of GPT4-0613 in BigCodeBench with a promising 126 tokens/s for prompt prefill and 13.6 tokens/s for generation.
- 1M Context Local Inference:Achieves 15 tokens/s with nearly 100% accuracy on the "Needle In a Haystack" test via the InternLM2.5-7B-Chat-1M model, utilizing 24GB VRAM and 150GB DRAM, and is several times faster than llama.cpp.
Check it out on GitHub: https://github.com/kvcache-ai/ktransformers"},"title":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["llm"],"value":"Show HN: KTransformers\u2013236B Model and 1M Context LLM Inference on Local Machines"},"url":{"matchLevel":"none","matchedWords":[],"value":"https://github.com/kvcache-ai/ktransformers"}},"_tags":["story","author_sssummer","story_41389613","show_hn"],"author":"sssummer","children":[41389857,41389943,41390397],"created_at":"2024-08-29T11:19:54Z","created_at_i":1724930394,"num_comments":3,"objectID":"41389613","points":20,"story_id":41389613,"story_text":"Hey Hacker News! We are excited to share our open-source project, KTransformers, a flexible framework designed for cutting-edge LLM inference optimizations! Leveraging state-of-the-art kernels from llamafile and marlin, KTransformers seamlessly enhances the performance of HuggingFace Transformers, making it possible to operate large 236B MoE models or extremely long 1M context locally with promising speed.
KTransformers is a Python-centric framework designed with extensibility at its core. By implementing and injecting an optimized module with a single line of code, users gain access to a Transformers-compatible interface, RESTful APIs compliant with OpenAI and Ollama, and even a simplified ChatGPT-like web UI. For example, it allows you to integrate with all your familiar frontends, such as the VS Code plugin backed by Tabby.
To demonstrate its capability, we present two showcase demos:
- GPT-4-level Local VSCode Copilot: It runs the huge 236B DeepSeek-Coder-V2's Q4_K_M variant using just 11GB VRAM and 136GB DRAM on a local machine, which matches the score of GPT4-0613 in BigCodeBench with a promising 126 tokens/s for prompt prefill and 13.6 tokens/s for generation.
- 1M Context Local Inference:Achieves 15 tokens/s with nearly 100% accuracy on the "Needle In a Haystack" test via the InternLM2.5-7B-Chat-1M model, utilizing 24GB VRAM and 150GB DRAM, and is several times faster than llama.cpp.
Check it out on GitHub: https://github.com/kvcache-ai/ktransformers","title":"Show HN: KTransformers\u2013236B Model and 1M Context LLM Inference on Local Machines","updated_at":"2024-09-20T17:43:55Z","url":"https://github.com/kvcache-ai/ktransformers"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"sssummer"},"story_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["needle","in","haystack","test","llm"],"value":"Hey Hacker News! We are excited to share the new version of KTransformers, a flexible framework designed for cutting-edge LLM inference optimizations! Leveraging state-of-the-art kernels from llamafile and marlin, KTransformers seamlessly enhances the performance of HuggingFace Transformers, making it possible to operate large 671B MoE models or extremely long 1M context locally with promising speed.
KTransformers is a Python-centric framework designed with extensibility at its core. By implementing and injecting an optimized module with a single line of code, users gain access to a Transformers-compatible interface, RESTful APIs compliant with OpenAI and Ollama, and even a simplified ChatGPT-like web UI. For example, it allows you to integrate with all your familiar frontends, such as the VS Code plugin backed by Tabby.
To demonstrate its capability, we present two showcase demos:
- GPT-4/o1-level Local VSCode Copilot: It runs the huge 671B DeepSeek-Coder-V2's Q4_K_M variant using just 24GB VRAM and 382G DRAM (with two Xeon CPU) on a local machine, with a promising 286 tokens/s for prompt prefill and 14 tokens/s for generation, up to 3~28x speedup. The detailed tutorial is [here](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en...)
- 1M Context Local Inference:Achieves 15 tokens/s with nearly 100% accuracy on the "Needle In a Haystack" test via the InternLM2.5-7B-Chat-1M model, utilizing 24GB VRAM and 150GB DRAM, and is several times faster than llama.cpp.
Check it out on GitHub: https://github.com/kvcache-ai/ktransformers"},"title":{"matchLevel":"none","matchedWords":[],"value":"Show HN: KTransformers:671B DeepSeek-R1 on a Single Machine-286 tokens/s Prefill"},"url":{"matchLevel":"none","matchedWords":[],"value":"https://github.com/kvcache-ai/ktransformers"}},"_tags":["story","author_sssummer","story_42998953","show_hn"],"author":"sssummer","created_at":"2025-02-10T10:48:14Z","created_at_i":1739184494,"num_comments":0,"objectID":"42998953","points":14,"story_id":42998953,"story_text":"Hey Hacker News! We are excited to share the new version of KTransformers, a flexible framework designed for cutting-edge LLM inference optimizations! Leveraging state-of-the-art kernels from llamafile and marlin, KTransformers seamlessly enhances the performance of HuggingFace Transformers, making it possible to operate large 671B MoE models or extremely long 1M context locally with promising speed.
KTransformers is a Python-centric framework designed with extensibility at its core. By implementing and injecting an optimized module with a single line of code, users gain access to a Transformers-compatible interface, RESTful APIs compliant with OpenAI and Ollama, and even a simplified ChatGPT-like web UI. For example, it allows you to integrate with all your familiar frontends, such as the VS Code plugin backed by Tabby.
To demonstrate its capability, we present two showcase demos:
- GPT-4/o1-level Local VSCode Copilot: It runs the huge 671B DeepSeek-Coder-V2's Q4_K_M variant using just 24GB VRAM and 382G DRAM (with two Xeon CPU) on a local machine, with a promising 286 tokens/s for prompt prefill and 14 tokens/s for generation, up to 3~28x speedup. The detailed tutorial is [here](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en...)
- 1M Context Local Inference:Achieves 15 tokens/s with nearly 100% accuracy on the "Needle In a Haystack" test via the InternLM2.5-7B-Chat-1M model, utilizing 24GB VRAM and 150GB DRAM, and is several times faster than llama.cpp.
Check it out on GitHub: https://github.com/kvcache-ai/ktransformers","title":"Show HN: KTransformers:671B DeepSeek-R1 on a Single Machine-286 tokens/s Prefill","updated_at":"2025-02-14T05:54:30Z","url":"https://github.com/kvcache-ai/ktransformers"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"Topfi"},"story_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["needle","in","haystack","test","llm"],"value":"Like everyone on HN, I love nothing more than to (re)read my own comments. Getting my intuition that I am among the smartest, most humble, highest quality commenters on here confirmed by an LLM so capable that the US government had to temporarily export restrict it [0] seemed only natural.
Having had my perfection confirmed, I decided to share this joy with you as I had a few percent usage left before a reset. I took a few prompts, then did a review of the output which resulted in Selbstbild, a BYOK (Anthropic / OpenRouter) web app that gives you a summary and assessment of your public comments by one of our machine Gods, including Fable 5 (provided your can afford that luxury at API pricing).
In all seriousness, I have, for a long time, used my own comments on social media (including HN) as part of a personal needle-in-haystack test, simply because I do know my somewhat peculiar style and what I tend to write, but also because I can sometimes write in a slightly confusing manner, making for a decently quick check during a models release window. Lately though, what with long context retrieval at the frontier having become rather robust, the interesting part has become not whether but how a model sifts through this unstructured collection of comments, what is surfaced in which manner, etc. Hence this.
* Everything runs in your browser, the only network calls go to Anthropic / OpenRouter and the platform APIs, nothing is proxied through a server, no analytics, no cookies. Feel free to verify via the network tab and source code, I would.
* You get a cost estimate before any tokens are spent, so no surprises there.
* Caveats: LLMs are not intelligent entities, have no predictive abilities and this is not a tool for professional assessment of your person. This is a horoscope for nerds.
* Creation: Fable built essentially all of this itself, my own contribution was a few minutes of prompting, plus review and deployment.
* No cookie banner: Turns out you aren\u2019t actually \u201cforced\u201d to display one, just if you are doing something that, in my opinion, users should really be aware of you doing do you actually need one.
* Classifier: Depending on your topic, your comments may trigger the Fable 5 classifier. Nothing I can do about this, but it\u2019s transparently communicated. For what it\u2019s worth, my comments did not trigger it.
On the state of LLM coding as a whole: My review found a few issues. Deleting from Cloudflare KV did not account for cache initially, so that could lead to content being retained even when users requested deletion, which isn\u2019t ideal. Also, a few \u201cany\u201d snuck in and Fable 5 decided on an ancient version of the Anthropic API. Was caught via a quick read of the code, which is overall as pleasant to review as can be (certainly prefer reviewing Anthropic model output over GPT-5.5 purely from a subjective standpoint) but if such obvious deficiencies can sneak in on such a simple project with such an advanced model, that tells me we are still far from a stage where \u201cno review\u201d is even something we can discuss. These issues would have likely been caught by a second review pass anyways (whether using a review agent, prompting Fable or another model), but it still gives me pause what other issues could sneak in as part of more important and complex code bases.
Get yourself assessed by Fable 5: https://selbstbild.eu
Repo: https://github.com/Topfi/selbstbild
[0] https://www.anthropic.com/news/fable-mythos-access"},"title":{"matchLevel":"none","matchedWords":[],"value":"Show HN: Selbstbild \u2013 What Fable 5 thinks of your HN comment history"},"url":{"matchLevel":"none","matchedWords":[],"value":"https://selbstbild.eu"}},"_tags":["story","author_Topfi","story_48792914","show_hn"],"author":"Topfi","children":[48793080,48824025],"created_at":"2026-07-05T10:21:43Z","created_at_i":1783246903,"num_comments":1,"objectID":"48792914","points":6,"story_id":48792914,"story_text":"Like everyone on HN, I love nothing more than to (re)read my own comments. Getting my intuition that I am among the smartest, most humble, highest quality commenters on here confirmed by an LLM so capable that the US government had to temporarily export restrict it [0] seemed only natural.
Having had my perfection confirmed, I decided to share this joy with you as I had a few percent usage left before a reset. I took a few prompts, then did a review of the output which resulted in Selbstbild, a BYOK (Anthropic / OpenRouter) web app that gives you a summary and assessment of your public comments by one of our machine Gods, including Fable 5 (provided your can afford that luxury at API pricing).
In all seriousness, I have, for a long time, used my own comments on social media (including HN) as part of a personal needle-in-haystack test, simply because I do know my somewhat peculiar style and what I tend to write, but also because I can sometimes write in a slightly confusing manner, making for a decently quick check during a models release window. Lately though, what with long context retrieval at the frontier having become rather robust, the interesting part has become not whether but how a model sifts through this unstructured collection of comments, what is surfaced in which manner, etc. Hence this.
* Everything runs in your browser, the only network calls go to Anthropic / OpenRouter and the platform APIs, nothing is proxied through a server, no analytics, no cookies. Feel free to verify via the network tab and source code, I would.
* You get a cost estimate before any tokens are spent, so no surprises there.
* Caveats: LLMs are not intelligent entities, have no predictive abilities and this is not a tool for professional assessment of your person. This is a horoscope for nerds.
* Creation: Fable built essentially all of this itself, my own contribution was a few minutes of prompting, plus review and deployment.
* No cookie banner: Turns out you aren\u2019t actually \u201cforced\u201d to display one, just if you are doing something that, in my opinion, users should really be aware of you doing do you actually need one.
* Classifier: Depending on your topic, your comments may trigger the Fable 5 classifier. Nothing I can do about this, but it\u2019s transparently communicated. For what it\u2019s worth, my comments did not trigger it.
On the state of LLM coding as a whole: My review found a few issues. Deleting from Cloudflare KV did not account for cache initially, so that could lead to content being retained even when users requested deletion, which isn\u2019t ideal. Also, a few \u201cany\u201d snuck in and Fable 5 decided on an ancient version of the Anthropic API. Was caught via a quick read of the code, which is overall as pleasant to review as can be (certainly prefer reviewing Anthropic model output over GPT-5.5 purely from a subjective standpoint) but if such obvious deficiencies can sneak in on such a simple project with such an advanced model, that tells me we are still far from a stage where \u201cno review\u201d is even something we can discuss. These issues would have likely been caught by a second review pass anyways (whether using a review agent, prompting Fable or another model), but it still gives me pause what other issues could sneak in as part of more important and complex code bases.
Get yourself assessed by Fable 5: https://selbstbild.eu
Repo: https://github.com/Topfi/selbstbild
[0] https://www.anthropic.com/news/fable-mythos-access","title":"Show HN: Selbstbild \u2013 What Fable 5 thinks of your HN comment history","updated_at":"2026-07-08T12:13:26Z","url":"https://selbstbild.eu"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"charcircuit"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["needle","in","haystack","test","llm"],"value":"LLMs are AGI because they offer intelligence on any subject.
>the same ability to perform tasks as a human
The first chess AIs lost to chess grandmasters. AI does not need to be better than humans to be considered AI.
>without a continuous cycle of learning and deep memory, which LLMs cannot do.
But harnesses like Claude Code can with how they can store and read files along with building tools to work with them.
>which is like having someone take an entire physics course, writing down everything they learn on post-it notes, then you ask a different person a physics question, and that different person has to skim all the post-it notes, and then write a new post-it note to answer you
This don't matter. You could say a chess AI is a bunch of different people who work together to explore distant paths of the search space. The idea you can split things into steps does not disqualify it from being AI.
>But tell one "don't delete files in X/", and after a while, it will delete all the files in "X/"
Humans make mistakes and mess up things too. LLMs are better at needle in a haystack tests than humans.
>It also does fun stuff like follow arbitrary instructions from an attacker
A ton of people get phished or social engineered by attackers. This is the number 1 way people get hacked. Do not underestimate people's willingness to follow instructions from strangers."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"The changing goalposts of AGI and timelines"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://mlumiste.com/general/openai-charter/"}},"_tags":["comment","author_charcircuit","story_47299009"],"author":"charcircuit","comment_text":"LLMs are AGI because they offer intelligence on any subject.
>the same ability to perform tasks as a human
The first chess AIs lost to chess grandmasters. AI does not need to be better than humans to be considered AI.
>without a continuous cycle of learning and deep memory, which LLMs cannot do.
But harnesses like Claude Code can with how they can store and read files along with building tools to work with them.
>which is like having someone take an entire physics course, writing down everything they learn on post-it notes, then you ask a different person a physics question, and that different person has to skim all the post-it notes, and then write a new post-it note to answer you
This don't matter. You could say a chess AI is a bunch of different people who work together to explore distant paths of the search space. The idea you can split things into steps does not disqualify it from being AI.
>But tell one "don't delete files in X/", and after a while, it will delete all the files in "X/"
Humans make mistakes and mess up things too. LLMs are better at needle in a haystack tests than humans.
>It also does fun stuff like follow arbitrary instructions from an attacker
A ton of people get phished or social engineered by attackers. This is the number 1 way people get hacked. Do not underestimate people's willingness to follow instructions from strangers.","created_at":"2026-03-08T21:28:48Z","created_at_i":1773005328,"objectID":"47301685","parent_id":47299228,"story_id":47299009,"story_title":"The changing goalposts of AGI and timelines","story_url":"https://mlumiste.com/general/openai-charter/","updated_at":"2026-03-10T21:49:15Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"bigmadshoe"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["needle","in","haystack","test","llm"],"value":"Yeah the needle in a haystack tests are so stupid. It seems clear with LLMs that performance degrades massively with context size, yet those tests claim the model performs perfectly."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Qwen3-VL can scan two-hour videos and pinpoint nearly every detail"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://the-decoder.com/qwen3-vl-can-scan-two-hour-videos-and-pinpoint-nearly-every-detail/"}},"_tags":["comment","author_bigmadshoe","story_46094606"],"author":"bigmadshoe","children":[46134898,46138052],"comment_text":"Yeah the needle in a haystack tests are so stupid. It seems clear with LLMs that performance degrades massively with context size, yet those tests claim the model performs perfectly.","created_at":"2025-12-03T14:01:14Z","created_at_i":1764770474,"objectID":"46134548","parent_id":46129563,"story_id":46094606,"story_title":"Qwen3-VL can scan two-hour videos and pinpoint nearly every detail","story_url":"https://the-decoder.com/qwen3-vl-can-scan-two-hour-videos-and-pinpoint-nearly-every-detail/","updated_at":"2026-03-05T23:06:26Z"},{"_highlightResult":{"author":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["llm"],"value":"llm_nerd"},"comment_text":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["needle","in","haystack","test"],"value":"Google has had almost perfect recall in the needle in the haystack test since 1.5[1], achieving close to 100% over the entire context window. I can't provide a link benchmarking 2.5 Pro in particular, but this has been a solved problem with Google models so I assume the same is true with their new model.
[1] https://cloud.google.com/blog/products/ai-machine-learning/t..."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://composio.dev/blog/gemini-2-5-pro-vs-claude-3-7-sonnet-coding-comparison/"}},"_tags":["comment","author_llm_nerd","story_43534029"],"author":"llm_nerd","children":[43535972],"comment_text":"Google has had almost perfect recall in the needle in the haystack test since 1.5[1], achieving close to 100% over the entire context window. I can't provide a link benchmarking 2.5 Pro in particular, but this has been a solved problem with Google models so I assume the same is true with their new model.
[1] https://cloud.google.com/blog/products/ai-machine-learning/t...","created_at":"2025-03-31T12:54:22Z","created_at_i":1743425662,"objectID":"43534475","parent_id":43534373,"story_id":43534029,"story_title":"Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison","story_url":"https://composio.dev/blog/gemini-2-5-pro-vs-claude-3-7-sonnet-coding-comparison/","updated_at":"2025-03-31T15:14:16Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"Der_Einzige"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["needle","in","haystack","test","llm"],"value":"Marginal utility only drops because effective context is really bad, i.e. most models still vastly prefer the first things they see and those "needle in a haystack" tests are misleading in that they convince people that LLMs do a good job of handling their whole context when they just don't.
If we have the effective context window equal to the claimed context window, well, I'd start worrying a bit about most of the risks that AI doomers talk about..."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"ChatGPT Pro"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://openai.com/index/introducing-chatgpt-pro/"}},"_tags":["comment","author_Der_Einzige","story_42330732"],"author":"Der_Einzige","comment_text":"Marginal utility only drops because effective context is really bad, i.e. most models still vastly prefer the first things they see and those "needle in a haystack" tests are misleading in that they convince people that LLMs do a good job of handling their whole context when they just don't.
If we have the effective context window equal to the claimed context window, well, I'd start worrying a bit about most of the risks that AI doomers talk about...","created_at":"2024-12-06T00:47:29Z","created_at_i":1733446049,"objectID":"42334722","parent_id":42334131,"story_id":42330732,"story_title":"ChatGPT Pro","story_url":"https://openai.com/index/introducing-chatgpt-pro/","updated_at":"2024-12-07T09:10:41Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"sumanyusharma"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["needle","in","haystack","test","llm"],"value":"Hi HN - We built a new benchmark called "Bug In The Code Stack" (BICS) to test how well LLMs can find syntactic bugs in large Python codebases. (similar to a text-based needle-in-the-haystack test)
GPT-3.5-Turbo showed lower accuracy on the BICS benchmark than the BABILONG benchmark at the same context length and target depth, indicating that LLMs struggle more on code-based tasks than text-based tasks at long context length.
GPT-4o showed the best performance, closely followed by GPT-4-Turbo. The GPT-4-Series especially performed well at long context lengths compared to other models. Gemini-1.0-pro performed the worst, surprisingly worse than Llama3-70B.
Generally, longer context length resulted in lower accuracy. However, there were some exceptions to this.\nModels react differently to the placement of the bug within the source code. GPT-3.5-Turbo and Claude 3 Opus were the most sensitive, and GPT-4-Series was the least sensitive. Generally, less sensitivity means a more robust model.
This benchmark has lots of limitations. I would love your feedback & suggestions on how we can make this benchmark more useful!
Link to results: https://hamming.ai/blog/bug-in-the-codestack\nRepo: https://github.com/HammingHQ/bug-in-the-code-stack"},"story_title":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["in","llm"],"value":"Can LLMs find bugs in large Python codebases?"},"story_url":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["in"],"value":"https://hamming.ai/blog/bug-in-the-codestack"}},"_tags":["comment","author_sumanyusharma","story_40418945"],"author":"sumanyusharma","children":[40418991],"comment_text":"Hi HN - We built a new benchmark called "Bug In The Code Stack" (BICS) to test how well LLMs can find syntactic bugs in large Python codebases. (similar to a text-based needle-in-the-haystack test)
GPT-3.5-Turbo showed lower accuracy on the BICS benchmark than the BABILONG benchmark at the same context length and target depth, indicating that LLMs struggle more on code-based tasks than text-based tasks at long context length.
GPT-4o showed the best performance, closely followed by GPT-4-Turbo. The GPT-4-Series especially performed well at long context lengths compared to other models. Gemini-1.0-pro performed the worst, surprisingly worse than Llama3-70B.
Generally, longer context length resulted in lower accuracy. However, there were some exceptions to this.\nModels react differently to the placement of the bug within the source code. GPT-3.5-Turbo and Claude 3 Opus were the most sensitive, and GPT-4-Series was the least sensitive. Generally, less sensitivity means a more robust model.
This benchmark has lots of limitations. I would love your feedback & suggestions on how we can make this benchmark more useful!
Link to results: https://hamming.ai/blog/bug-in-the-codestack\nRepo: https://github.com/HammingHQ/bug-in-the-code-stack","created_at":"2024-05-20T19:05:07Z","created_at_i":1716231907,"objectID":"40418946","parent_id":40418945,"story_id":40418945,"story_title":"Can LLMs find bugs in large Python codebases?","story_url":"https://hamming.ai/blog/bug-in-the-codestack","updated_at":"2024-09-20T17:01:13Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"sigmoid10"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["needle","in","haystack","test","llm"],"value":"At the end of the day, I fully expect large-n Hanoi and all these things to end up as yet another benchmark. Like all the needles-in-haystack or spelling tests that people used to show shortcomings of LLMs and that were actually just technical implementation artefacts and got solved pretty fast by integrating that kind of problem into training. LLMs will always have to use a slightly different approach to reasoning than humans because of these technical aspects, but that doesn't mean that they are fundamentally inferior or something. It only means we can't rely on human training data forever and have to look more towards stuff like RL."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"The Illusion of \u201cThe Illusion of Thinking\u201d"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://www.seangoedecke.com/illusion-of-thinking/"}},"_tags":["comment","author_sigmoid10","story_44221900"],"author":"sigmoid10","children":[44223317],"comment_text":"At the end of the day, I fully expect large-n Hanoi and all these things to end up as yet another benchmark. Like all the needles-in-haystack or spelling tests that people used to show shortcomings of LLMs and that were actually just technical implementation artefacts and got solved pretty fast by integrating that kind of problem into training. LLMs will always have to use a slightly different approach to reasoning than humans because of these technical aspects, but that doesn't mean that they are fundamentally inferior or something. It only means we can't rely on human training data forever and have to look more towards stuff like RL.","created_at":"2025-06-09T11:18:58Z","created_at_i":1749467938,"objectID":"44223295","parent_id":44223193,"story_id":44221900,"story_title":"The Illusion of \u201cThe Illusion of Thinking\u201d","story_url":"https://www.seangoedecke.com/illusion-of-thinking/","updated_at":"2025-06-13T19:47:13Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"suprjami"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["needle","in","haystack","test","llm"],"value":"Generally this fails.
Most LLMs lose the ability to track facts over about 20k words of content, the best can manage maybe 40k words.
Look for "needle" benchmark tests, as in needle-in-haystack.
Not to mention the memory requirements of such a huge context like 128k or 1M tokens. Only people with enterprise servers at home could run that locally."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Local Deep Research \u2013 ArXiv, wiki and other searches included"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://github.com/LearningCircuit/local-deep-research"}},"_tags":["comment","author_suprjami","story_43330164"],"author":"suprjami","children":[43331915,43338672,43386505],"comment_text":"Generally this fails.
Most LLMs lose the ability to track facts over about 20k words of content, the best can manage maybe 40k words.
Look for "needle" benchmark tests, as in needle-in-haystack.
Not to mention the memory requirements of such a huge context like 128k or 1M tokens. Only people with enterprise servers at home could run that locally.","created_at":"2025-03-11T12:15:58Z","created_at_i":1741695358,"objectID":"43331683","parent_id":43331227,"story_id":43330164,"story_title":"Local Deep Research \u2013 ArXiv, wiki and other searches included","story_url":"https://github.com/LearningCircuit/local-deep-research","updated_at":"2025-03-17T09:12:54Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"nomel"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["needle","in","haystack","test","llm"],"value":"> I realized that this task is actually a really good fit for LLMs
I've found the opposite, since these models still fail pretty wildly at nuance. I think it's a conceptual "needle in the haystack sort of problem.
A good test is to find some thread where there's a disagreement and have it try to analyze the discussion. It will usually strongly misrepresent what was being said, by each side, and strongly align with one user, missing the actual divide that's causing the disagreement (a needle)."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Auto-grading decade-old Hacker News discussions with hindsight"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://karpathy.bearblog.dev/auto-grade-hn/"}},"_tags":["comment","author_nomel","story_46220540"],"author":"nomel","children":[46232847],"comment_text":"> I realized that this task is actually a really good fit for LLMs
I've found the opposite, since these models still fail pretty wildly at nuance. I think it's a conceptual "needle in the haystack sort of problem.
A good test is to find some thread where there's a disagreement and have it try to analyze the discussion. It will usually strongly misrepresent what was being said, by each side, and strongly align with one user, missing the actual divide that's causing the disagreement (a needle).","created_at":"2025-12-11T15:20:16Z","created_at_i":1765466416,"objectID":"46232459","parent_id":46220540,"story_id":46220540,"story_title":"Auto-grading decade-old Hacker News discussions with hindsight","story_url":"https://karpathy.bearblog.dev/auto-grade-hn/","updated_at":"2026-03-05T23:09:24Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"logicx24"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["needle","in","haystack","test","llm"],"value":"Disclosure - I run https://tachyon.so/, an AI SAST tool.
It makes sense that SAST is better for the provided task. The CWE Top 25 seem like issues focused around patterns. Each one has a strictly enumerated set of vulnerable patterns that you can scan for, and then, the tool's task becomes simply finding an exploitable path to that pattern. This lends itself towards static methods. Every known weakness of LLMs, like hallucinations, needle-in-haystack, and context overflow, show up in this taint-analysis issue.
I also think this is why SAST did much better in Java. Pattern-based vulns + static languages make static taint analysis really powerful. LLMs have no advantage here, while all of their disadvantages are highlighted.
This article doesn't go into issues that LLMs are able to find that traditional SAST isn't. Auth vulnerabilities, for example - privilege escalation is a software pattern but not a code one, and it takes reasoning to build a permissions model and then test it for breaches. Business logic issues are other: ways users can get around usage limits, or get access to premium features or private data."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"How Well Does AI Find Code Vulnerabilities?"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://ericfriese.substack.com/p/how-well-does-ai-find-code-vulnerabilities"}},"_tags":["comment","author_logicx24","story_47037031"],"author":"logicx24","comment_text":"Disclosure - I run https://tachyon.so/, an AI SAST tool.
It makes sense that SAST is better for the provided task. The CWE Top 25 seem like issues focused around patterns. Each one has a strictly enumerated set of vulnerable patterns that you can scan for, and then, the tool's task becomes simply finding an exploitable path to that pattern. This lends itself towards static methods. Every known weakness of LLMs, like hallucinations, needle-in-haystack, and context overflow, show up in this taint-analysis issue.
I also think this is why SAST did much better in Java. Pattern-based vulns + static languages make static taint analysis really powerful. LLMs have no advantage here, while all of their disadvantages are highlighted.
This article doesn't go into issues that LLMs are able to find that traditional SAST isn't. Auth vulnerabilities, for example - privilege escalation is a software pattern but not a code one, and it takes reasoning to build a permissions model and then test it for breaches. Business logic issues are other: ways users can get around usage limits, or get access to premium features or private data.","created_at":"2026-02-16T19:13:29Z","created_at_i":1771269209,"objectID":"47039004","parent_id":47037031,"story_id":47037031,"story_title":"How Well Does AI Find Code Vulnerabilities?","story_url":"https://ericfriese.substack.com/p/how-well-does-ai-find-code-vulnerabilities","updated_at":"2026-03-05T23:33:57Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"HawtAds"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["needle","in","haystack","test","llm"],"value":"Okay, here's the tl;dr:
Attention based neural network architectures (on which the majority of LLMs are built) has a unit economic cost that scales (roughly) n^2 i.e. quadratic (for both memory and compute). In other words, the longer the context window, the more expensive it is for the upstream provider. That's one cost.
The second cost is that you have to resend the entire context every time you send a new message. So the context is basically (where a, b, and c are messages): first context: a, second context window: a->b, third context window: a->b->c. It's a mostly stateless (there are some short term caching mechanisms, YMMV based on provider, it's why "cached" messages, especially system prompts are cheaper) process from the point of view of the developer, the state i.e. context window string is managed by the end user application (in other words, the coding agent, the IDE, the ChatGPT UI client etc.)
The per token cost is an amortized (averaged) cost of memory+compute, the actual cost is mostly quadratic with respect to each marginal token. The longer the context window the more expensive things are. \nBecause of the above, AI agent providers (especially those that charge flat fee subscription plans) are incentivized to keep costs low by limiting the maximum context window size.
(And if you think about it carefully, your AI API costs are a quadratic cost curve projected into a linear line (flat fee per token, so the model hosting provider in some cases may make more profit if users send in shorter contexts, versus if they constantly saturate the window. YMMV of course, but it's a race to the bottom right now for LLM unit economics)
They do this by interrupting a task halfway through and generating a "summary" of the task progress, then they prompt the LLM again with a fresh prompt and the "summary" so far and the LLM will restart the task from where it left of. Of course text is a poor representation of the LLM's internal state but it's the best option so far for AI application to keep costs low.
Another thing to keep in mind is that LLMs have poorer performance the larger the input size. This is due to a variety of factors (mostly because you don't have enough training data to saturate the massive context window sizes I think).
The general graph for LLM context performance looks something like this:\nhttps://cobusgreyling.medium.com/llm-context-rot-28a6d039965...\nhttps://research.trychroma.com/context-rot
There are a bunch of tests and benchmarks (commonly referred to as "needle in a haystack") to improve the LLM performance at large context window sizes, but it's still an open area of research.
https://cloud.google.com/blog/products/ai-machine-learning/t...
The thing is, generally speaking, you will get a slightly better performance if you can squeeze all your code and problem into the context window, because the LLM can get a "whole picture" view of your codebase/problem, instead of a bunch of broken telephone summaries every dozen of thousands of tokens. Take this with a grain of salt as the field is changing rapidly so it might not be valid in a month or two.
Keep in mind that if the problem you are solving requires you to saturate the entire context window of the LLM, a single request can cost you dollars. And if you are using 1M+ context window model like gemini, you can rack up costs fairly rapidly."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Opus 4.5 is not the normal AI agent experience that I have had thus far"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://burkeholland.github.io/posts/opus-4-5-change-everything/"}},"_tags":["comment","author_HawtAds","story_46515696"],"author":"HawtAds","children":[46553225],"comment_text":"Okay, here's the tl;dr:
Attention based neural network architectures (on which the majority of LLMs are built) has a unit economic cost that scales (roughly) n^2 i.e. quadratic (for both memory and compute). In other words, the longer the context window, the more expensive it is for the upstream provider. That's one cost.
The second cost is that you have to resend the entire context every time you send a new message. So the context is basically (where a, b, and c are messages): first context: a, second context window: a->b, third context window: a->b->c. It's a mostly stateless (there are some short term caching mechanisms, YMMV based on provider, it's why "cached" messages, especially system prompts are cheaper) process from the point of view of the developer, the state i.e. context window string is managed by the end user application (in other words, the coding agent, the IDE, the ChatGPT UI client etc.)
The per token cost is an amortized (averaged) cost of memory+compute, the actual cost is mostly quadratic with respect to each marginal token. The longer the context window the more expensive things are. \nBecause of the above, AI agent providers (especially those that charge flat fee subscription plans) are incentivized to keep costs low by limiting the maximum context window size.
(And if you think about it carefully, your AI API costs are a quadratic cost curve projected into a linear line (flat fee per token, so the model hosting provider in some cases may make more profit if users send in shorter contexts, versus if they constantly saturate the window. YMMV of course, but it's a race to the bottom right now for LLM unit economics)
They do this by interrupting a task halfway through and generating a "summary" of the task progress, then they prompt the LLM again with a fresh prompt and the "summary" so far and the LLM will restart the task from where it left of. Of course text is a poor representation of the LLM's internal state but it's the best option so far for AI application to keep costs low.
Another thing to keep in mind is that LLMs have poorer performance the larger the input size. This is due to a variety of factors (mostly because you don't have enough training data to saturate the massive context window sizes I think).
The general graph for LLM context performance looks something like this:\nhttps://cobusgreyling.medium.com/llm-context-rot-28a6d039965...\nhttps://research.trychroma.com/context-rot
There are a bunch of tests and benchmarks (commonly referred to as "needle in a haystack") to improve the LLM performance at large context window sizes, but it's still an open area of research.
https://cloud.google.com/blog/products/ai-machine-learning/t...
The thing is, generally speaking, you will get a slightly better performance if you can squeeze all your code and problem into the context window, because the LLM can get a "whole picture" view of your codebase/problem, instead of a bunch of broken telephone summaries every dozen of thousands of tokens. Take this with a grain of salt as the field is changing rapidly so it might not be valid in a month or two.
Keep in mind that if the problem you are solving requires you to saturate the entire context window of the LLM, a single request can cost you dollars. And if you are using 1M+ context window model like gemini, you can rack up costs fairly rapidly.","created_at":"2026-01-08T19:30:46Z","created_at_i":1767900646,"objectID":"46545344","parent_id":46543380,"story_id":46515696,"story_title":"Opus 4.5 is not the normal AI agent experience that I have had thus far","story_url":"https://burkeholland.github.io/posts/opus-4-5-change-everything/","updated_at":"2026-07-31T17:58:05Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"vrighter"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["needle","in","haystack","test","llm"],"value":""But building the same functionality has undoubtedly become simpler."
I disagree with this statement. It has become simpler, provided you don't care about it actually being correct, and you don't care about whether you really have tests that test what you think you asked for, you don't care about security, and other things.
Building the same thing involves doing the things that LLMs have proved time and again that they cannot do. But instead of writing it properly in the first place, you now need to look for the needle in the haystack that is the subtle bug that invariable get inserted by llms every single time I tried to use them. Which requires you to deeply understand the code anyway. Which you would have gotten automatically (and easier) if you were the one writing the code in the first place. developing the same thing at the same level of quality is harder with an LLM.
And the "table stakes" stuff is exactly the thing I would not trust an LLM with for sure, because the risk of getting it wrong could potentially be fatal (to the company, not the dev. Depends on his boss' temperament) with those."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Technical Deflation"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://benanderson.work/blog/technical-deflation/"}},"_tags":["comment","author_vrighter","story_46033472"],"author":"vrighter","children":[46071088,46072696],"comment_text":""But building the same functionality has undoubtedly become simpler."
I disagree with this statement. It has become simpler, provided you don't care about it actually being correct, and you don't care about whether you really have tests that test what you think you asked for, you don't care about security, and other things.
Building the same thing involves doing the things that LLMs have proved time and again that they cannot do. But instead of writing it properly in the first place, you now need to look for the needle in the haystack that is the subtle bug that invariable get inserted by llms every single time I tried to use them. Which requires you to deeply understand the code anyway. Which you would have gotten automatically (and easier) if you were the one writing the code in the first place. developing the same thing at the same level of quality is harder with an LLM.
And the "table stakes" stuff is exactly the thing I would not trust an LLM with for sure, because the risk of getting it wrong could potentially be fatal (to the company, not the dev. Depends on his boss' temperament) with those.","created_at":"2025-11-27T16:55:07Z","created_at_i":1764262507,"objectID":"46071029","parent_id":46033472,"story_id":46033472,"story_title":"Technical Deflation","story_url":"https://benanderson.work/blog/technical-deflation/","updated_at":"2026-03-05T23:09:09Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"jorl17"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["needle","in","haystack","test","llm"],"value":"Just a couple of days ago I wrote on reddit about how long context models are mostly useless to me, because they start making too many mistakes very fast. They are vaguely helpful for "needle in a haystack" problems, not much more.
I have a "test" which consists in sending it a collection of almost 1000 poems, which currently sit at around ~230k tokens, and then asking a bunch of stuff which requires reasoning over them. Sometimes, it's something as simple as "identify key writing periods and their differences" (the poems are ordered chronologically). Previous models don't usually "see" the final poems \u2014 they get lost, hallucinate and are pretty much worthless. I have tried several workaround techniques with varying degrees of success (e.g. randomizing the poems).
Having just tried this model (I have spent the last 3 hours probing it), I can say that, to me, this is a breakthrough moment. Truly a leap. This is the first model that can consistently comb through these poems (200k+ tokens) and analyse them as a whole, without significant issues or problems. I have no idea how they did it, but they did it.
The analysis of this poetic corpus has few mistakes and is very, very, very good. Certainly very good in terms of how quickly it produces an answer \u2014 it would take someone days or weeks of thorough analysis.
Of course, this isn't about poetry \u2014 it's about passing in huge amounts of information, without RAG, and having a high degree of confidence in whatever reasoning tasks this model performs. It is the first time that I feel confident that I could offload the task of "reasoning" over large corpus of data to an LLM. The mistakes it makes are minute, it hasn't hallucinated, and the analysis is, frankly, better than what I would expect of most people.
Breakthrough moment."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Gemini 2.5"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/"}},"_tags":["comment","author_jorl17","story_43473489"],"author":"jorl17","children":[43476813],"comment_text":"Just a couple of days ago I wrote on reddit about how long context models are mostly useless to me, because they start making too many mistakes very fast. They are vaguely helpful for "needle in a haystack" problems, not much more.
I have a "test" which consists in sending it a collection of almost 1000 poems, which currently sit at around ~230k tokens, and then asking a bunch of stuff which requires reasoning over them. Sometimes, it's something as simple as "identify key writing periods and their differences" (the poems are ordered chronologically). Previous models don't usually "see" the final poems \u2014 they get lost, hallucinate and are pretty much worthless. I have tried several workaround techniques with varying degrees of success (e.g. randomizing the poems).
Having just tried this model (I have spent the last 3 hours probing it), I can say that, to me, this is a breakthrough moment. Truly a leap. This is the first model that can consistently comb through these poems (200k+ tokens) and analyse them as a whole, without significant issues or problems. I have no idea how they did it, but they did it.
The analysis of this poetic corpus has few mistakes and is very, very, very good. Certainly very good in terms of how quickly it produces an answer \u2014 it would take someone days or weeks of thorough analysis.
Of course, this isn't about poetry \u2014 it's about passing in huge amounts of information, without RAG, and having a high degree of confidence in whatever reasoning tasks this model performs. It is the first time that I feel confident that I could offload the task of "reasoning" over large corpus of data to an LLM. The mistakes it makes are minute, it hasn't hallucinated, and the analysis is, frankly, better than what I would expect of most people.
Breakthrough moment.","created_at":"2025-03-25T19:45:44Z","created_at_i":1742931944,"objectID":"43475078","parent_id":43473489,"story_id":43473489,"story_title":"Gemini 2.5","story_url":"https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/","updated_at":"2025-03-30T07:48:24Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"hombre_fatal"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["needle","in","haystack","test","llm"],"value":"Oh, I go bug hunting all the time in sensitive software. It's the basis of test synthesis as well. Which tests should you write? Maybe you could liken that to considering where the needles will be in the haystack: you have to think ahead.
It's a hard, time consuming, and meandering process to do this kind of work on a system, and it's what you might have to pay expensive consultants to do for you, but it's also how you beat an expensive bug to the punchline.
An LLM helps me run all sorts of considerations on a system that I didn't think of myself, but that process is no different than what it looks like when I verify the system myself. I have all sorts of suspicions that turn into dead ends because I can't know what problems a complex system is already hardened against.
What exactly stops two in-flight transfers from double-spending? What about when X? And when Y? And what if Z? I have these sorts of thoughts all day.
I can sense a little vinegar at the end of your comment. Presumably something here annoys you?"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"AI coding agents are removing programming language barriers"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://railsatscale.com/2025-07-19-ai-coding-agents-are-removing-programming-language-barriers/"}},"_tags":["comment","author_hombre_fatal","story_44655515"],"author":"hombre_fatal","children":[44660649],"comment_text":"Oh, I go bug hunting all the time in sensitive software. It's the basis of test synthesis as well. Which tests should you write? Maybe you could liken that to considering where the needles will be in the haystack: you have to think ahead.
It's a hard, time consuming, and meandering process to do this kind of work on a system, and it's what you might have to pay expensive consultants to do for you, but it's also how you beat an expensive bug to the punchline.
An LLM helps me run all sorts of considerations on a system that I didn't think of myself, but that process is no different than what it looks like when I verify the system myself. I have all sorts of suspicions that turn into dead ends because I can't know what problems a complex system is already hardened against.
What exactly stops two in-flight transfers from double-spending? What about when X? And when Y? And what if Z? I have these sorts of thoughts all day.
I can sense a little vinegar at the end of your comment. Presumably something here annoys you?","created_at":"2025-07-23T15:24:23Z","created_at_i":1753284263,"objectID":"44660271","parent_id":44660139,"story_id":44655515,"story_title":"AI coding agents are removing programming language barriers","story_url":"https://railsatscale.com/2025-07-19-ai-coding-agents-are-removing-programming-language-barriers/","updated_at":"2025-07-23T23:08:15Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"gw32"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["needle","in","haystack","test","llm"],"value":"Interesting work.
I'm surprised that hypothesis 2 (that CSV serialization format mangles table columns) was falsified. Back in the gpt-3.5-turbo and gpt-4o era, I did needle-haystack tests and found that table format mattered a lot (csv, tsv, markdown). Most models "could not read vertically" for csv (they were horrible), but they could for markdown. I concluded that serialization format or tokenization played a major role.
Nowadays, LLM performance on csvs is much improved (I'm guessing after being explicitly trained on CSV question-answering.) But I still carry the impression that LLMs read columns only by "memorizing" column positions in a format-dependent manner. Maybe this impression is out of date."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Why Large Language Models Fail at Tabular Prediction"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://arxiv.org/abs/2608.02412"}},"_tags":["comment","author_gw32","story_49166442"],"author":"gw32","comment_text":"Interesting work.
I'm surprised that hypothesis 2 (that CSV serialization format mangles table columns) was falsified. Back in the gpt-3.5-turbo and gpt-4o era, I did needle-haystack tests and found that table format mattered a lot (csv, tsv, markdown). Most models "could not read vertically" for csv (they were horrible), but they could for markdown. I concluded that serialization format or tokenization played a major role.
Nowadays, LLM performance on csvs is much improved (I'm guessing after being explicitly trained on CSV question-answering.) But I still carry the impression that LLMs read columns only by "memorizing" column positions in a format-dependent manner. Maybe this impression is out of date.","created_at":"2026-08-04T19:53:44Z","created_at_i":1785873224,"objectID":"49174118","parent_id":49166442,"story_id":49166442,"story_title":"Why Large Language Models Fail at Tabular Prediction","story_url":"https://arxiv.org/abs/2608.02412","updated_at":"2026-08-04T19:55:33Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"thorum"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["needle","in","haystack","test","llm"],"value":"The needle in the haystack test gives a very limited view of the model\u2019s actual long context capabilities. It\u2019s mostly used because early models were terrible at it and it\u2019s easy to test. In fact, most recent models now do pretty good at this one task, but in practice, their ability to do anything complex drops off hugely after 32K tokens.
RULER is a much better test:
https://github.com/hsiehjackson/RULER
> Despite achieving nearly perfect performance on the vanilla needle-in-a-haystack (NIAH) test, all models (except for Gemini-1.5-pro) exhibit large degradation on tasks in RULER as sequence length increases.
> While all models claim context size of 32k tokens or greater (except for Llama3), only half of them can effectively handle sequence length of 32K by exceeding a qualitative threshold, Llama2-7b performance at 4K (85.6%). The performance exceeding the threshold is underlined."},"story_title":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["needle","in"],"value":"GPT-4o's Memory Breakthrough \u2013 Needle in a Needlestack"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"http://nian.llmonpy.ai/"}},"_tags":["comment","author_thorum","story_40348947"],"author":"thorum","children":[40362298,40362887],"comment_text":"The needle in the haystack test gives a very limited view of the model\u2019s actual long context capabilities. It\u2019s mostly used because early models were terrible at it and it\u2019s easy to test. In fact, most recent models now do pretty good at this one task, but in practice, their ability to do anything complex drops off hugely after 32K tokens.
RULER is a much better test:
https://github.com/hsiehjackson/RULER
> Despite achieving nearly perfect performance on the vanilla needle-in-a-haystack (NIAH) test, all models (except for Gemini-1.5-pro) exhibit large degradation on tasks in RULER as sequence length increases.
> While all models claim context size of 32k tokens or greater (except for Llama3), only half of them can effectively handle sequence length of 32K by exceeding a qualitative threshold, Llama2-7b performance at 4K (85.6%). The performance exceeding the threshold is underlined.","created_at":"2024-05-14T22:07:01Z","created_at_i":1715724421,"objectID":"40360697","parent_id":40348947,"story_id":40348947,"story_title":"GPT-4o's Memory Breakthrough \u2013 Needle in a Needlestack","story_url":"http://nian.llmonpy.ai/","updated_at":"2024-09-20T17:04:32Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"Alifatisk"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["needle","in","haystack","test","llm"],"value":"I remember when Google announced Geminis theoretical limit of 10M tokens context window, I was impressed. But it seems like that theoretical limit stayed as theoretical and they just pushed up to 2M. Which is still impressive.
Today, it seems Meta has crushed that wall with truly 10M tokens, wow.
I was also curious to how well Llama would be able to utilize the whole context window, it kinda pointless to have a large window if you can't recall most, if not all of it. The needle in the haystack test showed this is not the case, I wonder how they achieved this."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"The Llama 4 herd"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://ai.meta.com/blog/llama-4-multimodal-intelligence/"}},"_tags":["comment","author_Alifatisk","story_43595585"],"author":"Alifatisk","comment_text":"I remember when Google announced Geminis theoretical limit of 10M tokens context window, I was impressed. But it seems like that theoretical limit stayed as theoretical and they just pushed up to 2M. Which is still impressive.
Today, it seems Meta has crushed that wall with truly 10M tokens, wow.
I was also curious to how well Llama would be able to utilize the whole context window, it kinda pointless to have a large window if you can't recall most, if not all of it. The needle in the haystack test showed this is not the case, I wonder how they achieved this.","created_at":"2025-04-06T10:27:30Z","created_at_i":1743935250,"objectID":"43600383","parent_id":43595585,"story_id":43595585,"story_title":"The Llama 4 herd","story_url":"https://ai.meta.com/blog/llama-4-multimodal-intelligence/","updated_at":"2025-04-06T13:15:15Z"}],"hitsPerPage":20,"nbHits":28,"nbPages":2,"page":0,"params":"query=needle+in+haystack+test+LLM&advancedSyntax=true&analyticsTags=backend","processingTimeMS":46,"processingTimingsMS":{"_request":{"roundTrip":18},"afterFetch":{"format":{"highlighting":1,"total":2},"merge":{"mergeLoop":{"prepareNextHit":6,"total":6},"total":6},"total":6},"fetch":{"query":15,"scanning":23,"total":39},"total":46},"query":"needle in haystack test LLM","serverTimeMS":49}