One could, perhaps, make the case that the inherent randomness in the LLM algorithm makes it a fundamentally different thing than an expert system.
... but we've had Monte Carlo analysis as a tool for ages too.
I'm not sure we promote the thing to "not a tool" by gluing a Monte Carlo algorithm onto an expert system.
To take a big step back from the LLM question and look at a similar space that perhaps sheds light on the question...
A lot of chip design these days involves a step where the desired circuit is described in a goals-and-constraints-oriented language and then fed to a layout algoirthm that is semi-deterministic and attempts to solve the optimization function based on specified goals (like fabrication cost, size, or power consumption).
The output of this process is a chip you can fab. The process is random (it's an important industry-standard step to capture the input seed, because if the chip ends up in something audited and the auditing firm needs to know how the chip was made, it's not good enough to shrug and say "Enh, we rolled some dice," they need the layout schema deterministically reproducible too).
I don't, generally, think anyone considers the fact we use these in the hardware space to be anything but using yet another tool. It's a tool that searches a solutions domain much faster than a human could doing something not too dissimilar from what a human would do (with perhaps a little human intuition thrown on the top to avoid obviously-pessimistic branches of exploration, but you can bake those shortcuts into the tool also)."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"AI Is Not a Tool"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://twitter.com/frascafrasca/status/2113637581426671850"}},"_tags":["comment","author_shadowgovt","story_50212079"],"author":"shadowgovt","comment_text":"An expert system is a tool, and we've had expert systems in computing for ages now.
One could, perhaps, make the case that the inherent randomness in the LLM algorithm makes it a fundamentally different thing than an expert system.
... but we've had Monte Carlo analysis as a tool for ages too.
I'm not sure we promote the thing to "not a tool" by gluing a Monte Carlo algorithm onto an expert system.
To take a big step back from the LLM question and look at a similar space that perhaps sheds light on the question...
A lot of chip design these days involves a step where the desired circuit is described in a goals-and-constraints-oriented language and then fed to a layout algoirthm that is semi-deterministic and attempts to solve the optimization function based on specified goals (like fabrication cost, size, or power consumption).
The output of this process is a chip you can fab. The process is random (it's an important industry-standard step to capture the input seed, because if the chip ends up in something audited and the auditing firm needs to know how the chip was made, it's not good enough to shrug and say "Enh, we rolled some dice," they need the layout schema deterministically reproducible too).
I don't, generally, think anyone considers the fact we use these in the hardware space to be anything but using yet another tool. It's a tool that searches a solutions domain much faster than a human could doing something not too dissimilar from what a human would do (with perhaps a little human intuition thrown on the top to avoid obviously-pessimistic branches of exploration, but you can bake those shortcuts into the tool also).","created_at":"2026-10-23T18:16:58Z","created_at_i":1792779418,"objectID":"50213228","parent_id":50212079,"story_id":50212079,"story_title":"AI Is Not a Tool","story_url":"https://twitter.com/frascafrasca/status/2113637581426671850","updated_at":"2026-10-23T18:58:26Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"jmyeet"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["llm","cost","optimization"],"value":"I'm a bit of an AI doomer but I'm going to disagree. I don't think this is the new mormal but there is going to be a dark transitional period.
What we're seeing is a bit of a "gold rush" where LLm advancements are coming thick and fast and they're being pointed out code bases that simply haven't had the level of scrutiny where relatively easy bugs to find have managed to survive. That's temporary. Here are my predictions:
1. Correctness and, in particular, memory safety is going to become increasingly important. I was bullish on Rust before this AI hype wave. LLMs finding and exploiting bugs has reinforced just important it is that Rust prevents as certain class of bugs that are the most prone to rmote exploitation ie certain memory safety issues.
I've long said that modern CPU architectures, out-of-order execution, hyperthreading (does that still exist? or did Intel fully kill it?), different CPU cores and compiler optimizations that it is essentially impossible for a human to write correct multithreaded code, particularly in userspace with "leaf node" code. LLMs just make that more apparent; and
2. Mature code bases for core functionality are going to become increasingly important because they've been "proven" in the wild (as much as you can prove a negative).
My sincerest hope, which is probably more wishful thinking than anything else, is that LLMs will be another nail in the coffin for C++, which is simply too complex and dragged down with legacy features and backwards compatibility to be salvageable."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Is this the new cyber normal?"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://drew.silcock.dev/blog/is-this-the-new-cyber-normal/"}},"_tags":["comment","author_jmyeet","story_50207115"],"author":"jmyeet","comment_text":"I'm a bit of an AI doomer but I'm going to disagree. I don't think this is the new mormal but there is going to be a dark transitional period.
What we're seeing is a bit of a "gold rush" where LLm advancements are coming thick and fast and they're being pointed out code bases that simply haven't had the level of scrutiny where relatively easy bugs to find have managed to survive. That's temporary. Here are my predictions:
1. Correctness and, in particular, memory safety is going to become increasingly important. I was bullish on Rust before this AI hype wave. LLMs finding and exploiting bugs has reinforced just important it is that Rust prevents as certain class of bugs that are the most prone to rmote exploitation ie certain memory safety issues.
I've long said that modern CPU architectures, out-of-order execution, hyperthreading (does that still exist? or did Intel fully kill it?), different CPU cores and compiler optimizations that it is essentially impossible for a human to write correct multithreaded code, particularly in userspace with "leaf node" code. LLMs just make that more apparent; and
2. Mature code bases for core functionality are going to become increasingly important because they've been "proven" in the wild (as much as you can prove a negative).
My sincerest hope, which is probably more wishful thinking than anything else, is that LLMs will be another nail in the coffin for C++, which is simply too complex and dragged down with legacy features and backwards compatibility to be salvageable.","created_at":"2026-10-23T12:37:55Z","created_at_i":1792759075,"objectID":"50208037","parent_id":50207115,"story_id":50207115,"story_title":"Is this the new cyber normal?","story_url":"https://drew.silcock.dev/blog/is-this-the-new-cyber-normal/","updated_at":"2026-10-23T12:39:56Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"everythingmeta"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["llm","cost","optimization"],"value":"One of the authors here. I work at Parallel. We got Jev-pilled a few weeks ago and have been trialing decision models nonstop across all sorts of use cases, so we wanted to see how the Jev-like models compare. The tasks are a mix: some where we previously used fine-tuned models, some where we used general LLMs, and some we hadn't pursued at all before Jev. What I think makes our analysis a little different is that we tested on human-labeled data from our production pipeline rather than synthetic benchmarks, which can be made to show almost whatever you want (I suspect that's how so many OSS models were ranking above Jev in early evals).
What we found:
The ergonomics of serverless decision models were really hard to beat. Cost, latency and accuracy were all near the top for Jev. Latency especially was very consistent, which matters a lot to us; historically that's where we'd have reached for a fine-tuned classifier.
Perplexity's decision model was a close second. It sometimes outperformed Jev, but its latency was more variable. That said, I believe it's also half the price now.
Self-hosting open-weight decision models wasn't a big unlock. In principle, at peak utilization we'd get into the ballpark of Jev's API performance. That said there is a world of optimizations that we just didn't have the time to pursue.
Iterating on prompts got meaningful improvements out of almost every model. That's much nicer UX than having to retrain a classifier.
Caveats: there's already a new Jev out, plus a few other models, and we can't keep up. You should ultimately try these on your own tasks."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Testing Jev and other decision models on search-related tasks"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://parallel.ai/blog/are-all-system-one-models-created-equal"}},"_tags":["comment","author_everythingmeta","story_50109686"],"author":"everythingmeta","comment_text":"One of the authors here. I work at Parallel. We got Jev-pilled a few weeks ago and have been trialing decision models nonstop across all sorts of use cases, so we wanted to see how the Jev-like models compare. The tasks are a mix: some where we previously used fine-tuned models, some where we used general LLMs, and some we hadn't pursued at all before Jev. What I think makes our analysis a little different is that we tested on human-labeled data from our production pipeline rather than synthetic benchmarks, which can be made to show almost whatever you want (I suspect that's how so many OSS models were ranking above Jev in early evals).
What we found:
The ergonomics of serverless decision models were really hard to beat. Cost, latency and accuracy were all near the top for Jev. Latency especially was very consistent, which matters a lot to us; historically that's where we'd have reached for a fine-tuned classifier.
Perplexity's decision model was a close second. It sometimes outperformed Jev, but its latency was more variable. That said, I believe it's also half the price now.
Self-hosting open-weight decision models wasn't a big unlock. In principle, at peak utilization we'd get into the ballpark of Jev's API performance. That said there is a world of optimizations that we just didn't have the time to pursue.
Iterating on prompts got meaningful improvements out of almost every model. That's much nicer UX than having to retrain a classifier.
Caveats: there's already a new Jev out, plus a few other models, and we can't keep up. You should ultimately try these on your own tasks.","created_at":"2026-10-15T20:21:16Z","created_at_i":1792095676,"objectID":"50109688","parent_id":50109686,"story_id":50109686,"story_title":"Testing Jev and other decision models on search-related tasks","story_url":"https://parallel.ai/blog/are-all-system-one-models-created-equal","updated_at":"2026-10-15T23:44:17Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"karamazov"},"story_text":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["cost","optimization"],"value":"If you're looking for a job, do -not- have Claude bulk email every company on "Who's Hiring".
These things all look the same and your message is just going to get marked spam.
For example, it's painfully obvious that our staff eng role's "combination of distributed execution, workload optimization, and cost-aware infrastructure" is not, in fact, "unusually close" to whatever line on your resume Claude decided is tangentially relevant.
Just write a normal email by hand and send it. If you it looks like you spent 5 minutes looking up what we do and are actually excited I'll at least look at your resume; if you spent 0 minutes thinking about the job posting I'm going to spend 0 minutes on this too."},"title":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["llm"],"value":"Tell HN: Don't mass spam employers with LLM generated emails"}},"_tags":["story","author_karamazov","story_50095323","ask_hn"],"author":"karamazov","children":[50112677,50095974,50142028,50113326],"created_at":"2026-10-14T21:09:24Z","created_at_i":1792012164,"num_comments":4,"objectID":"50095323","points":7,"story_id":50095323,"story_text":"If you're looking for a job, do -not- have Claude bulk email every company on "Who's Hiring".
These things all look the same and your message is just going to get marked spam.
For example, it's painfully obvious that our staff eng role's "combination of distributed execution, workload optimization, and cost-aware infrastructure" is not, in fact, "unusually close" to whatever line on your resume Claude decided is tangentially relevant.
Just write a normal email by hand and send it. If you it looks like you spent 5 minutes looking up what we do and are actually excited I'll at least look at your resume; if you spent 0 minutes thinking about the job posting I'm going to spend 0 minutes on this too.","title":"Tell HN: Don't mass spam employers with LLM generated emails","updated_at":"2026-10-18T15:41:25Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"toomuchtodo"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["llm","cost","optimization"],"value":"Yes, iphone, ipad, mac. Maybe your phone makes the inference requests to either Apple servers (requiring iCloud+ and credit based?) or your Mac (Tailscale equivalent).
My assumptions are that inference costs continue to decline (and/or the difficulty level of routing requests declines) and hardware acceleration continues into the future. These assumptions might be wrong, but that is the idea the thesis is based around.
(have built workflows and harnesses that currently target LLM provider APIs under the assumption I'll be able to point "locally-ish" eventually, my career had a strong focus on distributed computing about 10-15 years ago, think managing data and work clusters for LHC detector data processing, and I grew up on hobbyist distributed.net and BOINC projects, all the lenses I see this problem through; queues, jobs, sync vs async performance optimization, etc; recovered sysadmin|infra/devops engineer a lifetime ago)"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Character.ai's Bankruptcy Documents Show the Economics of AI Don't Work"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://www.404media.co/character-ais-bankruptcy-documents-show-the-economics-of-ai-dont-work/"}},"_tags":["comment","author_toomuchtodo","story_50089395"],"author":"toomuchtodo","children":[50090151],"comment_text":"Yes, iphone, ipad, mac. Maybe your phone makes the inference requests to either Apple servers (requiring iCloud+ and credit based?) or your Mac (Tailscale equivalent).
My assumptions are that inference costs continue to decline (and/or the difficulty level of routing requests declines) and hardware acceleration continues into the future. These assumptions might be wrong, but that is the idea the thesis is based around.
(have built workflows and harnesses that currently target LLM provider APIs under the assumption I'll be able to point "locally-ish" eventually, my career had a strong focus on distributed computing about 10-15 years ago, think managing data and work clusters for LHC detector data processing, and I grew up on hobbyist distributed.net and BOINC projects, all the lenses I see this problem through; queues, jobs, sync vs async performance optimization, etc; recovered sysadmin|infra/devops engineer a lifetime ago)","created_at":"2026-10-14T16:17:58Z","created_at_i":1791994678,"objectID":"50090111","parent_id":50089991,"story_id":50089395,"story_title":"Character.ai's Bankruptcy Documents Show the Economics of AI Don't Work","story_url":"https://www.404media.co/character-ais-bankruptcy-documents-show-the-economics-of-ai-dont-work/","updated_at":"2026-10-14T16:25:30Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"daishi55"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["llm","cost","optimization"],"value":"> Don't think the 0.2% really are in agreement that "large, complex projects that used to take them weeks/months can now be completed by agents with a prompt".
If you are at the top of field with unlimited access to the best (public) models then yes, you will agree that this is the case.
> The divide seems to be between people who think you may need to understand the codebase for maintenance, potential future token costs, optimisation, code erosion & being wary of the assumptions/mistakes that they have seen LLMs make, and people who don't review code and are confident that the AI has made no mistakes and will be available for next-to-nothing forever.
This is not the real divide, it\u2019s the divide you want to exist. The real divide is something along the lines of intellectual curiosity, capability, and confidence.
> if/when they break things
Idk man. I\u2019m one of those guys who \u201cdoesn\u2019t write code anymore\u201d since early this year. Nothing is breaking. If anything the chance of things breaking is shrinking dramatically, with both the enormous growth in tooling to support agentic software development (where I work, this mainly really good AI code reviews that catch so much more than human reviewers ever could), and the improvements in the models themselves.
Sorry, but the fantasy of all this coming crashing down is just never going to happen. This is just how it is now, and all the structures will continue to adapt to support it even better."},"story_title":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["llm"],"value":"The gap in shared understanding of LLM capability is widening"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://twitter.com/karpathy/status/2109361546505966046"}},"_tags":["comment","author_daishi55","story_50047422"],"author":"daishi55","children":[50051508],"comment_text":"> Don't think the 0.2% really are in agreement that "large, complex projects that used to take them weeks/months can now be completed by agents with a prompt".
If you are at the top of field with unlimited access to the best (public) models then yes, you will agree that this is the case.
> The divide seems to be between people who think you may need to understand the codebase for maintenance, potential future token costs, optimisation, code erosion & being wary of the assumptions/mistakes that they have seen LLMs make, and people who don't review code and are confident that the AI has made no mistakes and will be available for next-to-nothing forever.
This is not the real divide, it\u2019s the divide you want to exist. The real divide is something along the lines of intellectual curiosity, capability, and confidence.
> if/when they break things
Idk man. I\u2019m one of those guys who \u201cdoesn\u2019t write code anymore\u201d since early this year. Nothing is breaking. If anything the chance of things breaking is shrinking dramatically, with both the enormous growth in tooling to support agentic software development (where I work, this mainly really good AI code reviews that catch so much more than human reviewers ever could), and the improvements in the models themselves.
Sorry, but the fantasy of all this coming crashing down is just never going to happen. This is just how it is now, and all the structures will continue to adapt to support it even better.","created_at":"2026-10-12T01:53:28Z","created_at_i":1791770008,"objectID":"50050346","parent_id":50048729,"story_id":50047422,"story_title":"The gap in shared understanding of LLM capability is widening","story_url":"https://twitter.com/karpathy/status/2109361546505966046","updated_at":"2026-10-12T21:36:08Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"rico735"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["llm","cost","optimization"],"value":"Don't think the 0.2% really are in agreement that "large, complex projects that used to take them weeks/months can now be completed by agents with a prompt".
The divide seems to be between people who think you may need to understand the codebase for maintenance, potential future token costs, optimisation, code erosion & being wary of the assumptions/mistakes that they have seen LLMs make, and people who don't review code and are confident that the AI has made no mistakes and will be available for next-to-nothing forever.
It's certainly allowing the latter to move fast... but will their clients be happy if/when they break things. We'll find out soon, I'm sure."},"story_title":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["llm"],"value":"The gap in shared understanding of LLM capability is widening"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://twitter.com/karpathy/status/2109361546505966046"}},"_tags":["comment","author_rico735","story_50047422"],"author":"rico735","children":[50048990,50048982,50050070,50052768,50052269,50051677,50049417,50052061,50052434,50051114,50050842,50052249,50050346],"comment_text":"Don't think the 0.2% really are in agreement that "large, complex projects that used to take them weeks/months can now be completed by agents with a prompt".
The divide seems to be between people who think you may need to understand the codebase for maintenance, potential future token costs, optimisation, code erosion & being wary of the assumptions/mistakes that they have seen LLMs make, and people who don't review code and are confident that the AI has made no mistakes and will be available for next-to-nothing forever.
It's certainly allowing the latter to move fast... but will their clients be happy if/when they break things. We'll find out soon, I'm sure.","created_at":"2026-10-11T22:42:04Z","created_at_i":1791758524,"objectID":"50048729","parent_id":50047422,"story_id":50047422,"story_title":"The gap in shared understanding of LLM capability is widening","story_url":"https://twitter.com/karpathy/status/2109361546505966046","updated_at":"2026-10-16T08:14:04Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"flytomoon"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["llm","cost","optimization"],"value":"Whatever your opinion of the current state, LLMs are seemingly improving exponentially. You remember where we were a year ago. We can't fathom what it will be like in five years. Maybe even hard to imagine three. Once it hits the point of diminishing return on ability, it will move to speed and cost optimization. And of course everything will get smaller and take less energy in the standard progression. When viewing as a snapshot in infancy we can see the holes. But it's essentially a solved problem that is snowballing now. Not that it won't create other problems. And very likely solve many others."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Grieving the loss of details"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://purplesyringa.moe/blog/grieving-the-loss-of-details/"}},"_tags":["comment","author_flytomoon","story_49980880"],"author":"flytomoon","children":[50040135],"comment_text":"Whatever your opinion of the current state, LLMs are seemingly improving exponentially. You remember where we were a year ago. We can't fathom what it will be like in five years. Maybe even hard to imagine three. Once it hits the point of diminishing return on ability, it will move to speed and cost optimization. And of course everything will get smaller and take less energy in the standard progression. When viewing as a snapshot in infancy we can see the holes. But it's essentially a solved problem that is snowballing now. Not that it won't create other problems. And very likely solve many others.","created_at":"2026-10-11T01:06:29Z","created_at_i":1791680789,"objectID":"50038770","parent_id":49980880,"story_id":49980880,"story_title":"Grieving the loss of details","story_url":"https://purplesyringa.moe/blog/grieving-the-loss-of-details/","updated_at":"2026-10-11T16:22:49Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"divsh17"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["llm","cost","optimization"],"value":"Location: Bellevue, WA, USA\nRemote: Yes. Prefer in person
Willing to relocate: YES
Technologies: Python, TypeScript, FastAPI, Next.js, React, PostgreSQL, pgvector, Redis, Apache Kafka, AWS, GCP, PyTorch, LLMs, RAG, Prompt Engineering
R\u00e9sum\u00e9/CV: https://drive.google.com/file/d/1dhWtipC3WauUodrj3K0abG8rZn5...
Email: divyanshusharma17.work@gmail.com
Links: https://divyanshusharma.com | https://github.com/divyanshusharma1709
I am an AI/Full-stack Engineer with 3+ YoE focused on production AI pipelines, inference optimization, and high-precision RAG. Recently as a Founding Engineer at Cascade Intelligence (a16z speedrun), I re-architected a vector matching pipeline that tripled RFP match precision from 20% to 60%. I also built Elliot AI, a proactive companion utilizing hybrid RAG to guarantee <300ms context retrieval across 6+ months of history, and previously engineered an AI narration system for VR surgery syncing LLM inference with frame-accurate playback under 200ms latency.
My foundational infrastructure background includes scaling an async queue-based email system from 400K to 1.5M+ daily sends and driving $2.1M+ in annualized revenue impact through microservice A/B experimentation at Amazon. I am actively seeking senior or founding AI engineering roles to tackle massive system design challenges and build core agentic frameworks, and I require a STEM-OPT transfer (no sponsorship cost)."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Ask HN: Who wants to be hired? (October 2026)"}},"_tags":["comment","author_divsh17","story_49922568"],"author":"divsh17","comment_text":"Location: Bellevue, WA, USA\nRemote: Yes. Prefer in person
Willing to relocate: YES
Technologies: Python, TypeScript, FastAPI, Next.js, React, PostgreSQL, pgvector, Redis, Apache Kafka, AWS, GCP, PyTorch, LLMs, RAG, Prompt Engineering
R\u00e9sum\u00e9/CV: https://drive.google.com/file/d/1dhWtipC3WauUodrj3K0abG8rZn5...
Email: divyanshusharma17.work@gmail.com
Links: https://divyanshusharma.com | https://github.com/divyanshusharma1709
I am an AI/Full-stack Engineer with 3+ YoE focused on production AI pipelines, inference optimization, and high-precision RAG. Recently as a Founding Engineer at Cascade Intelligence (a16z speedrun), I re-architected a vector matching pipeline that tripled RFP match precision from 20% to 60%. I also built Elliot AI, a proactive companion utilizing hybrid RAG to guarantee <300ms context retrieval across 6+ months of history, and previously engineered an AI narration system for VR surgery syncing LLM inference with frame-accurate playback under 200ms latency.
My foundational infrastructure background includes scaling an async queue-based email system from 400K to 1.5M+ daily sends and driving $2.1M+ in annualized revenue impact through microservice A/B experimentation at Amazon. I am actively seeking senior or founding AI engineering roles to tackle massive system design challenges and build core agentic frameworks, and I require a STEM-OPT transfer (no sponsorship cost).","created_at":"2026-10-06T04:18:50Z","created_at_i":1791260330,"objectID":"49974120","parent_id":49922568,"story_id":49922568,"story_title":"Ask HN: Who wants to be hired? (October 2026)","updated_at":"2026-10-06T04:19:29Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"Karrot_Kream"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["llm","cost","optimization"],"value":"I mean what does "put you at a disadvantage" even mean concretely? To use a less popular language, it means you need to load up context related to the semantics of your language, load context on how to invoke tools to make sure the syntax with your language is correct, load up context related to each tool call you make (which will be more numerous in a niche language), and load up context on architectural decisions that might be specific to your language. All of this is simply a token cost. By forcing a model to load an initial amount of context per harness turn you also effectively shorten the max context window beyond which the model becomes stupid (which itself is much shorter than the max context length.)
Obviously it's not like people are specifically trimming each and every prompt they give a model to tokenmax their models to get the best output / input prompt, we instead live in a spectrum of how many tokens of input and context we're willing to provide to a model to make progress. If the cost of those tokens is low enough for the problem domain you're working in, then it's fine. For some the readability of a personal language may outstrip any of the token costs that one needs to pay to use it. Alternatively maybe you want something like an array language (J, K, APL, etc) which allows array programming and optimizations that conventional PLs just can't do. Maybe you want your language to compile to a target that is highly portable. There's actually a lot of stuff out there that previously wasn't feasible but with LLMs-as-force-multiplier absolutely is.
I also suspect the space is a continuum. There may be pareto optimal points, such as DSLs built atop languages, that are both highly readable but also fairly token efficient."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Zig v0.17.0"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://ziglang.org/download/0.17.0/release-notes.html"}},"_tags":["comment","author_Karrot_Kream","story_49938521"],"author":"Karrot_Kream","children":[49940813],"comment_text":"I mean what does "put you at a disadvantage" even mean concretely? To use a less popular language, it means you need to load up context related to the semantics of your language, load context on how to invoke tools to make sure the syntax with your language is correct, load up context related to each tool call you make (which will be more numerous in a niche language), and load up context on architectural decisions that might be specific to your language. All of this is simply a token cost. By forcing a model to load an initial amount of context per harness turn you also effectively shorten the max context window beyond which the model becomes stupid (which itself is much shorter than the max context length.)
Obviously it's not like people are specifically trimming each and every prompt they give a model to tokenmax their models to get the best output / input prompt, we instead live in a spectrum of how many tokens of input and context we're willing to provide to a model to make progress. If the cost of those tokens is low enough for the problem domain you're working in, then it's fine. For some the readability of a personal language may outstrip any of the token costs that one needs to pay to use it. Alternatively maybe you want something like an array language (J, K, APL, etc) which allows array programming and optimizations that conventional PLs just can't do. Maybe you want your language to compile to a target that is highly portable. There's actually a lot of stuff out there that previously wasn't feasible but with LLMs-as-force-multiplier absolutely is.
I also suspect the space is a continuum. There may be pareto optimal points, such as DSLs built atop languages, that are both highly readable but also fairly token efficient.","created_at":"2026-10-03T00:39:16Z","created_at_i":1790987956,"objectID":"49940308","parent_id":49940022,"story_id":49938521,"story_title":"Zig v0.17.0","story_url":"https://ziglang.org/download/0.17.0/release-notes.html","updated_at":"2026-10-03T02:20:04Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"twohaibei"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["llm","cost","optimization"],"value":"Location: Poland, Europe
Remote: Yes
Willing to relocate: No
Technologies: Javascript/typescript, react/react native (expo), node.js, ruby, elixir, AWS (lambda, s3, cf, bill optimization), Ruby on Rails, Next.js, Docker, and playing with LLMs extensively including building workflows.
R\u00e9sum\u00e9/CV: https://pavelloz.com/cv/
Email: me@pavelloz.com
I worked in startup for a long time, wearing lots of hats some in writing code, some in optimizing the process, infrastructure, cloud costs, sometimes writing docs and helping QA's fly with code, instead of manually clicking. I'm looking for a full stack role to deepen my backend, system desigd and hopefully AI integration skills further."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Ask HN: Who wants to be hired? (October 2026)"}},"_tags":["comment","author_twohaibei","story_49922568"],"author":"twohaibei","comment_text":"Location: Poland, Europe
Remote: Yes
Willing to relocate: No
Technologies: Javascript/typescript, react/react native (expo), node.js, ruby, elixir, AWS (lambda, s3, cf, bill optimization), Ruby on Rails, Next.js, Docker, and playing with LLMs extensively including building workflows.
R\u00e9sum\u00e9/CV: https://pavelloz.com/cv/
Email: me@pavelloz.com
I worked in startup for a long time, wearing lots of hats some in writing code, some in optimizing the process, infrastructure, cloud costs, sometimes writing docs and helping QA's fly with code, instead of manually clicking. I'm looking for a full stack role to deepen my backend, system desigd and hopefully AI integration skills further.","created_at":"2026-10-02T21:46:46Z","created_at_i":1790977606,"objectID":"49939010","parent_id":49922568,"story_id":49922568,"story_title":"Ask HN: Who wants to be hired? (October 2026)","updated_at":"2026-10-02T21:51:35Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"dannyw"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["llm","cost","optimization"],"value":"Serving LLMs at Anthropic scale is very, very different. It\u2019s not SGLang or vLLM.
If you\u2019ve tried setting either of these up, you\u2019ll know how various tricky settings can impact throughout and model output quality; and those are much simpler stacks.
Even homelabbers are getting into disaggregated compute; e.g. one GPU for prefill, another for decode.
Obviously Anthropic and co are using a mixture of GPUs and hardware and clusters; not everything is just GB300 or whatever; so you then get into hardware quirks, kernel optimisations that may deliver huge speedups at the cost of a tiny bit of KL divergence, etc.
And I believe they\u2019ve publicly said they use TPUs for inference too, but I doubt exclusively; and I\u2019m sure that\u2019s well optimised too.
Finally, Google has publicly stated they intentionally and silently degrade/poison models in response to distillation attacks; who knows what the other companies do."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Livenerf: Has Opus 5.5 been nerfed yet?"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://github.com/ninjahawk/livenerf"}},"_tags":["comment","author_dannyw","story_49901736"],"author":"dannyw","comment_text":"Serving LLMs at Anthropic scale is very, very different. It\u2019s not SGLang or vLLM.
If you\u2019ve tried setting either of these up, you\u2019ll know how various tricky settings can impact throughout and model output quality; and those are much simpler stacks.
Even homelabbers are getting into disaggregated compute; e.g. one GPU for prefill, another for decode.
Obviously Anthropic and co are using a mixture of GPUs and hardware and clusters; not everything is just GB300 or whatever; so you then get into hardware quirks, kernel optimisations that may deliver huge speedups at the cost of a tiny bit of KL divergence, etc.
And I believe they\u2019ve publicly said they use TPUs for inference too, but I doubt exclusively; and I\u2019m sure that\u2019s well optimised too.
Finally, Google has publicly stated they intentionally and silently degrade/poison models in response to distillation attacks; who knows what the other companies do.","created_at":"2026-09-30T07:39:03Z","created_at_i":1790753943,"objectID":"49905687","parent_id":49905323,"story_id":49901736,"story_title":"Livenerf: Has Opus 5.5 been nerfed yet?","story_url":"https://github.com/ninjahawk/livenerf","updated_at":"2026-10-01T21:47:31Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"weiqi554177834"},"story_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["llm","cost","optimization"],"value":"I've been following the Anthropic supply-chain ruling discussion here, and one thing keeps nagging at me as someone learning to build on Next.js while studying LLM app architecture: multi-model redundancy quietly stopped being an optimization and became a compliance requirement for anyone selling to the US government or large enterprises.
A single-model vendor is now officially a supply-chain risk. That makes sense at scale, but what does it actually mean for a small team? Abstracting calls behind one interface sounds easy until models diverge on tool calling, structured outputs, context windows, and latency. And who maintains the eval suite and fallback logic once you support two or three providers?
For those of you running LLM features in production: do you genuinely route across providers, or keep one hot standby? Is an abstraction layer worth it early, or does it just flatten every model to the weakest one? Curious what patterns actually hold up when vendor lock-in becomes a legal problem rather than just a cost one."},"title":{"matchLevel":"none","matchedWords":[],"value":"Ask HN: Is multi-model redundancy now a compliance requirement for small teams?"}},"_tags":["story","author_weiqi554177834","story_49852847","ask_hn"],"author":"weiqi554177834","children":[49917741,49859712,49853048],"created_at":"2026-09-26T03:12:00Z","created_at_i":1790392320,"num_comments":1,"objectID":"49852847","points":2,"story_id":49852847,"story_text":"I've been following the Anthropic supply-chain ruling discussion here, and one thing keeps nagging at me as someone learning to build on Next.js while studying LLM app architecture: multi-model redundancy quietly stopped being an optimization and became a compliance requirement for anyone selling to the US government or large enterprises.
A single-model vendor is now officially a supply-chain risk. That makes sense at scale, but what does it actually mean for a small team? Abstracting calls behind one interface sounds easy until models diverge on tool calling, structured outputs, context windows, and latency. And who maintains the eval suite and fallback logic once you support two or three providers?
For those of you running LLM features in production: do you genuinely route across providers, or keep one hot standby? Is an abstraction layer worth it early, or does it just flatten every model to the weakest one? Curious what patterns actually hold up when vendor lock-in becomes a legal problem rather than just a cost one.","title":"Ask HN: Is multi-model redundancy now a compliance requirement for small teams?","updated_at":"2026-10-07T22:20:22Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"aDyslecticCrow"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["llm","cost","optimization"],"value":"> Ergonomics, cost profile, and ease of use are new.
Following AI from the academic papers side; jev really feels silly. They one-pass the LLM tranformer stack and tune the output network for a probability value.
(some clever pararellization optimisations to make it viable to offer as an api, since the normal kv cashing no longer works if you oneshot the tranformer)
The largest change is the packaging; An api with a tolken based pricing, and a schema to define the output structure for quick setup.
Previous projects would probably involve installing pytorch, running a converter script on Qwen, and write a fair bit of matrix math to change the output shape.
I'm kinda amused that it took this long though."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"OpenAI is well positioned to fast-follow Jev"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://arcturus-labs.com/blog/2026/09/21/will-openai-eat-jevs-lunch/"}},"_tags":["comment","author_aDyslecticCrow","story_49802161"],"author":"aDyslecticCrow","comment_text":"> Ergonomics, cost profile, and ease of use are new.
Following AI from the academic papers side; jev really feels silly. They one-pass the LLM tranformer stack and tune the output network for a probability value.
(some clever pararellization optimisations to make it viable to offer as an api, since the normal kv cashing no longer works if you oneshot the tranformer)
The largest change is the packaging; An api with a tolken based pricing, and a schema to define the output structure for quick setup.
Previous projects would probably involve installing pytorch, running a converter script on Qwen, and write a fair bit of matrix math to change the output shape.
I'm kinda amused that it took this long though.","created_at":"2026-09-22T23:33:42Z","created_at_i":1790120022,"objectID":"49809717","parent_id":49808074,"story_id":49802161,"story_title":"OpenAI is well positioned to fast-follow Jev","story_url":"https://arcturus-labs.com/blog/2026/09/21/will-openai-eat-jevs-lunch/","updated_at":"2026-09-24T11:32:37Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"louthy"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["llm","cost","optimization"],"value":"> Even in some of your examples like a compiler, game engine, or collection library, the big oh matters
Sure, but for example - today - I am literally building and optimising high-performance collections for my open-source library. Big-O is irrelevant, because I have built pretty much all of the fundamental collection types, what I care about is lower than that: what happens when enumerating any one of those collection types. Big-O tells you the scale of the problem, but not the per-element cost, which is still important when you're building core data-structures.
I am concerned about cache-friendly memory-layouts, how to do collection compositions without unnecessary memory allocations, keeping enough guards in place to make the types safe whilst removing as many branches as possible, reducing memory copying as much as possible, and catching stupid shit the compiler or JIT does and try to work around it. Literally what happens per-instruction, per-iteration, not how to pick a big-O based data-structure: trying to make all data-structures as fast as possible.
Anyway, we seem to be talking past each other. You're talking about the basics, I'm talking about the original source of this thread which was that (apparently) LLMs are surprisingly bad at optimisation. Which, I am trying to highlight becomes almost voodoo at a low-enough level and that highly-optimised code looks progressively more strange and opaque (in the hunt for a few nanoseconds here and there), which for an LLM wouldn't look statistically significant. I think the basics of data-structure choice should be easily within the realms of an LLM's current capabilities."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Writing Rust code that's fast by asking agents to make the code faster"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://minimaxir.com/2026/09/agentic-iteration/"}},"_tags":["comment","author_louthy","story_49803085"],"author":"louthy","children":[49807716],"comment_text":"> Even in some of your examples like a compiler, game engine, or collection library, the big oh matters
Sure, but for example - today - I am literally building and optimising high-performance collections for my open-source library. Big-O is irrelevant, because I have built pretty much all of the fundamental collection types, what I care about is lower than that: what happens when enumerating any one of those collection types. Big-O tells you the scale of the problem, but not the per-element cost, which is still important when you're building core data-structures.
I am concerned about cache-friendly memory-layouts, how to do collection compositions without unnecessary memory allocations, keeping enough guards in place to make the types safe whilst removing as many branches as possible, reducing memory copying as much as possible, and catching stupid shit the compiler or JIT does and try to work around it. Literally what happens per-instruction, per-iteration, not how to pick a big-O based data-structure: trying to make all data-structures as fast as possible.
Anyway, we seem to be talking past each other. You're talking about the basics, I'm talking about the original source of this thread which was that (apparently) LLMs are surprisingly bad at optimisation. Which, I am trying to highlight becomes almost voodoo at a low-enough level and that highly-optimised code looks progressively more strange and opaque (in the hunt for a few nanoseconds here and there), which for an LLM wouldn't look statistically significant. I think the basics of data-structure choice should be easily within the realms of an LLM's current capabilities.","created_at":"2026-09-22T20:29:05Z","created_at_i":1790108945,"objectID":"49807624","parent_id":49807390,"story_id":49803085,"story_title":"Writing Rust code that's fast by asking agents to make the code faster","story_url":"https://minimaxir.com/2026/09/agentic-iteration/","updated_at":"2026-09-22T20:50:48Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"dhx"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["llm","cost","optimization"],"value":"100% agreed
It'd be great to see a description of even just a subset of training datasets. It feels very much under-reported how much expense is worth investing in preparing and selecting training datasets versus just using masses of random quality unprepared training data. This dashboard appears to be good though in showing the limits quickly reached when throwing parameters and compute at the problem.
For example, if they were to train on Wikipedia dumps, do they consider every article to be the same quality across each language, or have they done more work beyond Wikipedia's own article quality ratings to make training decisions such as "Ignore cebwiki it's machine-generated spam" and "Treat dewiki articles with coordinates within Germany as being higher quality (weight it higher) than their equivalent enwiki articles".
And let's say one of the datasets is all the source code of packages in the Gentoo package repository. Not every software package is a good example of how to write code. You perhaps wouldn't want to train your LLM on 1990s era PHP web application source code as an example of how to write code in 2026. Instead, you'd possibly want to use such PHP web application source code as a negative training example of what _not_ to write. But when training an LLM to detect software bugs, maybe outdated PHP source code is good for training.
Similarly for translation, perhaps UN treaty documents translated into 4+ languages are good translation examples because of high accuracy needed, professional translators being used, and bigger budgets. However this training data would perhaps be a negative training example towards translating chat messages, movie subtitles, etc because it doesn't use everyday slang and could result in output of nonsense such as "Pending Your Excellency's response, please accept, Your Excellency, my sincere greetings." for a prompt asking to write a birthday card for a child.
Preparing training data and deciding how to best use it for training I assume would be the largest expense (cost of labour -- mostly expert labour too) and also the greatest opportunity in the future for LLMs to improve. It seems to me somewhat irrelevant if the dashboard indicates a compute expense of $1m or $5m if good training datasets (prepared by experts in their fields) cost $10m/y to maintain. For example, hiring expert software developers to tag 1000's of open source software packages according to their quality, on different metrics, such as human readability, performance optimisation with choice of algorithms, reasonable trade-off between coherence and coupling in the software architecture, currency with state of the art programming trends/preferred dependencies/operating system APIs, etc. And keeping that metadata continually updated rather than a rapidly obsolete once off tagging project completed in 2005."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"MiMo v2.6"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://mimo.xiaomi.com/mimo-v2-6"}},"_tags":["comment","author_dhx","story_49792730"],"author":"dhx","comment_text":"100% agreed
It'd be great to see a description of even just a subset of training datasets. It feels very much under-reported how much expense is worth investing in preparing and selecting training datasets versus just using masses of random quality unprepared training data. This dashboard appears to be good though in showing the limits quickly reached when throwing parameters and compute at the problem.
For example, if they were to train on Wikipedia dumps, do they consider every article to be the same quality across each language, or have they done more work beyond Wikipedia's own article quality ratings to make training decisions such as "Ignore cebwiki it's machine-generated spam" and "Treat dewiki articles with coordinates within Germany as being higher quality (weight it higher) than their equivalent enwiki articles".
And let's say one of the datasets is all the source code of packages in the Gentoo package repository. Not every software package is a good example of how to write code. You perhaps wouldn't want to train your LLM on 1990s era PHP web application source code as an example of how to write code in 2026. Instead, you'd possibly want to use such PHP web application source code as a negative training example of what _not_ to write. But when training an LLM to detect software bugs, maybe outdated PHP source code is good for training.
Similarly for translation, perhaps UN treaty documents translated into 4+ languages are good translation examples because of high accuracy needed, professional translators being used, and bigger budgets. However this training data would perhaps be a negative training example towards translating chat messages, movie subtitles, etc because it doesn't use everyday slang and could result in output of nonsense such as "Pending Your Excellency's response, please accept, Your Excellency, my sincere greetings." for a prompt asking to write a birthday card for a child.
Preparing training data and deciding how to best use it for training I assume would be the largest expense (cost of labour -- mostly expert labour too) and also the greatest opportunity in the future for LLMs to improve. It seems to me somewhat irrelevant if the dashboard indicates a compute expense of $1m or $5m if good training datasets (prepared by experts in their fields) cost $10m/y to maintain. For example, hiring expert software developers to tag 1000's of open source software packages according to their quality, on different metrics, such as human readability, performance optimisation with choice of algorithms, reasonable trade-off between coherence and coupling in the software architecture, currency with state of the art programming trends/preferred dependencies/operating system APIs, etc. And keeping that metadata continually updated rather than a rapidly obsolete once off tagging project completed in 2005.","created_at":"2026-09-22T04:59:41Z","created_at_i":1790053181,"objectID":"49796955","parent_id":49793753,"story_id":49792730,"story_title":"MiMo v2.6","story_url":"https://mimo.xiaomi.com/mimo-v2-6","updated_at":"2026-09-26T20:32:14Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"adrianN"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["llm","cost","optimization"],"value":"The argument is probably that LLMs can find those optimizations cheaper than a human expert. Since LLM cost at fixed capability seems to be going down you either expect humans to be completely replaced or human wages to be lowered by LLMs."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Saving another 100TB of RAM"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://blog.cloudflare.com/saving-100-tb-of-ram-with-math/"}},"_tags":["comment","author_adrianN","story_49758580"],"author":"adrianN","children":[49763930],"comment_text":"The argument is probably that LLMs can find those optimizations cheaper than a human expert. Since LLM cost at fixed capability seems to be going down you either expect humans to be completely replaced or human wages to be lowered by LLMs.","created_at":"2026-09-19T02:45:52Z","created_at_i":1789785952,"objectID":"49762810","parent_id":49762403,"story_id":49758580,"story_title":"Saving another 100TB of RAM","story_url":"https://blog.cloudflare.com/saving-100-tb-of-ram-with-math/","updated_at":"2026-09-20T04:44:53Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"rao-v"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["llm","cost","optimization"],"value":"Hey Victor! Been following you since HVM/Kind, partly because I'm moderately unhappy with the state of out of the box automatic parallelism in modern languages!
Do you plan to invest in profile guided optimization or autotuning in Bend2 - using runtime profiles / cost models to make decisions around SIMD vs. multicore vs. GPU parallelization?
Bend2's model might give you a really nice view into available parallelization. Heck I can imagine integrating an LLM to profile and optimize in an absurdly expensive `-O7` optimization mode one day!"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Bend \u2013 a language that blocks AI mistakes via proof and runs on GPUs"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://bend-lang.com/"}},"_tags":["comment","author_rao-v","story_49746163"],"author":"rao-v","children":[49747412],"comment_text":"Hey Victor! Been following you since HVM/Kind, partly because I'm moderately unhappy with the state of out of the box automatic parallelism in modern languages!
Do you plan to invest in profile guided optimization or autotuning in Bend2 - using runtime profiles / cost models to make decisions around SIMD vs. multicore vs. GPU parallelization?
Bend2's model might give you a really nice view into available parallelization. Heck I can imagine integrating an LLM to profile and optimize in an absurdly expensive `-O7` optimization mode one day!","created_at":"2026-09-17T21:53:16Z","created_at_i":1789681996,"objectID":"49747126","parent_id":49746605,"story_id":49746163,"story_title":"Bend \u2013 a language that blocks AI mistakes via proof and runs on GPUs","story_url":"https://bend-lang.com/","updated_at":"2026-09-20T07:54:08Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"janwas"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["llm","cost","optimization"],"value":"(Co-)author here :)
I'd absolutely still use Highway, and do. My experience is that even two separate implementations diverge over time and I'd have low confidence in bringing the same updates and improvements to all, even with LLM assistance.
Our programming model is 1) an agent+human to generate the algorithmic approach, 2) a C++ library (Highway) to translate to intrinsics, while filling in gaps + allowing customization, 3) a compiler to generate the actual code with some optimizations.
Asking the compiler to do #1 is a pipe dream: compiler friends tell me they are not going to devise new shuffles/data layouts (like what VQSort does). Conflating #2 and #3 means a custom compiler/IR which has high engineering costs (ABI boundaries, hard to debug/profile/sanitize).\nAnd doing #3 at runtime (JIT), or moving fusions into #3 (MLIR), vastly complicates the compiler. We can still get runtime adaptability thanks to Highway's multi-target support. Fusion has been much easier to implement manually for LLMs than to construct a general fusion infrastructure. Templates hide most data type differences and we see 2-5x speedup vs llama.cpp for 128k prefill+batch decode on Zen5.
Instead of requiring a compiler to do heroic transforms at runtime, and get it right every time, we can do all kinds of agentic exploration, then verify the result/approach, check in the source code, then we 'just' have a C++ compiler afterwards. And if/when something breaks, it's easy to update centrally, in code we can modify directly, rather than indirectly via updating a compiler."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Vectorized and performance-portable Quicksort (2022)"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://opensource.googleblog.com/2022/06/Vectorized%20and%20performance%20portable%20Quicksort.html"}},"_tags":["comment","author_janwas","story_49731054"],"author":"janwas","comment_text":"(Co-)author here :)
I'd absolutely still use Highway, and do. My experience is that even two separate implementations diverge over time and I'd have low confidence in bringing the same updates and improvements to all, even with LLM assistance.
Our programming model is 1) an agent+human to generate the algorithmic approach, 2) a C++ library (Highway) to translate to intrinsics, while filling in gaps + allowing customization, 3) a compiler to generate the actual code with some optimizations.
Asking the compiler to do #1 is a pipe dream: compiler friends tell me they are not going to devise new shuffles/data layouts (like what VQSort does). Conflating #2 and #3 means a custom compiler/IR which has high engineering costs (ABI boundaries, hard to debug/profile/sanitize).\nAnd doing #3 at runtime (JIT), or moving fusions into #3 (MLIR), vastly complicates the compiler. We can still get runtime adaptability thanks to Highway's multi-target support. Fusion has been much easier to implement manually for LLMs than to construct a general fusion infrastructure. Templates hide most data type differences and we see 2-5x speedup vs llama.cpp for 128k prefill+batch decode on Zen5.
Instead of requiring a compiler to do heroic transforms at runtime, and get it right every time, we can do all kinds of agentic exploration, then verify the result/approach, check in the source code, then we 'just' have a C++ compiler afterwards. And if/when something breaks, it's easy to update centrally, in code we can modify directly, rather than indirectly via updating a compiler.","created_at":"2026-09-17T09:06:27Z","created_at_i":1789635987,"objectID":"49738185","parent_id":49731587,"story_id":49731054,"story_title":"Vectorized and performance-portable Quicksort (2022)","story_url":"https://opensource.googleblog.com/2022/06/Vectorized%20and%20performance%20portable%20Quicksort.html","updated_at":"2026-09-17T12:18:16Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"camdenreslink"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["llm","cost","optimization"],"value":"You can imagine with more operations being available to be done more cheaply and quickly the LLM doesn't need to "one shot" a solution. It could try many solutions, test them, throw some away, wiggle some of the parameters like a genetic algorithm, see how that changes the result, and converge on an optimal solution (based on whatever the cost function is). Basically producing a good result could become like an optimization problem. That would be way too expensive and slow right now."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"OpenAI begins rolling out GPT-6 Astra"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://www.cnbc.com/2026/09/03/open-ai-astra-gpt-6-cyber.html"}},"_tags":["comment","author_camdenreslink","story_49554273"],"author":"camdenreslink","comment_text":"You can imagine with more operations being available to be done more cheaply and quickly the LLM doesn't need to "one shot" a solution. It could try many solutions, test them, throw some away, wiggle some of the parameters like a genetic algorithm, see how that changes the result, and converge on an optimal solution (based on whatever the cost function is). Basically producing a good result could become like an optimization problem. That would be way too expensive and slow right now.","created_at":"2026-09-03T19:35:01Z","created_at_i":1788464101,"objectID":"49555521","parent_id":49555128,"story_id":49554273,"story_title":"OpenAI begins rolling out GPT-6 Astra","story_url":"https://www.cnbc.com/2026/09/03/open-ai-astra-gpt-6-cyber.html","updated_at":"2026-09-03T19:45:36Z"}],"hitsPerPage":20,"nbHits":360,"nbPages":18,"page":0,"params":"query=LLM+cost+optimization&advancedSyntax=true&analyticsTags=backend","processingTimeMS":17,"processingTimingsMS":{"_request":{"queue":120,"roundTrip":14},"afterFetch":{"format":{"highlighting":1,"total":2}},"fetch":{"query":8,"scanning":7,"total":16},"total":17},"query":"LLM cost optimization","serverTimeMS":140}