We created this handbook to make LLM inference concepts more accessible, especially for developers building real-world LLM applications. The goal is to pull together scattered knowledge into something clear, practical, and easy to build on.
We\u2019re continuing to improve it, so feedback is very welcome!
GitHub repo: https://github.com/bentoml/llm-inference-in-production"},"story_title":{"fullyHighlighted":true,"matchLevel":"full","matchedWords":["llm","inference","handbook"],"value":"LLM Inference Handbook"},"story_url":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["llm"],"value":"https://bentoml.com/llm/"}},"_tags":["comment","author_sherlockxu","story_44527947"],"author":"sherlockxu","children":[44530135,44535657,44536421,44537248],"comment_text":"Hi everyone. I'm one of the maintainers of this project. We're both excited and humbled to see it on Hacker News!
We created this handbook to make LLM inference concepts more accessible, especially for developers building real-world LLM applications. The goal is to pull together scattered knowledge into something clear, practical, and easy to build on.
We\u2019re continuing to improve it, so feedback is very welcome!
GitHub repo: https://github.com/bentoml/llm-inference-in-production","created_at":"2025-07-11T08:00:11Z","created_at_i":1752220811,"objectID":"44529483","parent_id":44527947,"story_id":44527947,"story_title":"LLM Inference Handbook","story_url":"https://bentoml.com/llm/","updated_at":"2025-07-29T14:41:07Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"gmerc"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["llm","inference","handbook"],"value":"Objectively the trajectory is what matters not todays capabilities. And the trajectory tells its own story.
As with any technology there is both hype and not hype, there is no boolean answer to this question.
Is there hype ? absolutely\u2026 Langchain is not a 100M technology and chatgpt plug-ins are an idea at this point that may take time to find its final form. But this is par for the course for Silicon valley ever since investment became a last-to-invest holds the bag game so perfected by A16z.
Bing and search are at best a mediocre application of LLMs, much better executed with embedding storage and supporting technology rather than static models - the choice of GTM in this regard is classic Silicon Valley gaming and distracts from the underlying mechanics and broad potential of the technology.
But compared to crypto, web3 and metaverse, today, with our own eyes we can see:
- Computers could not draw/paint 2 years ago now they do at the level of a master artist.
- 9 month ago we diffused a 512x512 picture with barely any control in 6 seconds on top end hardware, today any consumer rig with a nvidia gaming GPU can create almost unlimited size images and top end cards diffuse at up to 30-fps. We have full control over the composition with controlnet too.
- Computers could not draw hands six month ago, now they do
- Computers could not code 2 years ago, a year ago barely autocompleted a function and now they certainly can do smaller programs and assist engineers in solving complex problems
- Computers could not pass any tests 2 years ago, now they beat the US bar exam and various CS entry exams
- computer vision could could objects and classify a few dozen object classes a few years ago now it can tell timmy is sad because his toy is broken in an image and basically segment anything
- Computers could not compose 2 years ago and now you could do it on your home computer
- A year ago you needed an A100 to run an LLM now it runs on a fucking raspberry pi quantized and most stock macbooks and you can download some on an iphone.
- LLM hallucinations and failure to do math seemed a major issue 8 month ago, now there\u2019s paths with embedding storage that allows for verified citations and they can invoke tools like wolfram alpha or create code to solve equations.
- 2 Years ago siri and co still failed at even the slightest foreign accent, today whisper almost perfectly transcribes heavily accented or foreign language text at a rate of hours in single digit minutes or even seconds on high end hardware
- 2 years ago text to speech was still a robotic affair, now it can clone a human voice including inflection, emotion, speech impediment, etc.
- inference capability is up 30x on hardware and many times on software in a year. Llama went from A100 to raspberry pi in 2 months after release.
- We see the ability for LLMs to reason complex topics and act as decision makers, approaching on many human jobs that are rigidly encased in policies and employee handbooks and rarely require or allow human judgement.
Those of us old enough remember the same dynamics with the .com bust and the breathless declarations that the internet was a fad in response to people being unable to contain their FOMO investing.
for all the hype, there is an iceberg of unrealized change underneath the mountain that is the last 12 months of change that\u2019s severely under appreciated and misunderstood.
Wall street barely glimpsed it a few weeks ago when it sent Nvidia to 1T. In the hands of experts over the next months, it is likely this technology will start having a terrible impact on the job market, IBMs virtue signaling a few weeks ago should make that clear to anyone.
It baffles that people keep retreating on either on the current flaws or on ideological positions predicated on shifting definitions or picking on narrow aspects of these systems rather than the whole. Anyone of sound mind should be humbled by these developments and very carefully phrase their predictions of the future.
I\u2019m sure people were dismissive of the first automobile or the internal combustion engine on the account of it not moving on its own too. I don\u2019t remember anyone opiniating \u201cit\u2019s just gradient descent\u201d when ad delivery birthed much of big tech and was accused of throwing elections and destroying social cohesion globally.
What is also missing from this debate is the devastating realization that this is the first major technological leap that happened a almost completely unbounded from infrastructure requirements. No power lines to be strung, no cell towers to be built no app stores needed to propagate. Everyone withdrawing on \u2018we always found new jobs in response to technology affecting old ones\u2019 may be right, but may be missing this factor.
This technology scales at a 10 minute API integration only bounded by Nvidias ability to produce GPUs. Only a small minority of people have engaged with it yet, but it will find applications in every job, every company over time. And it will never get worse, never slow down, never sleep.
No the only reasonable course of action here is to be humble, it looks very likely that just like with the .com boom and bust the technology will end up being life changing (something Silicon Valley barely remembers after a decade of bullshit pumping)"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"The AI Hype Wall of Shame"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://criticalai.org/the-ai-hype-wall-of-shame/"}},"_tags":["comment","author_gmerc","story_36280603"],"author":"gmerc","children":[36284449],"comment_text":"Objectively the trajectory is what matters not todays capabilities. And the trajectory tells its own story.
As with any technology there is both hype and not hype, there is no boolean answer to this question.
Is there hype ? absolutely\u2026 Langchain is not a 100M technology and chatgpt plug-ins are an idea at this point that may take time to find its final form. But this is par for the course for Silicon valley ever since investment became a last-to-invest holds the bag game so perfected by A16z.
Bing and search are at best a mediocre application of LLMs, much better executed with embedding storage and supporting technology rather than static models - the choice of GTM in this regard is classic Silicon Valley gaming and distracts from the underlying mechanics and broad potential of the technology.
But compared to crypto, web3 and metaverse, today, with our own eyes we can see:
- Computers could not draw/paint 2 years ago now they do at the level of a master artist.
- 9 month ago we diffused a 512x512 picture with barely any control in 6 seconds on top end hardware, today any consumer rig with a nvidia gaming GPU can create almost unlimited size images and top end cards diffuse at up to 30-fps. We have full control over the composition with controlnet too.
- Computers could not draw hands six month ago, now they do
- Computers could not code 2 years ago, a year ago barely autocompleted a function and now they certainly can do smaller programs and assist engineers in solving complex problems
- Computers could not pass any tests 2 years ago, now they beat the US bar exam and various CS entry exams
- computer vision could could objects and classify a few dozen object classes a few years ago now it can tell timmy is sad because his toy is broken in an image and basically segment anything
- Computers could not compose 2 years ago and now you could do it on your home computer
- A year ago you needed an A100 to run an LLM now it runs on a fucking raspberry pi quantized and most stock macbooks and you can download some on an iphone.
- LLM hallucinations and failure to do math seemed a major issue 8 month ago, now there\u2019s paths with embedding storage that allows for verified citations and they can invoke tools like wolfram alpha or create code to solve equations.
- 2 Years ago siri and co still failed at even the slightest foreign accent, today whisper almost perfectly transcribes heavily accented or foreign language text at a rate of hours in single digit minutes or even seconds on high end hardware
- 2 years ago text to speech was still a robotic affair, now it can clone a human voice including inflection, emotion, speech impediment, etc.
- inference capability is up 30x on hardware and many times on software in a year. Llama went from A100 to raspberry pi in 2 months after release.
- We see the ability for LLMs to reason complex topics and act as decision makers, approaching on many human jobs that are rigidly encased in policies and employee handbooks and rarely require or allow human judgement.
Those of us old enough remember the same dynamics with the .com bust and the breathless declarations that the internet was a fad in response to people being unable to contain their FOMO investing.
for all the hype, there is an iceberg of unrealized change underneath the mountain that is the last 12 months of change that\u2019s severely under appreciated and misunderstood.
Wall street barely glimpsed it a few weeks ago when it sent Nvidia to 1T. In the hands of experts over the next months, it is likely this technology will start having a terrible impact on the job market, IBMs virtue signaling a few weeks ago should make that clear to anyone.
It baffles that people keep retreating on either on the current flaws or on ideological positions predicated on shifting definitions or picking on narrow aspects of these systems rather than the whole. Anyone of sound mind should be humbled by these developments and very carefully phrase their predictions of the future.
I\u2019m sure people were dismissive of the first automobile or the internal combustion engine on the account of it not moving on its own too. I don\u2019t remember anyone opiniating \u201cit\u2019s just gradient descent\u201d when ad delivery birthed much of big tech and was accused of throwing elections and destroying social cohesion globally.
What is also missing from this debate is the devastating realization that this is the first major technological leap that happened a almost completely unbounded from infrastructure requirements. No power lines to be strung, no cell towers to be built no app stores needed to propagate. Everyone withdrawing on \u2018we always found new jobs in response to technology affecting old ones\u2019 may be right, but may be missing this factor.
This technology scales at a 10 minute API integration only bounded by Nvidias ability to produce GPUs. Only a small minority of people have engaged with it yet, but it will find applications in every job, every company over time. And it will never get worse, never slow down, never sleep.
No the only reasonable course of action here is to be humble, it looks very likely that just like with the .com boom and bust the technology will end up being life changing (something Silicon Valley barely remembers after a decade of bullshit pumping)","created_at":"2023-06-11T13:29:33Z","created_at_i":1686490173,"objectID":"36281191","parent_id":36280603,"story_id":36280603,"story_title":"The AI Hype Wall of Shame","story_url":"https://criticalai.org/the-ai-hype-wall-of-shame/","updated_at":"2026-03-22T15:48:17Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"tpmoney"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["llm","inference","handbook"],"value":"> An image that is lossily compressed, converted to a bitstream, and encoded in DNA is very very different than the input, but if an image can be recovered that is indistinguishable or barely distinguishable from the original, I'd still call that copying and each intermediate step a significant but irrelevant transformation.
Sure as a broad rule of thumb that works. But the ability of a machine to produce a copyright violation doesn't mean the machine itself or distributing the machine is a copyright violation. To take an extreme example, if we take a room full infinite monkeys and put them on infinite typewriters and they generate a Harry Potter book, that doesn't mean Harry Potter is stored in the monkey room. If we have a random sound generator that produces random tones from the standard western musical note pallet and it generates the bass line from "Under Pressure" that doesn't mean our random sound generator contains or is a copy of "Under Pressure", even if we encoded all the same information and procedures for generating those individual notes at those durations among the data procedures we gave the machine.
> If an LLM is completely incapable of reproducing input text verbatim, yet could become so through targeted ablation (that does not itself incorporate the text in question!), then does it store that text or not?
I would argue not. Just like a xerox machine doesn't contain the books you make copies of when you use it to make a copy, and Handbrake doesn't contain the DVD's you use when you make a copy there.
I would further argue that copyright infringement is inherently a "human" act. It's sort of encoded in the language we use to talk about it (e.g. "fair use") but it's also something of a "if a tree falls in the middle of the woods" situation. If an LLM runs in an isolated room in an isolated bunker with no one around and generates verbatim copies of the Linux kernel, that frankly doesn't matter. On the other hand, if a Microsoft employee induces an LLM to produce verbatim copies of the Linux kernel, that does, especially if they did so with the intent to incorporate Linux kernel code into Windows. Not because of the LLM, but because a person made the choice to produce a copy of something they didn't have the right to make a copy of. The method by which they accomplished that copy is less relevant than making the copy at all, and that in turn is less relevant than the intent of making that copy for a purpose which is not allowed by copyright law.
> I'm not sure why I'm even debating this, other than for intellectual curiosity.
Frankly, that's the only reason to debate anything. 99% of the time, you as an individual will never have the power to influence the actual legal decisions made. But a intellectually curious conversation is infinitely more useful, not just to you and me but to other readers, than another retread of "AI is slop" "you're just jealous you can't code your way out of a paper bag" arguments that pervade so much discussion around AI. Or worse yet another "I used an LLM for a clearly stupid thing and it was stupid" or "I used an LLM to replace all my employees and I'm sure it's going to go great" blog post. For whatever acrimony there might have been in our interchange here, I'm sorry, because this sort of discussion is the only good way to exercise our thoughts on an issue and really test them out ourselves. It's easy to have a knee jerk opinion. It's harder to support that opinion with a philosophy and reasoning.
For what it's worth, I view the LLM/AI world as the best opportunity we've had in decades to really rethink and scale back/change how we deal with intellectual property. The ever expanding copyright terms, the sometimes bizarre protections of what seem to be blindingly obvious ideas. The technological age has demonstrated a number of weaknesses in the traditional systems and views. And frankly I think it's also demonstrated that many prior predictions of certain doom if copyright wasn't strictly enforced have been overwrought and even where they haven't, the actual result has been better for more people. Famously, IBM would have very much preferred to have won the BIOS copyright issue. But I think so much of the modern computer and tech industry owes their very careers to the effects of that decision. It might have been better for IBM if IBM had won, it's not clear at all that it would have been better for "[promoting] the Progress of Science and useful Arts".
We could live in a world where we recognize that LLMs and AIs are going to fundamentally change how we approach creative works. We could recognize that the intents of "[promoting] the Progress of Science and useful Arts" is still a relevant goal and something we can work to make compatible with the existence of LLMs and AI. To pitch my crazy idea again, we could:
1) Cut the terms of copyright substantially, back down to 10 or 15 years by default.
2) Offer a single extension that doubles that term, but only on the condition that the work is submitted to a central "library of congress" data set.
3) This could be used to produce known good and clean data sets for AI companies and organization to train models from, with the protection that any model trained from this data set is protected from copyright infringement claims for works in the data set. Heck we could even produce common models. This would save massive amounts of power and resources by cutting the need for everyone who wants to be in the AI space to go out and acquire, digitize and build their own library. The NIST numbers set is effectively the "hello world" set for anyone learning computer vision AI stuff. Let's do that for all sort of AI.
4) The data sets and models would be provided for a nominal fee, this fee will be used to pay royalties to people whose works are still under copyright and are in the data sets, proportional to the recency and quantity of work submitted. A cap would need to be put in place to prevent flooding the data set to game the royalties. These royalties would be part of recognizing the value the original works contributed to the data set, and act as a further incentive to contribute works to the system and contribute them sooner.
We could build a system like this, or tweak it, or even build something else entirely. But only if we stop trying to cram how we treat AI and LLMs and the consequences of this new technology into a binary "allowed / not allowed" outcome as determined by an aging system that has long needed an overhaul.
So please, continue to debate for intellectual curiosity. I'd rather spend hours reading a truly curious exploration of this than another manifesto about "AI slop""},"story_title":{"matchLevel":"none","matchedWords":[],"value":"The current state of the theory that GPL propagates to AI models"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://shujisado.org/2025/11/27/gpl-propagates-to-ai-models-trained-on-gpl-code/"}},"_tags":["comment","author_tpmoney","story_46068777"],"author":"tpmoney","comment_text":"> An image that is lossily compressed, converted to a bitstream, and encoded in DNA is very very different than the input, but if an image can be recovered that is indistinguishable or barely distinguishable from the original, I'd still call that copying and each intermediate step a significant but irrelevant transformation.
Sure as a broad rule of thumb that works. But the ability of a machine to produce a copyright violation doesn't mean the machine itself or distributing the machine is a copyright violation. To take an extreme example, if we take a room full infinite monkeys and put them on infinite typewriters and they generate a Harry Potter book, that doesn't mean Harry Potter is stored in the monkey room. If we have a random sound generator that produces random tones from the standard western musical note pallet and it generates the bass line from "Under Pressure" that doesn't mean our random sound generator contains or is a copy of "Under Pressure", even if we encoded all the same information and procedures for generating those individual notes at those durations among the data procedures we gave the machine.
> If an LLM is completely incapable of reproducing input text verbatim, yet could become so through targeted ablation (that does not itself incorporate the text in question!), then does it store that text or not?
I would argue not. Just like a xerox machine doesn't contain the books you make copies of when you use it to make a copy, and Handbrake doesn't contain the DVD's you use when you make a copy there.
I would further argue that copyright infringement is inherently a "human" act. It's sort of encoded in the language we use to talk about it (e.g. "fair use") but it's also something of a "if a tree falls in the middle of the woods" situation. If an LLM runs in an isolated room in an isolated bunker with no one around and generates verbatim copies of the Linux kernel, that frankly doesn't matter. On the other hand, if a Microsoft employee induces an LLM to produce verbatim copies of the Linux kernel, that does, especially if they did so with the intent to incorporate Linux kernel code into Windows. Not because of the LLM, but because a person made the choice to produce a copy of something they didn't have the right to make a copy of. The method by which they accomplished that copy is less relevant than making the copy at all, and that in turn is less relevant than the intent of making that copy for a purpose which is not allowed by copyright law.
> I'm not sure why I'm even debating this, other than for intellectual curiosity.
Frankly, that's the only reason to debate anything. 99% of the time, you as an individual will never have the power to influence the actual legal decisions made. But a intellectually curious conversation is infinitely more useful, not just to you and me but to other readers, than another retread of "AI is slop" "you're just jealous you can't code your way out of a paper bag" arguments that pervade so much discussion around AI. Or worse yet another "I used an LLM for a clearly stupid thing and it was stupid" or "I used an LLM to replace all my employees and I'm sure it's going to go great" blog post. For whatever acrimony there might have been in our interchange here, I'm sorry, because this sort of discussion is the only good way to exercise our thoughts on an issue and really test them out ourselves. It's easy to have a knee jerk opinion. It's harder to support that opinion with a philosophy and reasoning.
For what it's worth, I view the LLM/AI world as the best opportunity we've had in decades to really rethink and scale back/change how we deal with intellectual property. The ever expanding copyright terms, the sometimes bizarre protections of what seem to be blindingly obvious ideas. The technological age has demonstrated a number of weaknesses in the traditional systems and views. And frankly I think it's also demonstrated that many prior predictions of certain doom if copyright wasn't strictly enforced have been overwrought and even where they haven't, the actual result has been better for more people. Famously, IBM would have very much preferred to have won the BIOS copyright issue. But I think so much of the modern computer and tech industry owes their very careers to the effects of that decision. It might have been better for IBM if IBM had won, it's not clear at all that it would have been better for "[promoting] the Progress of Science and useful Arts".
We could live in a world where we recognize that LLMs and AIs are going to fundamentally change how we approach creative works. We could recognize that the intents of "[promoting] the Progress of Science and useful Arts" is still a relevant goal and something we can work to make compatible with the existence of LLMs and AI. To pitch my crazy idea again, we could:
1) Cut the terms of copyright substantially, back down to 10 or 15 years by default.
2) Offer a single extension that doubles that term, but only on the condition that the work is submitted to a central "library of congress" data set.
3) This could be used to produce known good and clean data sets for AI companies and organization to train models from, with the protection that any model trained from this data set is protected from copyright infringement claims for works in the data set. Heck we could even produce common models. This would save massive amounts of power and resources by cutting the need for everyone who wants to be in the AI space to go out and acquire, digitize and build their own library. The NIST numbers set is effectively the "hello world" set for anyone learning computer vision AI stuff. Let's do that for all sort of AI.
4) The data sets and models would be provided for a nominal fee, this fee will be used to pay royalties to people whose works are still under copyright and are in the data sets, proportional to the recency and quantity of work submitted. A cap would need to be put in place to prevent flooding the data set to game the royalties. These royalties would be part of recognizing the value the original works contributed to the data set, and act as a further incentive to contribute works to the system and contribute them sooner.
We could build a system like this, or tweak it, or even build something else entirely. But only if we stop trying to cram how we treat AI and LLMs and the consequences of this new technology into a binary "allowed / not allowed" outcome as determined by an aging system that has long needed an overhaul.
So please, continue to debate for intellectual curiosity. I'd rather spend hours reading a truly curious exploration of this than another manifesto about "AI slop"","created_at":"2025-11-28T04:07:40Z","created_at_i":1764302860,"objectID":"46075420","parent_id":46075139,"story_id":46068777,"story_title":"The current state of the theory that GPL propagates to AI models","story_url":"https://shujisado.org/2025/11/27/gpl-propagates-to-ai-models-trained-on-gpl-code/","updated_at":"2026-03-05T23:09:22Z"}],"hitsPerPage":20,"nbHits":5,"nbPages":1,"page":0,"params":"query=LLM+Inference+Handbook&advancedSyntax=true&analyticsTags=backend","processingTimeMS":19,"processingTimingsMS":{"_request":{"roundTrip":14},"afterFetch":{"format":{"highlighting":1,"total":1},"merge":{"mergeLoop":{"prepareNextHit":1,"total":1},"total":2},"total":2},"fetch":{"query":13,"scanning":3,"total":17},"total":19},"query":"LLM Inference Handbook","serverTimeMS":21}