We created this handbook to make LLM inference concepts more accessible, especially for developers building real-world LLM applications. The goal is to pull together scattered knowledge into something clear, practical, and easy to build on.
We\u2019re continuing to improve it, so feedback is very welcome!
GitHub repo: https://github.com/bentoml/llm-inference-in-production"},"story_title":{"fullyHighlighted":true,"matchLevel":"full","matchedWords":["llm","inference","handbook"],"value":"LLM Inference Handbook"},"story_url":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["llm"],"value":"https://bentoml.com/llm/"}},"_tags":["comment","author_sherlockxu","story_44527947"],"author":"sherlockxu","children":[44530135,44535657,44536421,44537248],"comment_text":"Hi everyone. I'm one of the maintainers of this project. We're both excited and humbled to see it on Hacker News!
We created this handbook to make LLM inference concepts more accessible, especially for developers building real-world LLM applications. The goal is to pull together scattered knowledge into something clear, practical, and easy to build on.
We\u2019re continuing to improve it, so feedback is very welcome!
GitHub repo: https://github.com/bentoml/llm-inference-in-production","created_at":"2025-07-11T08:00:11Z","created_at_i":1752220811,"objectID":"44529483","parent_id":44527947,"story_id":44527947,"story_title":"LLM Inference Handbook","story_url":"https://bentoml.com/llm/","updated_at":"2025-07-29T14:41:07Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"gmerc"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["llm","inference","handbook"],"value":"Objectively the trajectory is what matters not todays capabilities. And the trajectory tells its own story.
As with any technology there is both hype and not hype, there is no boolean answer to this question.
Is there hype ? absolutely\u2026 Langchain is not a 100M technology and chatgpt plug-ins are an idea at this point that may take time to find its final form. But this is par for the course for Silicon valley ever since investment became a last-to-invest holds the bag game so perfected by A16z.
Bing and search are at best a mediocre application of LLMs, much better executed with embedding storage and supporting technology rather than static models - the choice of GTM in this regard is classic Silicon Valley gaming and distracts from the underlying mechanics and broad potential of the technology.
But compared to crypto, web3 and metaverse, today, with our own eyes we can see:
- Computers could not draw/paint 2 years ago now they do at the level of a master artist.
- 9 month ago we diffused a 512x512 picture with barely any control in 6 seconds on top end hardware, today any consumer rig with a nvidia gaming GPU can create almost unlimited size images and top end cards diffuse at up to 30-fps. We have full control over the composition with controlnet too.
- Computers could not draw hands six month ago, now they do
- Computers could not code 2 years ago, a year ago barely autocompleted a function and now they certainly can do smaller programs and assist engineers in solving complex problems
- Computers could not pass any tests 2 years ago, now they beat the US bar exam and various CS entry exams
- computer vision could could objects and classify a few dozen object classes a few years ago now it can tell timmy is sad because his toy is broken in an image and basically segment anything
- Computers could not compose 2 years ago and now you could do it on your home computer
- A year ago you needed an A100 to run an LLM now it runs on a fucking raspberry pi quantized and most stock macbooks and you can download some on an iphone.
- LLM hallucinations and failure to do math seemed a major issue 8 month ago, now there\u2019s paths with embedding storage that allows for verified citations and they can invoke tools like wolfram alpha or create code to solve equations.
- 2 Years ago siri and co still failed at even the slightest foreign accent, today whisper almost perfectly transcribes heavily accented or foreign language text at a rate of hours in single digit minutes or even seconds on high end hardware
- 2 years ago text to speech was still a robotic affair, now it can clone a human voice including inflection, emotion, speech impediment, etc.
- inference capability is up 30x on hardware and many times on software in a year. Llama went from A100 to raspberry pi in 2 months after release.
- We see the ability for LLMs to reason complex topics and act as decision makers, approaching on many human jobs that are rigidly encased in policies and employee handbooks and rarely require or allow human judgement.
Those of us old enough remember the same dynamics with the .com bust and the breathless declarations that the internet was a fad in response to people being unable to contain their FOMO investing.
for all the hype, there is an iceberg of unrealized change underneath the mountain that is the last 12 months of change that\u2019s severely under appreciated and misunderstood.
Wall street barely glimpsed it a few weeks ago when it sent Nvidia to 1T. In the hands of experts over the next months, it is likely this technology will start having a terrible impact on the job market, IBMs virtue signaling a few weeks ago should make that clear to anyone.
It baffles that people keep retreating on either on the current flaws or on ideological positions predicated on shifting definitions or picking on narrow aspects of these systems rather than the whole. Anyone of sound mind should be humbled by these developments and very carefully phrase their predictions of the future.
I\u2019m sure people were dismissive of the first automobile or the internal combustion engine on the account of it not moving on its own too. I don\u2019t remember anyone opiniating \u201cit\u2019s just gradient descent\u201d when ad delivery birthed much of big tech and was accused of throwing elections and destroying social cohesion globally.
What is also missing from this debate is the devastating realization that this is the first major technological leap that happened a almost completely unbounded from infrastructure requirements. No power lines to be strung, no cell towers to be built no app stores needed to propagate. Everyone withdrawing on \u2018we always found new jobs in response to technology affecting old ones\u2019 may be right, but may be missing this factor.
This technology scales at a 10 minute API integration only bounded by Nvidias ability to produce GPUs. Only a small minority of people have engaged with it yet, but it will find applications in every job, every company over time. And it will never get worse, never slow down, never sleep.
No the only reasonable course of action here is to be humble, it looks very likely that just like with the .com boom and bust the technology will end up being life changing (something Silicon Valley barely remembers after a decade of bullshit pumping)"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"The AI Hype Wall of Shame"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://criticalai.org/the-ai-hype-wall-of-shame/"}},"_tags":["comment","author_gmerc","story_36280603"],"author":"gmerc","children":[36284449],"comment_text":"Objectively the trajectory is what matters not todays capabilities. And the trajectory tells its own story.
As with any technology there is both hype and not hype, there is no boolean answer to this question.
Is there hype ? absolutely\u2026 Langchain is not a 100M technology and chatgpt plug-ins are an idea at this point that may take time to find its final form. But this is par for the course for Silicon valley ever since investment became a last-to-invest holds the bag game so perfected by A16z.
Bing and search are at best a mediocre application of LLMs, much better executed with embedding storage and supporting technology rather than static models - the choice of GTM in this regard is classic Silicon Valley gaming and distracts from the underlying mechanics and broad potential of the technology.
But compared to crypto, web3 and metaverse, today, with our own eyes we can see:
- Computers could not draw/paint 2 years ago now they do at the level of a master artist.
- 9 month ago we diffused a 512x512 picture with barely any control in 6 seconds on top end hardware, today any consumer rig with a nvidia gaming GPU can create almost unlimited size images and top end cards diffuse at up to 30-fps. We have full control over the composition with controlnet too.
- Computers could not draw hands six month ago, now they do
- Computers could not code 2 years ago, a year ago barely autocompleted a function and now they certainly can do smaller programs and assist engineers in solving complex problems
- Computers could not pass any tests 2 years ago, now they beat the US bar exam and various CS entry exams
- computer vision could could objects and classify a few dozen object classes a few years ago now it can tell timmy is sad because his toy is broken in an image and basically segment anything
- Computers could not compose 2 years ago and now you could do it on your home computer
- A year ago you needed an A100 to run an LLM now it runs on a fucking raspberry pi quantized and most stock macbooks and you can download some on an iphone.
- LLM hallucinations and failure to do math seemed a major issue 8 month ago, now there\u2019s paths with embedding storage that allows for verified citations and they can invoke tools like wolfram alpha or create code to solve equations.
- 2 Years ago siri and co still failed at even the slightest foreign accent, today whisper almost perfectly transcribes heavily accented or foreign language text at a rate of hours in single digit minutes or even seconds on high end hardware
- 2 years ago text to speech was still a robotic affair, now it can clone a human voice including inflection, emotion, speech impediment, etc.
- inference capability is up 30x on hardware and many times on software in a year. Llama went from A100 to raspberry pi in 2 months after release.
- We see the ability for LLMs to reason complex topics and act as decision makers, approaching on many human jobs that are rigidly encased in policies and employee handbooks and rarely require or allow human judgement.
Those of us old enough remember the same dynamics with the .com bust and the breathless declarations that the internet was a fad in response to people being unable to contain their FOMO investing.
for all the hype, there is an iceberg of unrealized change underneath the mountain that is the last 12 months of change that\u2019s severely under appreciated and misunderstood.
Wall street barely glimpsed it a few weeks ago when it sent Nvidia to 1T. In the hands of experts over the next months, it is likely this technology will start having a terrible impact on the job market, IBMs virtue signaling a few weeks ago should make that clear to anyone.
It baffles that people keep retreating on either on the current flaws or on ideological positions predicated on shifting definitions or picking on narrow aspects of these systems rather than the whole. Anyone of sound mind should be humbled by these developments and very carefully phrase their predictions of the future.
I\u2019m sure people were dismissive of the first automobile or the internal combustion engine on the account of it not moving on its own too. I don\u2019t remember anyone opiniating \u201cit\u2019s just gradient descent\u201d when ad delivery birthed much of big tech and was accused of throwing elections and destroying social cohesion globally.
What is also missing from this debate is the devastating realization that this is the first major technological leap that happened a almost completely unbounded from infrastructure requirements. No power lines to be strung, no cell towers to be built no app stores needed to propagate. Everyone withdrawing on \u2018we always found new jobs in response to technology affecting old ones\u2019 may be right, but may be missing this factor.
This technology scales at a 10 minute API integration only bounded by Nvidias ability to produce GPUs. Only a small minority of people have engaged with it yet, but it will find applications in every job, every company over time. And it will never get worse, never slow down, never sleep.
No the only reasonable course of action here is to be humble, it looks very likely that just like with the .com boom and bust the technology will end up being life changing (something Silicon Valley barely remembers after a decade of bullshit pumping)","created_at":"2023-06-11T13:29:33Z","created_at_i":1686490173,"objectID":"36281191","parent_id":36280603,"story_id":36280603,"story_title":"The AI Hype Wall of Shame","story_url":"https://criticalai.org/the-ai-hype-wall-of-shame/","updated_at":"2026-03-22T15:48:17Z"}],"hitsPerPage":20,"nbHits":4,"nbPages":1,"page":0,"params":"query=LLM+Inference+Handbook&advancedSyntax=true&analyticsTags=backend","processingTimeMS":16,"processingTimingsMS":{"_request":{"roundTrip":18},"fetch":{"query":11,"scanning":3,"total":15},"total":16},"query":"LLM Inference Handbook","serverTimeMS":18}