I wouldn't normally think folks on HN would find this interesting as the general concept has been posted about already in the past few months. We were heavily inspired by Zerox[1].
However, the stack we went with was fun and over-engineered which is more likely to create interesting discussion. We use all the same tools at Trieve (our main product), but wanted to see if they would be a good fit for something that needed to get built in a tighter timeline and we think they were!
Took us 2 weeks to get this setup end-to-end and it's by no means complete (see roadmap in linked README). However, it's cool that a relatively cookie cutter web service like this can be created with pure open-source dependencies and non-standard Rust tooling so quickly. Rust won't kill your startup!
- Minijinja templates for the UI[2]
- PDFObject for doc display in-browser[3]
- actix/actix-web HTTP server framework[4]
- Redis queue macro for worker async processing[5]
- Clickhouse for task storage[6]
- chm CLI to handle Clickhouse migrations[7]
- MinIO S3 for object storage[8]
[1]: https://news.ycombinator.com/item?id=41048194
[2]: https://github.com/mitsuhiko/minijinja
[3]: https://github.com/pipwerks/pdfobject
[4]: https://github.com/actix/actix-web
[5]: https://github.com/devflowinc/trieve/blob/main/pdf2md/server...
[6]: https://github.com/ClickHouse/ClickHouse
[7]: https://docs.rs/chm/latest/chm/index.html
[8]: https://github.com/minio/minio"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["vllm"],"value":"Show HN: PDF2MD \u2013 Rust+Redis+ClickHouse+VLLM conversion pipeline for PDFs"},"url":{"matchLevel":"none","matchedWords":[],"value":"https://github.com/devflowinc/trieve/tree/main/pdf2md"}},"_tags":["story","author_skeptrune","story_42208648","show_hn"],"author":"skeptrune","created_at":"2024-11-21T21:05:41Z","created_at_i":1732223141,"num_comments":0,"objectID":"42208648","points":13,"story_id":42208648,"story_text":"If you just want to use it, try here - https://pdf2md.trieve.ai . I think the LLM's are astoundingly good at converting complex powerpoint style infographics.
I wouldn't normally think folks on HN would find this interesting as the general concept has been posted about already in the past few months. We were heavily inspired by Zerox[1].
However, the stack we went with was fun and over-engineered which is more likely to create interesting discussion. We use all the same tools at Trieve (our main product), but wanted to see if they would be a good fit for something that needed to get built in a tighter timeline and we think they were!
Took us 2 weeks to get this setup end-to-end and it's by no means complete (see roadmap in linked README). However, it's cool that a relatively cookie cutter web service like this can be created with pure open-source dependencies and non-standard Rust tooling so quickly. Rust won't kill your startup!
- Minijinja templates for the UI[2]
- PDFObject for doc display in-browser[3]
- actix/actix-web HTTP server framework[4]
- Redis queue macro for worker async processing[5]
- Clickhouse for task storage[6]
- chm CLI to handle Clickhouse migrations[7]
- MinIO S3 for object storage[8]
[1]: https://news.ycombinator.com/item?id=41048194
[2]: https://github.com/mitsuhiko/minijinja
[3]: https://github.com/pipwerks/pdfobject
[4]: https://github.com/actix/actix-web
[5]: https://github.com/devflowinc/trieve/blob/main/pdf2md/server...
[6]: https://github.com/ClickHouse/ClickHouse
[7]: https://docs.rs/chm/latest/chm/index.html
[8]: https://github.com/minio/minio","title":"Show HN: PDF2MD \u2013 Rust+Redis+ClickHouse+VLLM conversion pipeline for PDFs","updated_at":"2024-12-30T23:52:44Z","url":"https://github.com/devflowinc/trieve/tree/main/pdf2md"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"sherlockxu"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["vllm"],"value":"Benchmarking LLM Inference Back Ends: VLLM, LMDeploy, MLC-LLM, TRT-LLM, and TGI"},"url":{"matchLevel":"none","matchedWords":[],"value":"https://bentoml.com/blog/benchmarking-llm-inference-backends"}},"_tags":["story","author_sherlockxu","story_40601794"],"author":"sherlockxu","children":[40601960,40604730,40605032],"created_at":"2024-06-06T20:08:00Z","created_at_i":1717704480,"num_comments":2,"objectID":"40601794","points":12,"story_id":40601794,"title":"Benchmarking LLM Inference Back Ends: VLLM, LMDeploy, MLC-LLM, TRT-LLM, and TGI","updated_at":"2024-09-20T17:09:35Z","url":"https://bentoml.com/blog/benchmarking-llm-inference-backends"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"zhwu"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["vllm"],"value":"Serving LLM 24x Faster on the Cloud with VLLM and SkyPilot"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["vllm"],"value":"https://blog.skypilot.co/serving-llm-24x-faster-on-the-cloud-with-vllm-and-skypilot/"}},"_tags":["story","author_zhwu","story_36523357"],"author":"zhwu","children":[36523632],"created_at":"2023-06-29T17:11:17Z","created_at_i":1688058677,"num_comments":1,"objectID":"36523357","points":12,"story_id":36523357,"title":"Serving LLM 24x Faster on the Cloud with VLLM and SkyPilot","updated_at":"2024-09-20T14:23:54Z","url":"https://blog.skypilot.co/serving-llm-24x-faster-on-the-cloud-with-vllm-and-skypilot/"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"btwillard"},"story_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["vllm"],"value":"Our project, Outlines, now offers guided/constrained generation (e.g. according to a JSON schema) via the VLLM library.
My colleague, R\u00e9mi, created some patches that allow one to pass vLLM a JSON schema along with the prompt, which dramatically simplifies deployment of JSON-guided generation. He also added a new `serve` interface that puts it all together and makes serving such models a 2-3 line process.
Check it out and tell us what you think!"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["vllm"],"value":"Show HN: VLLM with JSON Guided Generation"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["vllm"],"value":"https://outlines-dev.github.io/outlines/reference/vllm/"}},"_tags":["story","author_btwillard","story_38735503","show_hn"],"author":"btwillard","children":[38735564],"created_at":"2023-12-22T16:12:11Z","created_at_i":1703261531,"num_comments":3,"objectID":"38735503","points":11,"story_id":38735503,"story_text":"Our project, Outlines, now offers guided/constrained generation (e.g. according to a JSON schema) via the VLLM library.
My colleague, R\u00e9mi, created some patches that allow one to pass vLLM a JSON schema along with the prompt, which dramatically simplifies deployment of JSON-guided generation. He also added a new `serve` interface that puts it all together and makes serving such models a 2-3 line process.
Check it out and tell us what you think!","title":"Show HN: VLLM with JSON Guided Generation","updated_at":"2024-09-20T15:58:59Z","url":"https://outlines-dev.github.io/outlines/reference/vllm/"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"twelvenmonkeys"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["vllm"],"value":"Free vLLM Course: Inference, Compression, Benchmarks"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["vllm"],"value":"https://www.deeplearning.ai/courses/fast-and-efficient-llm-inference-with-vllm"}},"_tags":["story","author_twelvenmonkeys","story_48386932"],"author":"twelvenmonkeys","children":[48386933],"created_at":"2026-06-03T17:27:14Z","created_at_i":1780507634,"num_comments":0,"objectID":"48386932","points":8,"story_id":48386932,"title":"Free vLLM Course: Inference, Compression, Benchmarks","updated_at":"2026-06-05T14:16:48Z","url":"https://www.deeplearning.ai/courses/fast-and-efficient-llm-inference-with-vllm"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"pember"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["vllm"],"value":"Heaps do lie: debugging a memory leak in vLLM"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["vllm"],"value":"https://mistral.ai/news/debugging-memory-leak-in-vllm"}},"_tags":["story","author_pember","story_46707015"],"author":"pember","children":[46824814],"created_at":"2026-01-21T15:27:39Z","created_at_i":1769009259,"num_comments":1,"objectID":"46707015","points":7,"story_id":46707015,"title":"Heaps do lie: debugging a memory leak in vLLM","updated_at":"2026-03-05T23:22:30Z","url":"https://mistral.ai/news/debugging-memory-leak-in-vllm"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"ericcurtin"},"story_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["vllm"],"value":"Hi HN, I\u2019m one of the authors of this post.
We\u2019ve updated Docker Model Runner to support vLLM alongside the existing llama.cpp backend. The goal is to bridge the gap between local prototyping (often done with GGUF/llama.cpp) and high-throughput production (often done with Safetensors/vLLM) using a consistent Docker workflow.
Key technical details:
Auto-routing: The tool detects the model format. If you pull a GGUF model, it routes to llama.cpp. If you pull a Safetensors model, it routes to vLLM.
API: It exposes an OpenAI-compatible API (/v1/chat/completions), so the client code doesn't need to change based on the backend.
Usage: It\u2019s just docker model run ai/smollm2-vllm.
Current Limitations:
Right now, the vLLM backend is optimized for x86_64 with Nvidia GPUs.
We are actively working on WSL2 support for Windows users and DGX Spark compatibility.
Happy to answer any questions about the integration or the roadmap!
https://www.docker.com/blog/docker-model-runner-integrates-v..."},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["vllm"],"value":"Show HN: Docker Model Runner Integrates vLLM for High-Throughput Inference"},"url":{"matchLevel":"none","matchedWords":[],"value":"https://github.com/docker/model-runner"}},"_tags":["story","author_ericcurtin","story_45996081","show_hn"],"author":"ericcurtin","children":[45996088],"created_at":"2025-11-20T18:41:51Z","created_at_i":1763664111,"num_comments":1,"objectID":"45996081","points":7,"story_id":45996081,"story_text":"Hi HN, I\u2019m one of the authors of this post.
We\u2019ve updated Docker Model Runner to support vLLM alongside the existing llama.cpp backend. The goal is to bridge the gap between local prototyping (often done with GGUF/llama.cpp) and high-throughput production (often done with Safetensors/vLLM) using a consistent Docker workflow.
Key technical details:
Auto-routing: The tool detects the model format. If you pull a GGUF model, it routes to llama.cpp. If you pull a Safetensors model, it routes to vLLM.
API: It exposes an OpenAI-compatible API (/v1/chat/completions), so the client code doesn't need to change based on the backend.
Usage: It\u2019s just docker model run ai/smollm2-vllm.
Current Limitations:
Right now, the vLLM backend is optimized for x86_64 with Nvidia GPUs.
We are actively working on WSL2 support for Windows users and DGX Spark compatibility.
Happy to answer any questions about the integration or the roadmap!
https://www.docker.com/blog/docker-model-runner-integrates-v...","title":"Show HN: Docker Model Runner Integrates vLLM for High-Throughput Inference","updated_at":"2026-04-01T17:11:45Z","url":"https://github.com/docker/model-runner"}],"hitsPerPage":20,"nbHits":307689,"nbPages":50,"page":0,"params":"query=vllm&advancedSyntax=true&analyticsTags=backend","processingTimeMS":25,"processingTimingsMS":{"_request":{"roundTrip":13},"fetch":{"query":2,"scanning":21,"total":24},"total":25},"query":"vllm","serverTimeMS":26}