I use Deepseek V4 flash 0731 Q8 quantisation from unsolth with reasoning turned off on a local server for quick easy tasks. Note I use another model locally for harder tasks that need reasoning.
I was a bit dissappointed that it knew it was Poe's cypher so didn't need to even try to crack the code."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"The Gold-Bug: Edgar Allan Poe's story that taught me about cryptography (2024)"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://robotsinplainenglish.com/e/2024-01-21-gold-bug.html"}},"_tags":["comment","author_ryan_glass","story_50055632"],"author":"ryan_glass","children":[50089274],"comment_text":"I thought it would be interesting to see if the local AI I use for quick question and answer stuff could crack the code.
I use Deepseek V4 flash 0731 Q8 quantisation from unsolth with reasoning turned off on a local server for quick easy tasks. Note I use another model locally for harder tasks that need reasoning.
I was a bit dissappointed that it knew it was Poe's cypher so didn't need to even try to crack the code.","created_at":"2026-10-14T15:31:26Z","created_at_i":1791991886,"objectID":"50089234","parent_id":50055632,"story_id":50055632,"story_title":"The Gold-Bug: Edgar Allan Poe's story that taught me about cryptography (2024)","story_url":"https://robotsinplainenglish.com/e/2024-01-21-gold-bug.html","updated_at":"2026-10-14T15:34:13Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"muestramuertos"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["unsloth"],"value":""spun up"/"spun this up" wfm. genuinely curious, are you surprised to learn that vibecoders often are unbothered by poor diction?"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Ask HN: Why do people say \"I built it\" when we should say \"Claude built it\"?"}},"_tags":["comment","author_muestramuertos","story_50085114"],"author":"muestramuertos","children":[50086772],"comment_text":""spun up"/"spun this up" wfm. genuinely curious, are you surprised to learn that vibecoders often are unbothered by poor diction?","created_at":"2026-10-14T12:30:15Z","created_at_i":1791981015,"objectID":"50086661","parent_id":50085114,"story_id":50085114,"story_title":"Ask HN: Why do people say \"I built it\" when we should say \"Claude built it\"?","updated_at":"2026-10-14T13:24:43Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"wavesplash"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["unsloth"],"value":"Having built a pulsed vibration wristband for someone with Parkinson's, just a few notes. For Parkinson's patients:
- there's a small-ish window in the disease progression where pulsed vibration is able to help a person regain some intentional motor control\n - the device doesn't stop the tremors\n - the pulsed vibration does let the brain's control signal come through stronger, so drawing/typing/moving the mouse is more intentional. \n - this is no more effective than the results of levidopa/carbidopa treatment, and the levidopa/carbidopa typically calms the overall tremors\n - as Parkinson's progresses, these devices become useless, and if the patient qualifies for deep brain stimulation, it's worth looking into (regain years of useful life if you're lucky).\nYou can make a very unsophisticated version of this without a lot of hardware:
- small microcontroller with 3v outputs (XIAO's are perfect)\n - 3v sealed eccentric mass motors (aliexpress $1/ea)\n - elastic wristband\n - usb battery\n - software to pulse the motors for about 200 miliseconds for an on off sequence of about 8 times, then rest for about a second and repeat.\nSide note: the idea was published in 2016-2017 in a few promotional videos from Microsoft research: https://www.windowscentral.com/microsofts-emma-watch-and-pro..."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"MIT class project turns into an FDA-cleared treatment for tremors"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://news.mit.edu/2026/mit-class-project-turns-into-fda-cleared-treatment-for-tremors-1002"}},"_tags":["comment","author_wavesplash","story_50061659"],"author":"wavesplash","children":[50087415],"comment_text":"Having built a pulsed vibration wristband for someone with Parkinson's, just a few notes. For Parkinson's patients:
- there's a small-ish window in the disease progression where pulsed vibration is able to help a person regain some intentional motor control\n - the device doesn't stop the tremors\n - the pulsed vibration does let the brain's control signal come through stronger, so drawing/typing/moving the mouse is more intentional. \n - this is no more effective than the results of levidopa/carbidopa treatment, and the levidopa/carbidopa typically calms the overall tremors\n - as Parkinson's progresses, these devices become useless, and if the patient qualifies for deep brain stimulation, it's worth looking into (regain years of useful life if you're lucky).\nYou can make a very unsophisticated version of this without a lot of hardware:
- small microcontroller with 3v outputs (XIAO's are perfect)\n - 3v sealed eccentric mass motors (aliexpress $1/ea)\n - elastic wristband\n - usb battery\n - software to pulse the motors for about 200 miliseconds for an on off sequence of about 8 times, then rest for about a second and repeat.\nSide note: the idea was published in 2016-2017 in a few promotional videos from Microsoft research: https://www.windowscentral.com/microsofts-emma-watch-and-pro...","created_at":"2026-10-14T00:29:17Z","created_at_i":1791937757,"objectID":"50080883","parent_id":50061659,"story_id":50061659,"story_title":"MIT class project turns into an FDA-cleared treatment for tremors","story_url":"https://news.mit.edu/2026/mit-class-project-turns-into-fda-cleared-treatment-for-tremors-1002","updated_at":"2026-10-14T15:30:14Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"vor_"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["unsloth"],"value":"I browse Hacker News stories via this: https://unslop.news/"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"11 year old unsoundness bugs found in Rust borrow checker"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://github.com/rust-lang/rust/issues/164250"}},"_tags":["comment","author_vor_","story_50069499"],"author":"vor_","children":[50085068,50079480,50078446],"comment_text":"I browse Hacker News stories via this: https://unslop.news/","created_at":"2026-10-13T20:36:37Z","created_at_i":1791923797,"objectID":"50078353","parent_id":50070934,"story_id":50069499,"story_title":"11 year old unsoundness bugs found in Rust borrow checker","story_url":"https://github.com/rust-lang/rust/issues/164250","updated_at":"2026-10-14T10:05:12Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"car"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["unsloth"],"value":"I should add that Llama.app and Unsloth Studio provide OpenAI-compatible API endpoints (both use Llama.cpp as their backend).
https://unsloth.ai/docs/new/studio#unsloth-as-an-api-endpoin..."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Show HN: Atticus \u2013 Hold-to-talk voice assistant for macOS that runs on Claude -p"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://github.com/SaarthurR/atticus-mac"}},"_tags":["comment","author_car","story_50049588"],"author":"car","comment_text":"I should add that Llama.app and Unsloth Studio provide OpenAI-compatible API endpoints (both use Llama.cpp as their backend).
https://unsloth.ai/docs/new/studio#unsloth-as-an-api-endpoin...","created_at":"2026-10-13T19:55:28Z","created_at_i":1791921328,"objectID":"50077702","parent_id":50069978,"story_id":50049588,"story_title":"Show HN: Atticus \u2013 Hold-to-talk voice assistant for macOS that runs on Claude -p","story_url":"https://github.com/SaarthurR/atticus-mac","updated_at":"2026-10-13T20:00:13Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"car"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["unsloth"],"value":"The choice of model could be up to the user, depending on the available RAM on their M class Mac. Newer and bigger usually being smarter.
Llama.app/.cpp and Unsloth Studio on Mac already provide guidance to pick the optimal model for the available memory, all model wheights hosted by huggingface. Both apps can use the local huggingface model cache locally.
If you make it configurable, the user could also run a separate machine with an NVidia card and Linux to serve tokens on the LAN. Or use OpenRouter or any other cloud hosted model offering (DeepSeek, Z.ai, etc.)."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Show HN: Atticus \u2013 Hold-to-talk voice assistant for macOS that runs on Claude -p"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://github.com/SaarthurR/atticus-mac"}},"_tags":["comment","author_car","story_50049588"],"author":"car","children":[50077702],"comment_text":"The choice of model could be up to the user, depending on the available RAM on their M class Mac. Newer and bigger usually being smarter.
Llama.app/.cpp and Unsloth Studio on Mac already provide guidance to pick the optimal model for the available memory, all model wheights hosted by huggingface. Both apps can use the local huggingface model cache locally.
If you make it configurable, the user could also run a separate machine with an NVidia card and Linux to serve tokens on the LAN. Or use OpenRouter or any other cloud hosted model offering (DeepSeek, Z.ai, etc.).","created_at":"2026-10-13T11:28:13Z","created_at_i":1791890893,"objectID":"50069978","parent_id":50064951,"story_id":50049588,"story_title":"Show HN: Atticus \u2013 Hold-to-talk voice assistant for macOS that runs on Claude -p","story_url":"https://github.com/SaarthurR/atticus-mac","updated_at":"2026-10-13T19:56:27Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"car"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["unsloth"],"value":"This looks and works great, super impressed. This is what Siri/Apple AI should be like.
One concern I have is about sending private info to Anthropic. So it would be great to have the option to use local AI.
Local options should support the OpenAI API that seemingly is the standard for these connections, as provided by Llama.app and Unsloth Studio for example. That way one could run against a local model and keep personal information private.
That of course begs the question if a local model is smart enough for the tasks at hand.
I'm also trying out OOMU, a similar but closed source personal assistant app for MacOS. It emphasizes privacy with local models, and even has an 'Airplane' mode. The app has a built in llama.cpp server and uses Gemma-4-4B, their own router model plus a vision model."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Show HN: Atticus \u2013 Hold-to-talk voice assistant for macOS that runs on Claude -p"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://github.com/SaarthurR/atticus-mac"}},"_tags":["comment","author_car","story_50049588"],"author":"car","children":[50064951,50087169,50069436],"comment_text":"This looks and works great, super impressed. This is what Siri/Apple AI should be like.
One concern I have is about sending private info to Anthropic. So it would be great to have the option to use local AI.
Local options should support the OpenAI API that seemingly is the standard for these connections, as provided by Llama.app and Unsloth Studio for example. That way one could run against a local model and keep personal information private.
That of course begs the question if a local model is smart enough for the tasks at hand.
I'm also trying out OOMU, a similar but closed source personal assistant app for MacOS. It emphasizes privacy with local models, and even has an 'Airplane' mode. The app has a built in llama.cpp server and uses Gemma-4-4B, their own router model plus a vision model.","created_at":"2026-10-12T23:20:44Z","created_at_i":1791847244,"objectID":"50064434","parent_id":50049588,"story_id":50049588,"story_title":"Show HN: Atticus \u2013 Hold-to-talk voice assistant for macOS that runs on Claude -p","story_url":"https://github.com/SaarthurR/atticus-mac","updated_at":"2026-10-14T13:08:28Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"notthemessiah"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["unsloth"],"value":"In 2014, he fired his PR person when she asked for a raise (she was very good at her job).
In Spring of 2018, investigative reporters at the LA Times found that Elon acted like a tyrant removing yellow hi-vis safety tape at Tesla factories because he didn't like the aesthetic, which lead to more workplace accidents. This was when Elon started getting extremely adversarial with the press.
https://www.latimes.com/business/autos/la-fi-hy-tesla-factor...
A few months later, during the Thai cave rescue, he flaunted some kind of coffin made from spare rocket parts as his plan to save children who were trapped in a Thai cave, and when one of the actual rescuers Vernon Unsworth told him the cave was too narrow for that to be of use, Elon called him a "pedo" (note, Elon also begged to go to Jeffrey Epstein's parties).
It all went downhill from there."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"X Is Bad, Though"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://lexontech.org/x-is-bad-though"}},"_tags":["comment","author_notthemessiah","story_50051172"],"author":"notthemessiah","children":[50054788],"comment_text":"In 2014, he fired his PR person when she asked for a raise (she was very good at her job).
In Spring of 2018, investigative reporters at the LA Times found that Elon acted like a tyrant removing yellow hi-vis safety tape at Tesla factories because he didn't like the aesthetic, which lead to more workplace accidents. This was when Elon started getting extremely adversarial with the press.
https://www.latimes.com/business/autos/la-fi-hy-tesla-factor...
A few months later, during the Thai cave rescue, he flaunted some kind of coffin made from spare rocket parts as his plan to save children who were trapped in a Thai cave, and when one of the actual rescuers Vernon Unsworth told him the cave was too narrow for that to be of use, Elon called him a "pedo" (note, Elon also begged to go to Jeffrey Epstein's parties).
It all went downhill from there.","created_at":"2026-10-12T05:26:26Z","created_at_i":1791782786,"objectID":"50051695","parent_id":50051601,"story_id":50051172,"story_title":"X Is Bad, Though","story_url":"https://lexontech.org/x-is-bad-though","updated_at":"2026-10-14T11:58:28Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"singingtoday"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["unsloth"],"value":"Everything. It's horrible. Bad colors, cringe delivery, awkward prose.
It's quintessential ai slop.
But that's fine, because nobody forced me to read it, so I'm unbothered."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Total Annihilation, rebuilt for modern computers"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://nanolathe.gg"}},"_tags":["comment","author_singingtoday","story_50045984"],"author":"singingtoday","comment_text":"Everything. It's horrible. Bad colors, cringe delivery, awkward prose.
It's quintessential ai slop.
But that's fine, because nobody forced me to read it, so I'm unbothered.","created_at":"2026-10-11T23:25:48Z","created_at_i":1791761148,"objectID":"50049167","parent_id":50046474,"story_id":50045984,"story_title":"Total Annihilation, rebuilt for modern computers","story_url":"https://nanolathe.gg","updated_at":"2026-10-14T09:27:27Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"woctordho"},"story_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["unsloth"],"value":"Open weight AI is like open source software. Users not only run the weights, but also modify the weights. It matters to develop training framework for local hardware.
The method to train large AI models on local hardware is generally called QLoRA (Low Rank Adaptation over Quantized base model). In the last few years it's usually done with HuggingFace Transformers (which is the basis of training frameworks such as Unsloth and Axolotl) and bnb 4-bit base model. However, bnb does not yet support MoE models, so the local training of recent MoE models seemed stall for some time.
Since Transformers 5.18, it's started to support GGUF, and it supports recent models with MoE and sparse attentions (and it's not slow, already faster than llama.cpp on Mac). GGUF is a versatile container format. It can be smaller than 4 bpw with surprisingly good quantization quality, so there are new possibilities for local training.
I've shown that we can train Qwen3.8-Flash-Next (125B-A6B + 51B engram) in 40 GiB VRAM without CPU offload. I've optimized it on Strix Halo and it trains at 200 token/s. There is still room to optimize, compared to > 1600 token/s prompt processing we've achieved, and the common sense that LoRA training (with gradient checkpointing) takes 4-5x work of prompt processing. CPU/disk offload (like Strata) and multi-GPU also need more work that I'm not currently focusing on.
On Strix Halo we can also train DeepSeek-V4-Flash (284B-A13B) in 90 GiB VRAM at 100 token/s, but I think it's less practical than Qwen3.8FN for local use."},"title":{"matchLevel":"none","matchedWords":[],"value":"Show HN: LoRA over GGUF \u2013 Train Qwen3.8-Flash-Next in 40G VRAM"},"url":{"matchLevel":"none","matchedWords":[],"value":"https://github.com/woct0rdho/transformers5-qwen3.5-recipe"}},"_tags":["story","author_woctordho","story_50034625","show_hn"],"author":"woctordho","created_at":"2026-10-10T16:43:43Z","created_at_i":1791650623,"num_comments":0,"objectID":"50034625","points":1,"story_id":50034625,"story_text":"Open weight AI is like open source software. Users not only run the weights, but also modify the weights. It matters to develop training framework for local hardware.
The method to train large AI models on local hardware is generally called QLoRA (Low Rank Adaptation over Quantized base model). In the last few years it's usually done with HuggingFace Transformers (which is the basis of training frameworks such as Unsloth and Axolotl) and bnb 4-bit base model. However, bnb does not yet support MoE models, so the local training of recent MoE models seemed stall for some time.
Since Transformers 5.18, it's started to support GGUF, and it supports recent models with MoE and sparse attentions (and it's not slow, already faster than llama.cpp on Mac). GGUF is a versatile container format. It can be smaller than 4 bpw with surprisingly good quantization quality, so there are new possibilities for local training.
I've shown that we can train Qwen3.8-Flash-Next (125B-A6B + 51B engram) in 40 GiB VRAM without CPU offload. I've optimized it on Strix Halo and it trains at 200 token/s. There is still room to optimize, compared to > 1600 token/s prompt processing we've achieved, and the common sense that LoRA training (with gradient checkpointing) takes 4-5x work of prompt processing. CPU/disk offload (like Strata) and multi-GPU also need more work that I'm not currently focusing on.
On Strix Halo we can also train DeepSeek-V4-Flash (284B-A13B) in 90 GiB VRAM at 100 token/s, but I think it's less practical than Qwen3.8FN for local use.","title":"Show HN: LoRA over GGUF \u2013 Train Qwen3.8-Flash-Next in 40G VRAM","updated_at":"2026-10-10T16:48:32Z","url":"https://github.com/woct0rdho/transformers5-qwen3.5-recipe"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"vinhnx"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["unsloth"],"value":"Train your own Decision Model with Unsloth"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["unsloth"],"value":"https://unsloth.ai/docs/basics/train-your-own-decision-model-with-unsloth"}},"_tags":["story","author_vinhnx","story_50030016"],"author":"vinhnx","created_at":"2026-10-10T05:53:26Z","created_at_i":1791611606,"num_comments":0,"objectID":"50030016","points":1,"story_id":50030016,"title":"Train your own Decision Model with Unsloth","updated_at":"2026-10-10T05:57:29Z","url":"https://unsloth.ai/docs/basics/train-your-own-decision-model-with-unsloth"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"armcat"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["unsloth"],"value":"Train your own decision models with Unsloth"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["unsloth"],"value":"https://unsloth.ai/docs/basics/train-your-own-decision-model-with-unsloth"}},"_tags":["story","author_armcat","story_50024959"],"author":"armcat","created_at":"2026-10-09T18:42:04Z","created_at_i":1791571324,"num_comments":0,"objectID":"50024959","points":1,"story_id":50024959,"title":"Train your own decision models with Unsloth","updated_at":"2026-10-09T19:50:28Z","url":"https://unsloth.ai/docs/basics/train-your-own-decision-model-with-unsloth"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"armcat"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["unsloth"],"value":"It's weird because two days after Jev was released there were a dozen decision models, a week later there are several dozen, mostly open source, OpenAI's own Decisions API [1] beats it, and you can easily finetune your own [2]. But as others have pointed out, this doesn't matter.
EDIT: As I wrote this Microsoft just released their own Decision-1 model [3].
[1] https://developers.openai.com/api/docs/guides/decisions
[2] https://unsloth.ai/docs/basics/train-your-own-decision-model...
[3] https://commandline.microsoft.com/microsoft-decision-1-model..."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Typesafe AI raises $870M at $7.5B"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://typesafe.ai/blog/series-ai"}},"_tags":["comment","author_armcat","story_50023450"],"author":"armcat","children":[50025599,50025244,50025032,50027570,50025104,50025216,50027807,50026128,50026956,50031096,50026329,50028525,50026376,50029194,50026008,50027019,50030503,50025811,50029104,50030325,50025522,50027489,50025656,50025533,50025397,50025155,50025618,50025079],"comment_text":"It's weird because two days after Jev was released there were a dozen decision models, a week later there are several dozen, mostly open source, OpenAI's own Decisions API [1] beats it, and you can easily finetune your own [2]. But as others have pointed out, this doesn't matter.
EDIT: As I wrote this Microsoft just released their own Decision-1 model [3].
[1] https://developers.openai.com/api/docs/guides/decisions
[2] https://unsloth.ai/docs/basics/train-your-own-decision-model...
[3] https://commandline.microsoft.com/microsoft-decision-1-model...","created_at":"2026-10-09T18:41:42Z","created_at_i":1791571302,"objectID":"50024954","parent_id":50023450,"story_id":50023450,"story_title":"Typesafe AI raises $870M at $7.5B","story_url":"https://typesafe.ai/blog/series-ai","updated_at":"2026-10-14T09:37:13Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"lanesun"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["unsloth"],"value":"Because the version officially released by DS is an extremely quantized version, and there are many new things in the architecture, Unsloth needs time to handle this, just as was the case with the previous DS V4 Flash."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Why isn't the industry freaking out about DeepSeek 4.1 Flash?"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://www.dgt.is/blog/2026-10-07-deepseek-freek-out/"}},"_tags":["comment","author_lanesun","story_50000488"],"author":"lanesun","comment_text":"Because the version officially released by DS is an extremely quantized version, and there are many new things in the architecture, Unsloth needs time to handle this, just as was the case with the previous DS V4 Flash.","created_at":"2026-10-09T13:32:22Z","created_at_i":1791552742,"objectID":"50020303","parent_id":50011264,"story_id":50000488,"story_title":"Why isn't the industry freaking out about DeepSeek 4.1 Flash?","story_url":"https://www.dgt.is/blog/2026-10-07-deepseek-freek-out/","updated_at":"2026-10-09T13:36:44Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"Iolaum"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["unsloth"],"value":"Indeed - Even with llama.cpp you can store n-grams to disk and use it. I m using unsloth's llama fork atm but a new llama release is out that may be better - hadn't had time to test it yet."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Step 5 Preview, a 1M-context MoE from StepFun, shows up on OpenRouter"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://openrouter.ai/stepfun/step-5-preview"}},"_tags":["comment","author_Iolaum","story_50007764"],"author":"Iolaum","comment_text":"Indeed - Even with llama.cpp you can store n-grams to disk and use it. I m using unsloth's llama fork atm but a new llama release is out that may be better - hadn't had time to test it yet.","created_at":"2026-10-09T13:26:54Z","created_at_i":1791552414,"objectID":"50020225","parent_id":50019856,"story_id":50007764,"story_title":"Step 5 Preview, a 1M-context MoE from StepFun, shows up on OpenRouter","story_url":"https://openrouter.ai/stepfun/step-5-preview","updated_at":"2026-10-09T13:31:31Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"horsawlarway"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["unsloth"],"value":"bookstack
keycloak
immich
jellyfin
several personal sites
home assistant
unsloth studio
several synology tools on a nas (drive/security/backups)
etc...
If it's "just you" your laptop is likely fine. Hosted tools really start to shine when you've got a group of folks using them. Ex - Wife/Parents/Kids/Extended Family.
Simple k3s cluster and the shallow end of kubernetes actually saves you a ton of time and effort (don't go into complex orchestrators or helm charts - just write small deployments in static yaml). You can get to roughly the same spot with just docker/compose if you want, but the capabilities cap out earlier (ex - multi-machine, resilient to any one machine going down, gpu sharding, etc)
Maintenance is basically "sudo pacman -Syyu" on a cron job."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Ask HN: What do you run on a $5 VPS that's worth keeping online 24/7?"}},"_tags":["comment","author_horsawlarway","story_49985548"],"author":"horsawlarway","comment_text":"bookstack
keycloak
immich
jellyfin
several personal sites
home assistant
unsloth studio
several synology tools on a nas (drive/security/backups)
etc...
If it's "just you" your laptop is likely fine. Hosted tools really start to shine when you've got a group of folks using them. Ex - Wife/Parents/Kids/Extended Family.
Simple k3s cluster and the shallow end of kubernetes actually saves you a ton of time and effort (don't go into complex orchestrators or helm charts - just write small deployments in static yaml). You can get to roughly the same spot with just docker/compose if you want, but the capabilities cap out earlier (ex - multi-machine, resilient to any one machine going down, gpu sharding, etc)
Maintenance is basically "sudo pacman -Syyu" on a cron job.","created_at":"2026-10-09T12:55:45Z","created_at_i":1791550545,"objectID":"50019802","parent_id":49985548,"story_id":49985548,"story_title":"Ask HN: What do you run on a $5 VPS that's worth keeping online 24/7?","updated_at":"2026-10-09T12:59:57Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"hypfer"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["unsloth"],"value":"Is it known why unsloth seems to not have touched DeepSeek 4.1 Flash?"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Why isn't the industry freaking out about DeepSeek 4.1 Flash?"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://www.dgt.is/blog/2026-10-07-deepseek-freek-out/"}},"_tags":["comment","author_hypfer","story_50000488"],"author":"hypfer","children":[50020303,50013025,50011315],"comment_text":"Is it known why unsloth seems to not have touched DeepSeek 4.1 Flash?","created_at":"2026-10-08T20:00:45Z","created_at_i":1791489645,"objectID":"50011264","parent_id":50000488,"story_id":50000488,"story_title":"Why isn't the industry freaking out about DeepSeek 4.1 Flash?","story_url":"https://www.dgt.is/blog/2026-10-07-deepseek-freek-out/","updated_at":"2026-10-12T22:22:41Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"DiabloD3"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["unsloth"],"value":"This calculator is kinda useless. Different models work differently, you cannot generically define them by parameter count, and the models it does list seem to just be it prefilling the param counts for you.
It also misses most cards being used for inference, only limiting it to relatively recent Nvidia cards. Even if this was meant purely for datacenters, it would be useless for AMD customers.
And as for different models being different, it also doesn't understand kv cache size, how many concurrent sessions you need to manage, the compute cost overhead of kv cache quantization, nor the compute cost overhead of model quantization. It also doesn't know what MTP nor DFlash is, and cannot increase your effective tps to match.
As an example: compare Unsloth's quant of Qwen 3.8 Q4_K_M (4.5 BPW), a decent smaller model, without MTP, you'd get a baseline of, say, around 3-6 TPS per 100W. With the built in MTP model, you're closer to 6-12 TPS per 100W; then, switch quants to Byteshape's IQ4-XS (3.84 BPW) and use their suggested Dflash drafter, you're now in the realm of 15-30 TPS
... yet if I scribble into the calculator "27B, 27B active, 4-bit, 1B per one day", it seems to be the low end of my non-MTP figure: 1B per day = 11574 per second, it recommends 3x B200s, each B200 is 1200W, so 11574 / (1200*3) = 3.215, yet real world results would be 2x to 10x higher.
Also, quoting from the website, "A100 and V100 results for the large models are theoretical.". The small scale inference people absolutely know how these perform and it is well-enough documented.
The whole thing seems like AI slop."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"How many GPUs is 1M/B/T tokens?"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://cedana.com/resources/tokens-to-gpus/"}},"_tags":["comment","author_DiabloD3","story_49982096"],"author":"DiabloD3","children":[49997607],"comment_text":"This calculator is kinda useless. Different models work differently, you cannot generically define them by parameter count, and the models it does list seem to just be it prefilling the param counts for you.
It also misses most cards being used for inference, only limiting it to relatively recent Nvidia cards. Even if this was meant purely for datacenters, it would be useless for AMD customers.
And as for different models being different, it also doesn't understand kv cache size, how many concurrent sessions you need to manage, the compute cost overhead of kv cache quantization, nor the compute cost overhead of model quantization. It also doesn't know what MTP nor DFlash is, and cannot increase your effective tps to match.
As an example: compare Unsloth's quant of Qwen 3.8 Q4_K_M (4.5 BPW), a decent smaller model, without MTP, you'd get a baseline of, say, around 3-6 TPS per 100W. With the built in MTP model, you're closer to 6-12 TPS per 100W; then, switch quants to Byteshape's IQ4-XS (3.84 BPW) and use their suggested Dflash drafter, you're now in the realm of 15-30 TPS
... yet if I scribble into the calculator "27B, 27B active, 4-bit, 1B per one day", it seems to be the low end of my non-MTP figure: 1B per day = 11574 per second, it recommends 3x B200s, each B200 is 1200W, so 11574 / (1200*3) = 3.215, yet real world results would be 2x to 10x higher.
Also, quoting from the website, "A100 and V100 results for the large models are theoretical.". The small scale inference people absolutely know how these perform and it is well-enough documented.
The whole thing seems like AI slop.","created_at":"2026-10-07T15:59:35Z","created_at_i":1791388775,"objectID":"49994662","parent_id":49982096,"story_id":49982096,"story_title":"How many GPUs is 1M/B/T tokens?","story_url":"https://cedana.com/resources/tokens-to-gpus/","updated_at":"2026-10-08T08:23:40Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"throwa356262"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["unsloth"],"value":"The thing with 3.8 next is that it uses a variant of ngrams. Part of the network is replaced by a lookup table you can store on a fast ssd.
In practice, you will be able to run models a bit bigger than 35B.
https://unsloth.ai/docs/models/qwen3.8-next"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Mistral Large 4"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://mistral.ai/news/mistral-large-4/\\"}},"_tags":["comment","author_throwa356262","story_49977979"],"author":"throwa356262","children":[49982709],"comment_text":"The thing with 3.8 next is that it uses a variant of ngrams. Part of the network is replaced by a lookup table you can store on a fast ssd.
In practice, you will be able to run models a bit bigger than 35B.
https://unsloth.ai/docs/models/qwen3.8-next","created_at":"2026-10-06T17:53:12Z","created_at_i":1791309192,"objectID":"49981829","parent_id":49981265,"story_id":49977979,"story_title":"Mistral Large 4","story_url":"https://mistral.ai/news/mistral-large-4/\\","updated_at":"2026-10-06T21:38:49Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"walrus01"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["unsloth"],"value":"I have to concur with you here because I'm running 3.8-flash-next in the largest unsloth Q8 version on a 256GB system for daily use, and I've recently compared it head to head vs a Q4 that I have on a 128GB system and the Q4 is just plain not as good. The Q4 is given only more basic tasks.
I can't even imaging letting a Q2 anywhere near a code base that I care about. To generate a bird on a bicycle as a test? Sure. Actual things I don't want it to screw up? nooooooooo."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://github.com/Niko1221/Strata"}},"_tags":["comment","author_walrus01","story_49953495"],"author":"walrus01","children":[49964751],"comment_text":"I have to concur with you here because I'm running 3.8-flash-next in the largest unsloth Q8 version on a 256GB system for daily use, and I've recently compared it head to head vs a Q4 that I have on a 128GB system and the Q4 is just plain not as good. The Q4 is given only more basic tasks.
I can't even imaging letting a Q2 anywhere near a code base that I care about. To generate a bird on a bicycle as a test? Sure. Actual things I don't want it to screw up? nooooooooo.","created_at":"2026-10-05T12:27:33Z","created_at_i":1791203253,"objectID":"49963963","parent_id":49955565,"story_id":49953495,"story_title":"Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s","story_url":"https://github.com/Niko1221/Strata","updated_at":"2026-10-05T13:39:58Z"}],"hitsPerPage":20,"nbHits":1332,"nbPages":50,"page":0,"params":"query=unsloth&advancedSyntax=true&analyticsTags=backend","processingTimeMS":8,"processingTimingsMS":{"_request":{"queue":1,"roundTrip":22},"afterFetch":{"format":{"highlighting":2,"total":2}},"fetch":{"query":6,"total":7},"total":8},"query":"unsloth","serverTimeMS":12}