Bump it to native (or -c 0 may work too)","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T23:20:02.000Z","created_at_i":1775431202,"id":47654953,"options":[],"parent_id":47654177,"points":null,"story_id":47651540,"text":"Even tried gemma4:31b and gemma4:31b with 128k context (I have 72GiB VRAM). Nothing. I'm cursed I guess. That's ollama-rocm if that matters (I had weird bugs on Vulkan, maybe gemma misbehaves on radeons somehow?..).
UPD: tried ollama-vulkan. It works, gemma4:31b-it-q8_0 with 64k context!","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T21:42:27.000Z","created_at_i":1775425347,"id":47654177,"options":[],"parent_id":47653802,"points":null,"story_id":47651540,"text":"Since that defaults to the q4 variant, try the q8 one:
ollama launch claude --model gemma4:26b-a4b-it-q8_0 OLLAMA_CONTEXT_LENGTH=64000 ollama serve\nCodex also works:
ollama launch codex --model gemma4:26b ollama launch claude --model gemma4:26bhttps://github.com/JuliusBrussee/caveman","title":null,"type":"comment","url":null},{"author":"unsnap_biceps","children":[{"author":"jaggederest","children":[{"author":"blitzar","children":[],"created_at":"2026-04-05T22:10:34.000Z","created_at_i":1775427034,"id":47654461,"options":[],"parent_id":47653924,"points":null,"story_id":47651540,"text":"I am hopeful the leaked claude code narrows the capability, perhaps even googles offering will be viable once they borrow some ideas from claude.","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T21:10:03.000Z","created_at_i":1775423403,"id":47653924,"options":[],"parent_id":47653258,"points":null,"story_id":47651540,"text":"If you want to experiment with same-harness-different-models Opencode is classically the one to use. After their recent kerfluffle with Anthropic you'll have to use API pricing for opus/sonnet/haiku which makes it kind of a non-starter, but it lets you swap out any number of cloud or local models using e.g. ollama or z.ai or whatever backend provider you like.
I'd rate their coding agent harness as slightly to significantly less capable than claude code, but it also plays better with alternate models.","title":null,"type":"comment","url":null},{"author":"satvikpendem","children":[],"created_at":"2026-04-06T01:48:42.000Z","created_at_i":1775440122,"id":47656038,"options":[],"parent_id":47653258,"points":null,"story_id":47651540,"text":"OpenCode","title":null,"type":"comment","url":null},{"author":"andhuman","children":[],"created_at":"2026-04-06T06:57:50.000Z","created_at_i":1775458670,"id":47657738,"options":[],"parent_id":47653258,"points":null,"story_id":47651540,"text":"I have good experience with Mistral Vibe.","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T19:58:59.000Z","created_at_i":1775419139,"id":47653258,"options":[],"parent_id":47653082,"points":null,"story_id":47651540,"text":"I use CC at work, so I haven't explored other options. Is there a better one to use locally? I presumed they were all going to be pretty similar.","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T19:41:56.000Z","created_at_i":1775418116,"id":47653082,"options":[],"parent_id":47652904,"points":null,"story_id":47651540,"text":"Claude Code is fairly notoriously token inefficient as far as coding agent/harnesses go (i come from aider pre-CC). It's only viable because the Max subscriptions give you approximately unlimited token budget, which resets in a few hours even if you hit the limit. But this also only works because cloud models have massive token windows (1M tokens on opus right now) which is a bit difficult to make happen locally with the VRAM needed.
And if you somehow managed to open up a big enough VRAM playground, the open weights models are not quite as good at wrangling such large context windows (even opus is hardly capable) without basically getting confused about what they were doing before they finish parsing it.","title":null,"type":"comment","url":null},{"author":"mbesto","children":[{"author":"bdangubic","children":[{"author":"NamlchakKhandro","children":[],"created_at":"2026-04-06T00:21:18.000Z","created_at_i":1775434878,"id":47655412,"options":[],"parent_id":47654245,"points":null,"story_id":47651540,"text":"No it's more like, why use a Microsoft paid for distro of nvim when lazyvim, astronvim exist","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T21:50:04.000Z","created_at_i":1775425804,"id":47654245,"options":[],"parent_id":47654161,"points":null,"story_id":47651540,"text":"this is like asking why use intellij or vscode or \u2026 when there is vim and emacs","title":null,"type":"comment","url":null},{"author":"blitzar","children":[],"created_at":"2026-04-05T22:07:24.000Z","created_at_i":1775426844,"id":47654428,"options":[],"parent_id":47654161,"points":null,"story_id":47651540,"text":"previously I have found claude code to be just better than the alternatives, using large models or local. It is, however, closer now and not much excuse for the competition after the claude code leak. Personally, I will be giving this a go with OpenCode.","title":null,"type":"comment","url":null},{"author":"superb_dev","children":[],"created_at":"2026-04-05T22:49:29.000Z","created_at_i":1775429369,"id":47654734,"options":[],"parent_id":47654161,"points":null,"story_id":47651540,"text":"From my experience, Claude Code is just better. Although I recently started using Zed and it\u2019s pretty good","title":null,"type":"comment","url":null},{"author":"panagathon","children":[],"created_at":"2026-04-06T03:25:22.000Z","created_at_i":1775445922,"id":47656638,"options":[],"parent_id":47654161,"points":null,"story_id":47651540,"text":"> I don't get why I would use Claude Code when OpenCode, Cursor, Zed, etc. all exist, are "free" and work with virtually any llm. Seems like a weird use case unless I'm missing something.
I'm with you on this. I've tried Gemma and Claude code and it's not good. Forgets it can use bash!
However, Gemma running locally with Pi as the harness is a beast.","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T21:41:01.000Z","created_at_i":1775425261,"id":47654161,"options":[],"parent_id":47652904,"points":null,"story_id":47651540,"text":"I don't get why I would use Claude Code when OpenCode, Cursor, Zed, etc. all exist, are "free" and work with virtually any llm. Seems like a weird use case unless I'm missing something.","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T19:21:50.000Z","created_at_i":1775416910,"id":47652904,"options":[],"parent_id":47652866,"points":null,"story_id":47651540,"text":"lm studio offers an Anthropic compatible local endpoint, so you can point Claude code at it and it'll use your local model for it's requests, however, I've had a lot of problems with LM Studio and Claude code losing it's place. It'll think for awhile, come up with a plan, start to do it and then just halt in the middle. I'll ask it to continue and it'll do a small change and get stuck again.
Using ollama's api doesn't have the same issue, so I've stuck to using ollama for local development work.","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T19:17:48.000Z","created_at_i":1775416668,"id":47652866,"options":[],"parent_id":47651540,"points":null,"story_id":47651540,"text":"So wait what is the interaction between Gemma and Claude?","title":null,"type":"comment","url":null},{"author":"Someone1234","children":[{"author":"moomin","children":[{"author":"phainopepla2","children":[{"author":"quinnjh","children":[{"author":"kenmacd","children":[],"created_at":"2026-04-05T23:45:46.000Z","created_at_i":1775432746,"id":47655159,"options":[],"parent_id":47653513,"points":null,"story_id":47651540,"text":"not to use the cli tool. You can install it and change the settings to point to pretty much any other model.
It's an okay-enough tool, but I don't see a lot of point in using it when open sources tools like Pi and OpenCode exist (or octofriend, or forge, or droid, etc).","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T20:23:54.000Z","created_at_i":1775420634,"id":47653513,"options":[],"parent_id":47653166,"points":null,"story_id":47651540,"text":"so no subscription is needed?","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T19:49:38.000Z","created_at_i":1775418578,"id":47653166,"options":[],"parent_id":47653097,"points":null,"story_id":47651540,"text":"You don't pay anything to use Claude Code as a front end to non-Anthropic models","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T19:43:06.000Z","created_at_i":1775418186,"id":47653097,"options":[],"parent_id":47652945,"points":null,"story_id":47651540,"text":"Right now it suits them down to the ground. You pay for the product and you don\u2019t cost their servers anything.","title":null,"type":"comment","url":null},{"author":"chvid","children":[],"created_at":"2026-04-05T20:06:51.000Z","created_at_i":1775419611,"id":47653341,"options":[],"parent_id":47652945,"points":null,"story_id":47651540,"text":"Is it not about the same as using OpenCode?
And is running a local model with Claude Code actually usable for any practical work compared to the hosted Anthropic models?","title":null,"type":"comment","url":null},{"author":"wyre","children":[],"created_at":"2026-04-05T20:26:07.000Z","created_at_i":1775420767,"id":47653532,"options":[],"parent_id":47652945,"points":null,"story_id":47651540,"text":"I think CC is popular because they are catering to the common denominator programmer and are going to continue to do that, not because CC is particularly turn-key.","title":null,"type":"comment","url":null},{"author":"nerdix","children":[{"author":"deskamess","children":[],"created_at":"2026-04-05T23:43:33.000Z","created_at_i":1775432613,"id":47655140,"options":[],"parent_id":47653743,"points":null,"story_id":47651540,"text":"Are the Claude Code (desktop) models very different from what Bedrock has? I thought you could hook up VSCode (not Claude Desktop) to Bedrock Anthropic models. Are there features in Claude Desktop that are not in VSCode/cli?","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T20:50:47.000Z","created_at_i":1775422247,"id":47653743,"options":[],"parent_id":47652945,"points":null,"story_id":47651540,"text":"I don't think there is any incentive to do so right now because the open models aren't as good. The vast majority of businesses are going to just pay the extra cost for access to a frontier model. The model is what gives them a competitive advantage, not the harness. The harness is a lot easier to replicate than Opus.
There are benefits too. Some developers might learn to use Claude Code outside of work with cheaper models and then advocate for using Claude Code at work (where their companies will just buy access from Anthropic, Bedrock, etc). Similar to how free ESXi licenses for personal use helped infrastructure folks gain skills with that product which created a healthy supply of labor and VMware evangelists that were eager to spread the gospel. Anthropic can't just give away access to Claude models because of cost so there is use in allowing alternative ways for developers to learn how to use Claude Code and develop a workflow with it.","title":null,"type":"comment","url":null},{"author":"falcor84","children":[],"created_at":"2026-04-05T22:03:28.000Z","created_at_i":1775426608,"id":47654383,"options":[],"parent_id":47652945,"points":null,"story_id":47651540,"text":"Well, if they did, it would probably be shooting themselves in the foot, seeing that the Claude Code source is out there now, and people are waiting for an excuse to "clean-room" reimplement and fork it","title":null,"type":"comment","url":null},{"author":"alfiedotwtf","children":[],"created_at":"2026-04-06T08:06:22.000Z","created_at_i":1775462782,"id":47658166,"options":[],"parent_id":47652945,"points":null,"story_id":47651540,"text":"Yet Codex specifically aims out to be compatible with all backends! Up until Gemma 4 though it\u2019s been pretty solid, but totally fails with unknown tool (I\u2019m guessing a template issue)","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T19:25:52.000Z","created_at_i":1775417152,"id":47652945,"options":[],"parent_id":47651540,"points":null,"story_id":47651540,"text":"Using Claude Code seems like a popular frontend currently, I wonder how long until Anthropic releases an update to make it a little to a lot less turn-key? They've been very clear that they aren't exactly champions of this stuff being used outside of very specific ways.","title":null,"type":"comment","url":null},{"author":"martinald","children":[{"author":"IceWreck","children":[{"author":"Yukonv","children":[{"author":"zozbot234","children":[],"created_at":"2026-04-05T22:39:55.000Z","created_at_i":1775428795,"id":47654683,"options":[],"parent_id":47654135,"points":null,"story_id":47651540,"text":"10 tok/s is quite fine for chatting, though less so for interaction with agentic workloads. So the technique itself is still worthwhile for running a huge model locally.","title":null,"type":"comment","url":null},{"author":"IceWreck","children":[{"author":"zozbot234","children":[{"author":"IceWreck","children":[],"created_at":"2026-04-06T21:10:34.000Z","created_at_i":1775509834,"id":47667166,"options":[],"parent_id":47654826,"points":null,"story_id":47651540,"text":"Yep, I understand I was giving an example to the person I was replying to.","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T23:03:13.000Z","created_at_i":1775430193,"id":47654826,"options":[],"parent_id":47654715,"points":null,"story_id":47651540,"text":"Your 120B model likely has way more active parameters, so it can probably only fit a few shared layers in the VRAM for your dGPU. You might be better off running that model on a unified memory platform, slower VRAM but a lot more of it.","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T22:45:05.000Z","created_at_i":1775429105,"id":47654715,"options":[],"parent_id":47654135,"points":null,"story_id":47651540,"text":"> So while feasible it's only great for batch jobs not interactive usage.
I mean yeah true but depends on how big the model is. The example I gave (Qwen 3.5 35BA3B) was fitting a 35B Q4 K_M (say 20 GB in size) model in 12 GB VRAM. With a 4070Ti + high speed 32 GB DDR5 ram you can easily get 700 token/sec prompt processing and 55-60 token/sec generation which is quite fast.
On the other hand if I try to fit a 120B model in 96 GB of DDR5 + the same 12 GB VRAM I get 2-5 token/sec generation.","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T21:38:57.000Z","created_at_i":1775425137,"id":47654135,"options":[],"parent_id":47653312,"points":null,"story_id":47651540,"text":"With that you are taking a significant performance penalty and become severely I/O bottlenecked. I've been able to stream Qwen3.5-397B-A17B from my M5 Max (12 GB/s SSD Read) using the Flash MoE technique at the brisk pace of 10 tokens per second. As tokens are generated different experts need to be consulted resulting in a lot of I/O churn. So while feasible it's only great for batch jobs not interactive usage.","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T20:03:28.000Z","created_at_i":1775419408,"id":47653312,"options":[],"parent_id":47653114,"points":null,"story_id":47651540,"text":"It does if you use an inference engine where you can offload some of the experts from VRAM to CPU RAM.\nThat means I can fit a 35 billion param MoE in let's say 12 GB VRAM GPU + 16 gigs of memory.","title":null,"type":"comment","url":null},{"author":"charcircuit","children":[{"author":"martinald","children":[{"author":"zozbot234","children":[{"author":"mikkupikku","children":[{"author":"the_pwner224","children":[],"created_at":"2026-04-06T11:26:29.000Z","created_at_i":1775474789,"id":47659500,"options":[],"parent_id":47659276,"points":null,"story_id":47651540,"text":"AMD Strix Halo. Available in the Framework desktop, various mini PCs, and the Asus Rog Flow Z13 "gaming tablet." The Z13 is still at $2700 for 128 GB which is an incredible deal with today's RAM prices.
There's also the Nvidia DGX Spark.","title":null,"type":"comment","url":null}],"created_at":"2026-04-06T10:59:02.000Z","created_at_i":1775473142,"id":47659276,"options":[],"parent_id":47654709,"points":null,"story_id":47651540,"text":"In practical terms, is this kind of architecture available to consumers except through Apple?","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T22:44:00.000Z","created_at_i":1775429040,"id":47654709,"options":[],"parent_id":47654448,"points":null,"story_id":47651540,"text":"With unified memory, reading from RAM to GPU compute buffer is not that painful, and you can use partial RAM caching to minimize the impact of other kinds of swapping.","title":null,"type":"comment","url":null},{"author":"charcircuit","children":[],"created_at":"2026-04-06T00:30:17.000Z","created_at_i":1775435417,"id":47655474,"options":[],"parent_id":47654448,"points":null,"story_id":47651540,"text":"You don't have to only have the experts being actively used in VRAM. You can load as many weights as will fit. If there is a "cache miss" you have to pay the price to swap in the weights, but if there is a hit you don't.","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T22:09:35.000Z","created_at_i":1775426975,"id":47654448,"options":[],"parent_id":47653791,"points":null,"story_id":47651540,"text":"Yes you're right technically, but in reality you'd be swapping them the (vast?) majority in and out per inference request so would create an enormous bottleneck for the use case the author is using for.","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T20:57:20.000Z","created_at_i":1775422640,"id":47653791,"options":[],"parent_id":47653114,"points":null,"story_id":47651540,"text":"You never need to have all weights in memory. You can swap them in from RAM, disk, the network, etc. MOE reduces the amount of data that will need to be swapped in for the next forward pass.","title":null,"type":"comment","url":null},{"author":"functional_dev","children":[],"created_at":"2026-04-06T16:17:39.000Z","created_at_i":1775492259,"id":47662883,"options":[],"parent_id":47653114,"points":null,"story_id":47651540,"text":"This confused me at first as well.. inactive experts skip compute, but weights are sill loaded. So memory does not shrink at all.
I found this visualisation helpful - https://vectree.io/c/sparse-activation-patterns-and-memory-e...","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T19:44:45.000Z","created_at_i":1775418285,"id":47653114,"options":[],"parent_id":47651540,"points":null,"story_id":47651540,"text":"Just FYI, MoE doesn't really save (V)RAM. You still need all weights loaded in memory, it just means you consult less per forward pass. So it improves tok/s but not vram usage.","title":null,"type":"comment","url":null},{"author":"asymmetric","children":[{"author":"pshirshov","children":[{"author":"nl","children":[{"author":"pshirshov","children":[],"created_at":"2026-04-06T11:16:54.000Z","created_at_i":1775474214,"id":47659414,"options":[],"parent_id":47656910,"points":null,"story_id":47651540,"text":"That's discrete DDR5, it's not as fast as your regular VRAM.","title":null,"type":"comment","url":null},{"author":"pshirshov","children":[],"created_at":"2026-04-06T13:39:41.000Z","created_at_i":1775482781,"id":47660791,"options":[],"parent_id":47656910,"points":null,"story_id":47651540,"text":"Ah, forgot to add, it's not really "unified" you have to explicitly specify your allocations. You may have a reasonably good 48gb chunk assigned to the GPU, but that DDR5 is 5-10 times slower than GDDR/HBM and the GPU itself isn't stellar.
So, framework laptops are great for chatting but nearly useless in agentic coding.
My Radeon W7900 answers a question ("what is this project") in 2 minutes, it takes my Framework 16 with 5070 addon around 11 minutes without the addon - around 23 (qwen 3.5 27b, claude code)","title":null,"type":"comment","url":null}],"created_at":"2026-04-06T04:11:43.000Z","created_at_i":1775448703,"id":47656910,"options":[],"parent_id":47655018,"points":null,"story_id":47651540,"text":"Doesn't the framework desktop have a Ryzen 395 AI? That's a unified memory architecture like the Macs.","title":null,"type":"comment","url":null},{"author":"ac29","children":[],"created_at":"2026-04-06T06:35:27.000Z","created_at_i":1775457327,"id":47657640,"options":[],"parent_id":47655018,"points":null,"story_id":47651540,"text":"This article is about a MoE model with only 4B active parameters, it shouldn't take 10 minutes to answer a question about a small project.
I measured a 4bit quant of this model at 1300t/s prefill and ~60t/s decode on Ryzen 395+.","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T23:29:26.000Z","created_at_i":1775431766,"id":47655018,"options":[],"parent_id":47654287,"points":null,"story_id":47651540,"text":"Only for chat sessions, not for agentic coding. It's just too slow to be practical (10 minutes to answer a simple question about a 2k LoC project - and that's with a 5070 addon card).","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T21:55:10.000Z","created_at_i":1775426110,"id":47654287,"options":[],"parent_id":47651540,"points":null,"story_id":47651540,"text":"Is a framework desktop with >48GB of RAM a good machine to try this out?","title":null,"type":"comment","url":null},{"author":"aetherspawn","children":[],"created_at":"2026-04-06T00:00:21.000Z","created_at_i":1775433621,"id":47655275,"options":[],"parent_id":47651540,"points":null,"story_id":47651540,"text":"Can you use the smaller Gemma 4B model as speculative decoding for the larger 31B model?
Why/why not?","title":null,"type":"comment","url":null},{"author":"NamlchakKhandro","children":[{"author":"loveparade","children":[{"author":"tarruda","children":[{"author":"loveparade","children":[],"created_at":"2026-04-06T00:45:28.000Z","created_at_i":1775436328,"id":47655582,"options":[],"parent_id":47655510,"points":null,"story_id":47651540,"text":"Interesting, I don't like codex exactly because of its built-in sandboxing. If I need a sandbox I rather do a simple bwrap myself around the agent process, I prefer that over the agent cli doing a bunch of sandboxing magic that gets in my way.","title":null,"type":"comment","url":null},{"author":"prettyblocks","children":[],"created_at":"2026-04-06T01:50:52.000Z","created_at_i":1775440252,"id":47656056,"options":[],"parent_id":47655510,"points":null,"story_id":47651540,"text":"how is codex sandbox different from /sandbox on claude code?","title":null,"type":"comment","url":null}],"created_at":"2026-04-06T00:34:51.000Z","created_at_i":1775435691,"id":47655510,"options":[],"parent_id":47655414,"points":null,"story_id":47651540,"text":"Codex is the best out-of-box experience, especially due to its builtin sandboxing. Only drawback is that its edit tool requires the LLM to output a diff which only GPTs are trained to do correctly.","title":null,"type":"comment","url":null}],"created_at":"2026-04-06T00:21:37.000Z","created_at_i":1775434897,"id":47655414,"options":[],"parent_id":47655402,"points":null,"story_id":47651540,"text":"What do you recommend? I've tried both pi and opencode and both are better than claude imo, but I wonder if there are others.","title":null,"type":"comment","url":null},{"author":"z0mghii","children":[{"author":"threethirtytwo","children":[],"created_at":"2026-04-06T09:07:02.000Z","created_at_i":1775466422,"id":47658509,"options":[],"parent_id":47655416,"points":null,"story_id":47651540,"text":"it has visual artifacts when inferencing.","title":null,"type":"comment","url":null}],"created_at":"2026-04-06T00:21:52.000Z","created_at_i":1775434912,"id":47655416,"options":[],"parent_id":47655402,"points":null,"story_id":47651540,"text":"Can you elaborate what is jank about it?","title":null,"type":"comment","url":null},{"author":"dimgl","children":[],"created_at":"2026-04-06T00:25:48.000Z","created_at_i":1775435148,"id":47655440,"options":[],"parent_id":47655402,"points":null,"story_id":47651540,"text":"Vagueposting in Hacker News?","title":null,"type":"comment","url":null}],"created_at":"2026-04-06T00:19:11.000Z","created_at_i":1775434751,"id":47655402,"options":[],"parent_id":47651540,"points":null,"story_id":47651540,"text":"I don't know why people bother with Claude code.
It's so jank, there are far superior cli coding harness out there","title":null,"type":"comment","url":null},{"author":"inzlab","children":[],"created_at":"2026-04-06T00:27:01.000Z","created_at_i":1775435221,"id":47655451,"options":[],"parent_id":47651540,"points":null,"story_id":47651540,"text":"awesome, the lighter the hardware running big softwares the more novelty.","title":null,"type":"comment","url":null},{"author":"seifbenayed1992","children":[{"author":"SeriousM","children":[{"author":"seifbenayed1992","children":[],"created_at":"2026-04-06T17:24:07.000Z","created_at_i":1775496247,"id":47663983,"options":[],"parent_id":47658780,"points":null,"story_id":47651540,"text":"pi-mono is a great toolkit \u2014 coding agent CLI, unified LLM API, web UI, Slack bot, vLLM pods.
cloclo is a runtime for agent toolkits. You plug it into your own agents and it gives them multi-agent orchestration (AICL protocol), 13 providers, skill registry, native browser/docs/phone tools, memory, and an NDJSON bridge. Zero native deps.","title":null,"type":"comment","url":null}],"created_at":"2026-04-06T09:46:25.000Z","created_at_i":1775468785,"id":47658780,"options":[],"parent_id":47658062,"points":null,"story_id":47651540,"text":"How does cloclo differ from pi-mono?","title":null,"type":"comment","url":null}],"created_at":"2026-04-06T07:50:47.000Z","created_at_i":1775461847,"id":47658062,"options":[],"parent_id":47651540,"points":null,"story_id":47651540,"text":"Local models are finally starting to feel pleasant instead of just "possible." The headless LM Studio flow is especially nice because it makes local inference usable from real tools instead of as a demo.
Related note from someone building in this space: I've been working on cloclo (https://www.npmjs.com/package/cloclo), an open-source coding agent CLI, and this is exactly the direction I'm excited about. It natively supports LM Studio, Ollama, vLLM, Jan, and llama.cpp as providers alongside cloud models, so you can swap between local and hosted backends without changing how you work.
Feels like we're getting closer to a good default setup where local models are private/cheap enough to use daily, and cloud models are still there when you need the extra capability.","title":null,"type":"comment","url":null},{"author":"janalsncm","children":[],"created_at":"2026-04-06T08:04:20.000Z","created_at_i":1775462660,"id":47658152,"options":[],"parent_id":47651540,"points":null,"story_id":47651540,"text":"Qwen3-coder has been better for coding in my experience and has similar sizes. Either way, after a bunch of frustration with the quality and price of CC lately I\u2019m happy there are local options.","title":null,"type":"comment","url":null},{"author":"AbuAssar","children":[],"created_at":"2026-04-06T08:43:25.000Z","created_at_i":1775465005,"id":47658351,"options":[],"parent_id":47651540,"points":null,"story_id":47651540,"text":"omlx gives better performance than ollama on apple silicon","title":null,"type":"comment","url":null},{"author":"smcleod","children":[],"created_at":"2026-04-06T10:33:00.000Z","created_at_i":1775471580,"id":47659108,"options":[],"parent_id":47651540,"points":null,"story_id":47651540,"text":"Did you try the MLX model instead? In general MLX tends provide much better performance than GGUF/Llama.cpp on macOS.","title":null,"type":"comment","url":null},{"author":"d4rkp4ttern","children":[{"author":"peder","children":[{"author":"selectodude","children":[{"author":"vlowther","children":[],"created_at":"2026-04-06T17:16:37.000Z","created_at_i":1775495797,"id":47663852,"options":[],"parent_id":47661697,"points":null,"story_id":47651540,"text":"Same. Opencode + oMLX (0.3.4) + unsloth-Qwen3-Coder-Next-mlx-8bit on my M5 Max w 128GB is the sweet spot for me locally. The prompt decode caching keeps things coherent and fast even when contexts get north of 100k tokens.","title":null,"type":"comment","url":null},{"author":"peder","children":[{"author":"d4rkp4ttern","children":[],"created_at":"2026-04-06T23:39:15.000Z","created_at_i":1775518755,"id":47668860,"options":[],"parent_id":47667020,"points":null,"story_id":47651540,"text":"you can set it in the .omlx/settings.json - ask a code-agent to figure it out by pointing it at the omlx repo","title":null,"type":"comment","url":null}],"created_at":"2026-04-06T21:00:29.000Z","created_at_i":1775509229,"id":47667020,"options":[],"parent_id":47661697,"points":null,"story_id":47651540,"text":"Have you been using `omlx serve`? If so, how are you bumping up the max context size? I'm not seeing a param to go above 32k?","title":null,"type":"comment","url":null},{"author":"d4rkp4ttern","children":[],"created_at":"2026-04-06T23:37:46.000Z","created_at_i":1775518666,"id":47668849,"options":[],"parent_id":47661697,"points":null,"story_id":47651540,"text":"At least for the Gemma4-26B-A4B, Token-gen speed with OMLX is far worse on my M1 Max 64GB Macbook, compared to llama-server:
Quick benchmark on M1 Max 64GB, Gemma 4 26B-A4B (MoE), comparing matched dynamic 4-bit quants. Workload\n was Claude Code, which sends ~35K tokens of input context per request (system prompt + tools + user\n message):\n\n llama.cpp (unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL, llama-server -fa on -c 131072 --jinja --temp 1.0\n --top-p 0.95 --top-k 64):\n - pp \u2248 395 tok/s\n - tg \u2248 40 tok/s\n\n oMLX (unsloth/gemma-4-26b-a4b-it-UD-MLX-4bit, omlx serve --model-dir ~/models/omlx, with\n sampling.max_context_window and max_tokens bumped to 131072 in ~/.omlx/settings.json):\n - pp \u2248 350 tok/s\n - tg \u2248 5\u201313 tok/s\n\n Same model family and quant tier. Prompt processing is comparable, but oMLX's token generation is 3\u20137x\n slower than llama.cpp's Metal backend. Counter-intuitive given MLX is Apple's native ML framework.https://pchalasani.github.io/claude-code-tools/integrations/...
The 26BA4B is the most interesting to run on such hardware, and I get nearly double the token-gen speed (40 tok/s) compared to Qwen3.5 35BA3B. However the tau2 bench results[1] for this Gemma4 variant lag far behind the Qwen variant (68% vs 81%), so I don\u2019t expect the former to do well on heavy agentic tool-heavy tasks:
[1] https://news.ycombinator.com/item?id=47616761","title":null,"type":"comment","url":null},{"author":"tiku","children":[],"created_at":"2026-04-06T14:13:37.000Z","created_at_i":1775484817,"id":47661204,"options":[],"parent_id":47651540,"points":null,"story_id":47651540,"text":"I hate that my M5 with 24 gb has so much trouble with these models. Not getting any good speeds, even with simple models.","title":null,"type":"comment","url":null},{"author":"Imanari","children":[],"created_at":"2026-04-06T14:18:31.000Z","created_at_i":1775485111,"id":47661254,"options":[],"parent_id":47651540,"points":null,"story_id":47651540,"text":"How well do the Gemma 4 models perform on agentic coding? What are your impressions?","title":null,"type":"comment","url":null},{"author":"ttul","children":[{"author":"dominotw","children":[{"author":"ttul","children":[],"created_at":"2026-04-06T16:09:47.000Z","created_at_i":1775491787,"id":47662782,"options":[],"parent_id":47661577,"points":null,"story_id":47651540,"text":"I don't think so. Acquiring hardware for inference is a chokepoint on growth. If they can offload some inference to the customer's machine, that allows them to use more of their online capacity to generate money.","title":null,"type":"comment","url":null}],"created_at":"2026-04-06T14:42:03.000Z","created_at_i":1775486523,"id":47661577,"options":[],"parent_id":47661421,"points":null,"story_id":47651540,"text":"wouldnt that be counter to their whole business model?","title":null,"type":"comment","url":null}],"created_at":"2026-04-06T14:29:54.000Z","created_at_i":1775485794,"id":47661421,"options":[],"parent_id":47651540,"points":null,"story_id":47651540,"text":"I could see a future in which the major AI labs run a local LLM to offload much of the computational effort currently undertaken in the cloud, leaving the heavy lifting to cloud-hosted models and the easier stuff for local inference.","title":null,"type":"comment","url":null},{"author":"jedisct1","children":[],"created_at":"2026-04-06T15:21:56.000Z","created_at_i":1775488916,"id":47662103,"options":[],"parent_id":47651540,"points":null,"story_id":47651540,"text":"Running Gemma 4 with llama.cpp and Swival:
$ llama-server\n --reasoning auto\n --fit on\n -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL\n --temp 1.0 --top-p 0.95 --top-k 64
$ uvx swival --provider llamacpp
Done.","title":null,"type":"comment","url":null},{"author":"drob518","children":[{"author":"robot_jesus","children":[{"author":"drob518","children":[],"created_at":"2026-04-06T19:28:59.000Z","created_at_i":1775503739,"id":47665728,"options":[],"parent_id":47665164,"points":null,"story_id":47651540,"text":"You\u2019re right about inference speed being a concern. I was assuming it\u2019s a small model but even then, one of the browser automation frameworks is going to be faster.","title":null,"type":"comment","url":null}],"created_at":"2026-04-06T18:50:25.000Z","created_at_i":1775501425,"id":47665164,"options":[],"parent_id":47662141,"points":null,"story_id":47651540,"text":"Yeah. I think that's an interesting use case. Especially if I can kick it off or schedule it when I'm not actively working. Inference speed (especially with tool calling involved) won't be great on my machines, but if I schedule nightly usability tests of dev sites while I sleep, that could be really cool.","title":null,"type":"comment","url":null}],"created_at":"2026-04-06T15:24:38.000Z","created_at_i":1775489078,"id":47662141,"options":[],"parent_id":47651540,"points":null,"story_id":47651540,"text":"Seems like this might be a great way to do web software testing. We\u2019ve had Selenium and Puppeteer for a long time but they are a bit brittle with respect to the web design. Change something about the design and there\u2019s a high likelihood that a test will break. Seems like this might be able to be smarter about adapting to changes. That\u2019s also a great use for a smaller model like this.","title":null,"type":"comment","url":null},{"author":"pseudosavant","children":[{"author":"ffsm8","children":[],"created_at":"2026-04-07T15:04:46.000Z","created_at_i":1775574286,"id":47676524,"options":[],"parent_id":47669808,"points":null,"story_id":47651540,"text":"Fwiw, the real reason we don't have 100+ GB GPUs is because Nvidia likes to segment their markets. They could sell the consumer cards with 200gb gddr RAM on it, they just know that'd eat into their enterprise offering which is quiet literally all their profit margin (which I may add is gargantuan as of 2025)","title":null,"type":"comment","url":null}],"created_at":"2026-04-07T01:53:16.000Z","created_at_i":1775526796,"id":47669808,"options":[],"parent_id":47651540,"points":null,"story_id":47651540,"text":"I want local models to succeed, but today the gap vs cloud models still seems continually too large. Even with a $2k GPU or a $4k MBP, the quality and speed tradeoff usually isn\u2019t sensible.
Credit to Google for releasing Gemma 4, though. I\u2019d love to see local models reach the point where a 32 GB machine can handle high quality agentic coding at a practical speed.","title":null,"type":"comment","url":null},{"author":"alfiedotwtf","children":[],"created_at":"2026-04-07T16:05:37.000Z","created_at_i":1775577937,"id":47677440,"options":[],"parent_id":47651540,"points":null,"story_id":47651540,"text":"PSA: For those getting stuck in a repetitive loop or just stopping without completing a task, try the interactive template. I just tried it now and it's blowing my already impressive results out of the water (llama.cpp):
--jinja --chat-template-file models/templates/google-gemma-4-31B-it-interleaved.jinjaCurious if anyone else is exploring per-request model execution rather than keeping models resident all the time.","title":null,"type":"comment","url":null}],"created_at":"2026-04-05T17:13:51.000Z","created_at_i":1775409231,"id":47651540,"options":[],"parent_id":null,"points":407,"story_id":47651540,"text":null,"title":"Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code","type":"story","url":"https://ai.georgeliu.com/p/running-google-gemma-4-locally-with"}