The interesting part was the quant format choice. NVFP4 is Blackwell\u2019s native 4-bit format and theoretically the fastest path, but MoE support for Gemma 4 specifically was blocked on an unmerged vLLM PR (#39045) \u2014 linear layers loaded, expert weights didn\u2019t. Falling back to nightly didn\u2019t help because that day\u2019s nightly was broken by someone landing an unconditional pandas import in the AITER code path without updating the image\u2019s deps. Ended up on AWQ + Marlin kernels, which has been stable in vLLM for over a year. For single-user memory-bandwidth-bound decode the gap to NVFP4 is smaller than you\u2019d expect \u2014 both hit the same 4x weight compression, and AWQ dequantizes to FP16 in-register rather than using FP4 tensor cores. I\u2019m getting ~196 tok/s; I\u2019d estimate NVFP4 would be 220-240 if it had worked.
Happy to dig into the vLLM config, the RunPod Serverless side, or the NVFP4 vs AWQ tradeoff in more depth."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Deploying Gemma 4 26B on an RTX 5090"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://datapnt.com/blog/deploying-gemma-4-26b-a4b-on-rtx-5090"}},"_tags":["comment","author_sudo_ls_ads","story_47824986"],"author":"sudo_ls_ads","comment_text":"Author here. Quick context on what made this worth writing up: Gemma 4 26B A4B is an MoE \u2014 26B total params, 4B active per token \u2014 which fundamentally changes what\u2019s viable on a single consumer GPU. During decode you pay the memory bandwidth cost of a 4B model but get the quality of a 26B. That\u2019s what makes a 5090 a real option for it; a dense 26B wouldn\u2019t be.
The interesting part was the quant format choice. NVFP4 is Blackwell\u2019s native 4-bit format and theoretically the fastest path, but MoE support for Gemma 4 specifically was blocked on an unmerged vLLM PR (#39045) \u2014 linear layers loaded, expert weights didn\u2019t. Falling back to nightly didn\u2019t help because that day\u2019s nightly was broken by someone landing an unconditional pandas import in the AITER code path without updating the image\u2019s deps. Ended up on AWQ + Marlin kernels, which has been stable in vLLM for over a year. For single-user memory-bandwidth-bound decode the gap to NVFP4 is smaller than you\u2019d expect \u2014 both hit the same 4x weight compression, and AWQ dequantizes to FP16 in-register rather than using FP4 tensor cores. I\u2019m getting ~196 tok/s; I\u2019d estimate NVFP4 would be 220-240 if it had worked.
Happy to dig into the vLLM config, the RunPod Serverless side, or the NVFP4 vs AWQ tradeoff in more depth.","created_at":"2026-04-19T15:29:13Z","created_at_i":1776612553,"objectID":"47825012","parent_id":47824986,"story_id":47824986,"story_title":"Deploying Gemma 4 26B on an RTX 5090","story_url":"https://datapnt.com/blog/deploying-gemma-4-26b-a4b-on-rtx-5090","updated_at":"2026-04-20T03:16:28Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"benreesman"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["fp4","nvfp4","blackwell"],"value":"If Moore's Law is Ending leaks are to be believed, there are going to be 24GB GDDR7 5080 Super and maybe even 5070 Super Ti variants in the 1k (MSRP) range and one assumes fast Blackwell NVFP4 Tensor Cores.
Depends on what you're doing, but at FP4 that goes pretty far."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://github.com/b4rtaz/distributed-llama/discussions/255"}},"_tags":["comment","author_benreesman","story_45148237"],"author":"benreesman","comment_text":"If Moore's Law is Ending leaks are to be believed, there are going to be 24GB GDDR7 5080 Super and maybe even 5070 Super Ti variants in the 1k (MSRP) range and one assumes fast Blackwell NVFP4 Tensor Cores.
Depends on what you're doing, but at FP4 that goes pretty far.","created_at":"2025-09-07T02:07:50Z","created_at_i":1757210870,"objectID":"45154733","parent_id":45149273,"story_id":45148237,"story_title":"Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5","story_url":"https://github.com/b4rtaz/distributed-llama/discussions/255","updated_at":"2026-03-05T22:40:17Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"reissbaker"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["fp4","nvfp4","blackwell"],"value":"The datasheet isn't telling you the quantization (intentionally). Model weights at FP16 are roughly 2GB per billion params. A 200B model at FP16 would take 400GB just to load the weights; a single DGX Spark has 128GB. Even two networked together couldn't do it at FP16.
You can do it, if you quantize to FP4 \u2014 and Nvidia's special variant of FP4, NVFP4, isn't too bad (and it's optimized on Blackwell). Some models are even trained at FP4 these days, like the gpt-oss models. But gigabytes are gigabytes, and you can't squeeze 400GB of FP16 weights into only 128GB (or 256GB) of space.
The datasheet is telling you the truth: you can fit a 200B model. But it's not saying you can do that at FP16 \u2014 because you can't. You can only do it at FP4."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Nvidia DGX Spark"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://www.nvidia.com/en-us/products/workstations/dgx-spark/"}},"_tags":["comment","author_reissbaker","story_45008434"],"author":"reissbaker","children":[45062650],"comment_text":"The datasheet isn't telling you the quantization (intentionally). Model weights at FP16 are roughly 2GB per billion params. A 200B model at FP16 would take 400GB just to load the weights; a single DGX Spark has 128GB. Even two networked together couldn't do it at FP16.
You can do it, if you quantize to FP4 \u2014 and Nvidia's special variant of FP4, NVFP4, isn't too bad (and it's optimized on Blackwell). Some models are even trained at FP4 these days, like the gpt-oss models. But gigabytes are gigabytes, and you can't squeeze 400GB of FP16 weights into only 128GB (or 256GB) of space.
The datasheet is telling you the truth: you can fit a 200B model. But it's not saying you can do that at FP16 \u2014 because you can't. You can only do it at FP4.","created_at":"2025-08-29T03:11:13Z","created_at_i":1756437073,"objectID":"45059716","parent_id":45057762,"story_id":45008434,"story_title":"Nvidia DGX Spark","story_url":"https://www.nvidia.com/en-us/products/workstations/dgx-spark/","updated_at":"2026-03-05T22:33:41Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"jcartu"},"story_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["fp4","nvfp4","blackwell"],"value":"Qwen 3.5 122B-A10B (MoE, ~10B active parameters) running in native NVFP4 on a single RTX PRO 6000 Blackwell GPU. 31 tokens/sec, 89GB VRAM, piecewise CUDA graphs. No multi-GPU, no cloud.
Why this matters: NVIDIA's TRT-LLM explicitly blocks desktop Blackwell from FP4 \u2014 the error literally says "FP4 Gemm not supported before Blackwell, nor GeForce Blackwell." The RTX 5090, PRO 6000, and DGX Spark all use SM120 \u2014 same FP4 tensor cores as the B100/B200 datacenter chips (SM100). The lock is artificial product segmentation, not a hardware limitation.
CUTLASS 4.2+ already ships SM120 FP4 kernels. They're compiled into vLLM. The problem is purely dispatch logic \u2014 Python-level capability checks that only recognize SM100, not SM120.
Setup (vLLM 0.17.0, stable pip install):
CUDA_VISIBLE_DEVICES=0 python -m vllm.entrypoints.openai.api_server \n--model Sehyo/Qwen3.5-122B-A10B-NVFP4 \n--port 8100 --max-model-len 4096 \n--gpu-memory-utilization 0.85 \n--compilation-config '{"cudagraph_mode": "piecewise"}'
Key gotchas: (1) Do NOT pass --quantization flag, model uses compressed-tensors format and vLLM auto-detects. (2) Full CUDA graphs OOM \u2014 use piecewise mode (31 tok/s vs 12 tok/s eager). (3) Python 3.14 breaks numba, stick with 3.13.
Results: 31 tok/s on 1 GPU vs 54 tok/s on 2 GPUs with Q8_0 llama.cpp. Half the hardware, ~60% the speed, ~98% the quality.
The broader point: SM120 and SM100 share the same FP4 tensor core architecture. CUTLASS has the kernels. The frameworks just need to route SM120 to them. A 122B MoE model on a single desktop GPU at 31 tok/s was datacenter-only six months ago.
Relevant issues: vLLM #33416, SGLang #18954, CUTLASS #2800. We're submitting a PR (~10 lines of Python).
Model: https://huggingface.co/Sehyo/Qwen3.5-122B-A10B-NVFP4"},"title":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["nvfp4","blackwell"],"value":"Show HN: NVFP4 on Desktop Blackwell \u2013 122B MoE on a Single RTX PRO 6000 31 tok/s"}},"_tags":["story","author_jcartu","story_47310428","show_hn"],"author":"jcartu","created_at":"2026-03-09T15:34:01Z","created_at_i":1773070441,"num_comments":0,"objectID":"47310428","points":2,"story_id":47310428,"story_text":"Qwen 3.5 122B-A10B (MoE, ~10B active parameters) running in native NVFP4 on a single RTX PRO 6000 Blackwell GPU. 31 tokens/sec, 89GB VRAM, piecewise CUDA graphs. No multi-GPU, no cloud.
Why this matters: NVIDIA's TRT-LLM explicitly blocks desktop Blackwell from FP4 \u2014 the error literally says "FP4 Gemm not supported before Blackwell, nor GeForce Blackwell." The RTX 5090, PRO 6000, and DGX Spark all use SM120 \u2014 same FP4 tensor cores as the B100/B200 datacenter chips (SM100). The lock is artificial product segmentation, not a hardware limitation.
CUTLASS 4.2+ already ships SM120 FP4 kernels. They're compiled into vLLM. The problem is purely dispatch logic \u2014 Python-level capability checks that only recognize SM100, not SM120.
Setup (vLLM 0.17.0, stable pip install):
CUDA_VISIBLE_DEVICES=0 python -m vllm.entrypoints.openai.api_server \n--model Sehyo/Qwen3.5-122B-A10B-NVFP4 \n--port 8100 --max-model-len 4096 \n--gpu-memory-utilization 0.85 \n--compilation-config '{"cudagraph_mode": "piecewise"}'
Key gotchas: (1) Do NOT pass --quantization flag, model uses compressed-tensors format and vLLM auto-detects. (2) Full CUDA graphs OOM \u2014 use piecewise mode (31 tok/s vs 12 tok/s eager). (3) Python 3.14 breaks numba, stick with 3.13.
Results: 31 tok/s on 1 GPU vs 54 tok/s on 2 GPUs with Q8_0 llama.cpp. Half the hardware, ~60% the speed, ~98% the quality.
The broader point: SM120 and SM100 share the same FP4 tensor core architecture. CUTLASS has the kernels. The frameworks just need to route SM120 to them. A 122B MoE model on a single desktop GPU at 31 tok/s was datacenter-only six months ago.
Relevant issues: vLLM #33416, SGLang #18954, CUTLASS #2800. We're submitting a PR (~10 lines of Python).
Model: https://huggingface.co/Sehyo/Qwen3.5-122B-A10B-NVFP4","title":"Show HN: NVFP4 on Desktop Blackwell \u2013 122B MoE on a Single RTX PRO 6000 31 tok/s","updated_at":"2026-03-12T04:10:50Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"halfdeadcat"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["fp4","nvfp4","blackwell"],"value":"After thrashing on it for a day, I could not get it going on DGX Spark with FP4 quantization. I find this irksome, since Nvidia created the NVFP4 specifically for Blackwell. The Nvidia cookbooks for this model are all for H100. I tried vLLM, Ollama and various patches.\nAs of right now on DGX: you can feasibly do FP4 on dense models. But FP4 + MoE is a largely a dead-end."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Nvidia Nemotron 3.5 Lightning and NeMo Switchyard"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://blogs.nvidia.com/blog/nemotron-lightning-switchyard-rtx-dgx/"}},"_tags":["comment","author_halfdeadcat","story_49263340"],"author":"halfdeadcat","comment_text":"After thrashing on it for a day, I could not get it going on DGX Spark with FP4 quantization. I find this irksome, since Nvidia created the NVFP4 specifically for Blackwell. The Nvidia cookbooks for this model are all for H100. I tried vLLM, Ollama and various patches.\nAs of right now on DGX: you can feasibly do FP4 on dense models. But FP4 + MoE is a largely a dead-end.","created_at":"2026-08-14T18:33:57Z","created_at_i":1786732437,"objectID":"49302819","parent_id":49270353,"story_id":49263340,"story_title":"Nvidia Nemotron 3.5 Lightning and NeMo Switchyard","story_url":"https://blogs.nvidia.com/blog/nemotron-lightning-switchyard-rtx-dgx/","updated_at":"2026-08-14T18:35:39Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"ineedasername"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["fp4","nvfp4","blackwell"],"value":"For FP4, yes... sometimes... it depends. But newer Nvidia architecture eg Blackwell w/ NVFP4 does not, they perform micro block scaling in the core. For older architectures, low quants like FP4 are also often not done native, and instead inflated back to BF16, eg with BnB."},"story_title":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["fp4"],"value":"4-bit floating point FP4"},"story_url":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["fp4"],"value":"https://www.johndcook.com/blog/2026/04/17/fp4/"}},"_tags":["comment","author_ineedasername","story_47817639"],"author":"ineedasername","comment_text":"For FP4, yes... sometimes... it depends. But newer Nvidia architecture eg Blackwell w/ NVFP4 does not, they perform micro block scaling in the core. For older architectures, low quants like FP4 are also often not done native, and instead inflated back to BF16, eg with BnB.","created_at":"2026-04-19T22:29:33Z","created_at_i":1776637773,"objectID":"47828215","parent_id":47827356,"story_id":47817639,"story_title":"4-bit floating point FP4","story_url":"https://www.johndcook.com/blog/2026/04/17/fp4/","updated_at":"2026-04-19T22:34:27Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"Schiendelman"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["fp4","nvfp4","blackwell"],"value":"We're getting to the limit of my understanding, but I believe most Blackwell users still usually run FP8 passes through the transformer engine - they'll just store weights at NVFP4. Nvidia has model-specific stabilization recipes for NVFP4 end to end, but they're taking fixes all the time.
Nvidia says Rubin should have fewer stability problems training with FP4 because of hardware changes - "adaptive compression". There will still be outlier instability inherently, but something they're designing in reduces the cost of managing it.
But yeah, grain of salt - we haven't seen this in practice."},"story_title":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["blackwell"],"value":"GLM5.2 on AMD MI355X at 2626 tok/s/node at over 2x lower cost than Blackwell"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://www.wafer.ai/blog/glm52-amd"}},"_tags":["comment","author_Schiendelman","story_48780417"],"author":"Schiendelman","comment_text":"We're getting to the limit of my understanding, but I believe most Blackwell users still usually run FP8 passes through the transformer engine - they'll just store weights at NVFP4. Nvidia has model-specific stabilization recipes for NVFP4 end to end, but they're taking fixes all the time.
Nvidia says Rubin should have fewer stability problems training with FP4 because of hardware changes - "adaptive compression". There will still be outlier instability inherently, but something they're designing in reduces the cost of managing it.
But yeah, grain of salt - we haven't seen this in practice.","created_at":"2026-07-04T04:17:21Z","created_at_i":1783138641,"objectID":"48782596","parent_id":48782427,"story_id":48780417,"story_title":"GLM5.2 on AMD MI355X at 2626 tok/s/node at over 2x lower cost than Blackwell","story_url":"https://www.wafer.ai/blog/glm52-amd","updated_at":"2026-07-04T04:21:55Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"waybarrios"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["fp4","nvfp4","blackwell"],"value":"I built a toolkit to fine-tune LLMs using LoRA + native 4-bit quantization on NVIDIA's new Blackwell GPUs (DGX Spark with GB10).
Key features:\n - NVFP4 (4-bit) via Transformer Engine - fastest option\n - MXFP8 (8-bit) for higher precision\n - bitsandbytes FP4 fallback for any CUDA GPU\n - ~240MB LoRA adapters instead of ~6GB full models\n\n Tested on DGX Spark (128GB VRAM). Training SmolLM3-3B takes ~70GB VRAM with NVFP4. Key features:\n - NVFP4 (4-bit) via Transformer Engine - fastest option\n - MXFP8 (8-bit) for higher precision\n - bitsandbytes FP4 fallback for any CUDA GPU\n - ~240MB LoRA adapters instead of ~6GB full models\n\n Tested on DGX Spark (128GB VRAM). Training SmolLM3-3B takes ~70GB VRAM with NVFP4.They support FP8/BF8 with F32 accumulate and also IU4 with I32 accumulate. The max matrix size is 16x16. For comparison, NVIDIA Blackwell GB200 supports matrices up to 256x32 for FP8 and 256x96 for NVFP4.
This matters for overall throughput, as feeding a bigger matrix unit is actually cheaper in terms of memory bandwidth, as the number of FLOPs grows O(n^2) when increasing the size of a systolic array, while the number of inputs/outputs as O(n).
1. https://www.amd.com/content/dam/amd/en/documents/radeon-tech...
2. https://semianalysis.com/2025/06/23/nvidia-tensor-core-evolu..."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"AMD\u2019s RDNA4 GPU architecture"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://chipsandcheese.com/p/amds-rdna4-gpu-architecture-at-hot"}},"_tags":["comment","author_krasin","story_45235293"],"author":"krasin","children":[45239330],"comment_text":"I refer to the RDNA4 instruction set manual ([1]), page 90, Table 41. WMMA Instructions.
They support FP8/BF8 with F32 accumulate and also IU4 with I32 accumulate. The max matrix size is 16x16. For comparison, NVIDIA Blackwell GB200 supports matrices up to 256x32 for FP8 and 256x96 for NVFP4.
This matters for overall throughput, as feeding a bigger matrix unit is actually cheaper in terms of memory bandwidth, as the number of FLOPs grows O(n^2) when increasing the size of a systolic array, while the number of inputs/outputs as O(n).
1. https://www.amd.com/content/dam/amd/en/documents/radeon-tech...
2. https://semianalysis.com/2025/06/23/nvidia-tensor-core-evolu...","created_at":"2025-09-14T02:28:06Z","created_at_i":1757816886,"objectID":"45236928","parent_id":45236773,"story_id":45235293,"story_title":"AMD\u2019s RDNA4 GPU architecture","story_url":"https://chipsandcheese.com/p/amds-rdna4-gpu-architecture-at-hot","updated_at":"2026-03-05T22:40:45Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"elliotbnvl"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["fp4","nvfp4","blackwell"],"value":"$500k is a vast overestimation. For massive concurrency at FP8 or even BF16 maybe.
NVFP4 at reasonable speeds (~120 tok/s) and concurrency is possible at a $80/90k figure with today's prices, maybe even less. That buys you 6 RTX 6000 PRO Blackwells, a decent CPU and motherboard, power supply. 576gb of VRAM.
You could do it for under $50k if you're OK with 40 tok/s decode, ~1200 tok/s prefill."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"GLM-5.2 \u2013 How to Run Locally"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://unsloth.ai/docs/models/glm-5.2"}},"_tags":["comment","author_elliotbnvl","story_48636377"],"author":"elliotbnvl","children":[48638270,48638671,48638862,48643398],"comment_text":"$500k is a vast overestimation. For massive concurrency at FP8 or even BF16 maybe.
NVFP4 at reasonable speeds (~120 tok/s) and concurrency is possible at a $80/90k figure with today's prices, maybe even less. That buys you 6 RTX 6000 PRO Blackwells, a decent CPU and motherboard, power supply. 576gb of VRAM.
You could do it for under $50k if you're OK with 40 tok/s decode, ~1200 tok/s prefill.","created_at":"2026-06-22T23:11:02Z","created_at_i":1782169862,"objectID":"48637747","parent_id":48637098,"story_id":48636377,"story_title":"GLM-5.2 \u2013 How to Run Locally","story_url":"https://unsloth.ai/docs/models/glm-5.2","updated_at":"2026-06-25T11:47:52Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"hadlock"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["fp4","nvfp4","blackwell"],"value":"For high concurrency, using the blackwell's native native W4A4 MLP compute path, nvfp4 is something like a 1.2-1.5x performance increase over FP8. We're doing data enrichment (so, tasks completed successfully + tokens/second) so the performance bump shows up in the tasks/month number.
I am just now getting the benchmarks running against 3.8 27b but I expect similar results from benching 3.6 27b at the same quant."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Qwen 3.8 27B"},"story_url":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["fp4"],"value":"https://huggingface.co/Qwen/Qwen3.8-27B-FP8"}},"_tags":["comment","author_hadlock","story_49299605"],"author":"hadlock","children":[49305844],"comment_text":"For high concurrency, using the blackwell's native native W4A4 MLP compute path, nvfp4 is something like a 1.2-1.5x performance increase over FP8. We're doing data enrichment (so, tasks completed successfully + tokens/second) so the performance bump shows up in the tasks/month number.
I am just now getting the benchmarks running against 3.8 27b but I expect similar results from benching 3.6 27b at the same quant.","created_at":"2026-08-14T20:00:02Z","created_at_i":1786737602,"objectID":"49303863","parent_id":49303557,"story_id":49299605,"story_title":"Qwen 3.8 27B","story_url":"https://huggingface.co/Qwen/Qwen3.8-27B-FP8","updated_at":"2026-08-15T05:30:55Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"reissbaker"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["fp4","nvfp4","blackwell"],"value":"At 16GB a Q4 quant of Mistral Small 3.1, or Qwen3-14B at FP8, will probably serve you best. You'd be cutting it a little close on context length due to the VRAM usage... If you want longer context, a Q4 quant of Qwen3-14B will be a bit dumber than FP8 but will leave you more breathing room. Mistral Small can take images as input, and Qwen3 will be a bit better at math/coding; YMMV otherwise.
Going below Q4 isn't worth it IMO. If you want significantly more context, probably drop down to a Q4 quant of Qwen3-8B rather than continuing to lobotomize the 14B.
Some folks have been recommending Qwen3-30B-A3, but I think 16GB of VRAM is probably not quite enough for that: at Q4 you'd be looking at 15GB for the weights alone. Qwen3-14B should be pretty similar in practice though despite being lower in param count, since it's a dense model rather than a sparse one: dense models are generally smarter-per-param than sparse models, but somewhat slower. Your 5060 should be plenty fast enough for the 14B as long as you keep everything on-GPU and stay away from CPU offloading.
Since you're on a Blackwell-generation Nvidia chip, using LLMs quantized to NVFP4 specifically will provide some speed improvements at some quality cost compared to FP8 (and will be faster than Q4 GGUF, although ~equally dumb). Ollama doesn't support NVFP4 yet, so you'd need to use vLLM (which isn't too hard, and will give better token throughput anyway). Finding pre-quantized models at NVFP4 will be more difficult since there's less-broad support, but you can use llmcompressor [1] to statically compress any FP16 LLM to NVFP4 locally \u2014 you'll probably need to use accelerate to offload params to CPU during the one-time compression process, which llmcompressor has documentation for.
I wouldn't reach for this particular power tool until you've decided on an LLM already, and just want faster perf, since it's a bit more involved than just using ollama and the initial quantization process will be slow due to CPU offload during compression (albeit it's only a one-time cost). But if you land on a Q4 model, it's not a bad choice once you have a favorite.
1: https://github.com/vllm-project/llm-compressor"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Ask HN: What is the best LLM for consumer grade hardware?"}},"_tags":["comment","author_reissbaker","story_44134896"],"author":"reissbaker","comment_text":"At 16GB a Q4 quant of Mistral Small 3.1, or Qwen3-14B at FP8, will probably serve you best. You'd be cutting it a little close on context length due to the VRAM usage... If you want longer context, a Q4 quant of Qwen3-14B will be a bit dumber than FP8 but will leave you more breathing room. Mistral Small can take images as input, and Qwen3 will be a bit better at math/coding; YMMV otherwise.
Going below Q4 isn't worth it IMO. If you want significantly more context, probably drop down to a Q4 quant of Qwen3-8B rather than continuing to lobotomize the 14B.
Some folks have been recommending Qwen3-30B-A3, but I think 16GB of VRAM is probably not quite enough for that: at Q4 you'd be looking at 15GB for the weights alone. Qwen3-14B should be pretty similar in practice though despite being lower in param count, since it's a dense model rather than a sparse one: dense models are generally smarter-per-param than sparse models, but somewhat slower. Your 5060 should be plenty fast enough for the 14B as long as you keep everything on-GPU and stay away from CPU offloading.
Since you're on a Blackwell-generation Nvidia chip, using LLMs quantized to NVFP4 specifically will provide some speed improvements at some quality cost compared to FP8 (and will be faster than Q4 GGUF, although ~equally dumb). Ollama doesn't support NVFP4 yet, so you'd need to use vLLM (which isn't too hard, and will give better token throughput anyway). Finding pre-quantized models at NVFP4 will be more difficult since there's less-broad support, but you can use llmcompressor [1] to statically compress any FP16 LLM to NVFP4 locally \u2014 you'll probably need to use accelerate to offload params to CPU during the one-time compression process, which llmcompressor has documentation for.
I wouldn't reach for this particular power tool until you've decided on an LLM already, and just want faster perf, since it's a bit more involved than just using ollama and the initial quantization process will be slow due to CPU offload during compression (albeit it's only a one-time cost). But if you land on a Q4 model, it's not a bad choice once you have a favorite.
1: https://github.com/vllm-project/llm-compressor","created_at":"2025-05-30T16:19:26Z","created_at_i":1748621966,"objectID":"44137711","parent_id":44134896,"story_id":44134896,"story_title":"Ask HN: What is the best LLM for consumer grade hardware?","updated_at":"2025-06-05T18:27:24Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"spwa4"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["fp4","nvfp4","blackwell"],"value":"The problem for nvidia is ... where do you go from here and still have spectacular performance improvements?
Nvidia got extremely lucky again and again and again, and what specifically did it is that right in time non-Nvidia researchers learned to train on smaller floating point bit lengths, which Nvidia raced to support. And great, well, done! A list of ironies though ... for example it's Google Deepmind that made the Turing generation of cards viable for nvidia. However, the new floating point formats train has arrived at it's last station, the NXFP4 station. There is no FP3 and no FP2 to go to. Is there a new train to get on? I'm not aware of one.
Nvidia's argument is "Blackwell easily doubles Ada performance!" ... but that is deceptive. The actual improvement is that Blackwell NXFP4 (4-bits) is more than double Ada FP8 (8-bit) performance in ops. That's the train that's arrived at its last station. Go back further and the same is true, just with larger and larger FP formats, starting at FP32 (single precision). Aside from a small FP64 detour, and a few "oopses" in some of the format they chose turning out useless or unstable, all quickly abandoned that's the story of nvidia in ML.
Comparing, for example, FP32 you don't see big improvements: e.g. 4090: 83 FP32 TFLOPS, 5090: 104 FP32 TFLOPS. Given the power requirements involved that's actually a regression. If you're stuck at 8 bits, nvidia's story breaks down and Ada cards beat Blackwell cards in performance per watt: 4090: 5.44 Watt/FP32 TFLOP, 5090: 5.5 Watt/FP32 TFLOP. Or, FP8, same story: 4090 is 0.681 Watt/FP8 TFLOP, 5090 is 0.686 Watt/FP8 TFLOP. Now effectively the new memory still buys some improvement but not much.
Will the next generation after Blackwell, with the same floating format as the previous generation be a 10% improvement and subject to further diminishing returns and stuck there until ... well, until we find something better than silicon? I should point out 10% is generous, because for FP8, Blackwell is actually not an improvement at all over Ada, on a per-watt basis for equivalent floating point lengths.
Plus Blackwell is ahead of the competition ... but only 1 generation. If nvidia doesn't get on a new train, the next generation of AMD cards will match the current nvidia generation. Then the next TPU generation will match nvidia."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Palantir could be the most overvalued company that ever existed"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://247wallst.com/investing/2025/11/25/palantir-could-be-the-most-overvalued-company-that-ever-existed/"}},"_tags":["comment","author_spwa4","story_46188451"],"author":"spwa4","children":[46226867],"comment_text":"The problem for nvidia is ... where do you go from here and still have spectacular performance improvements?
Nvidia got extremely lucky again and again and again, and what specifically did it is that right in time non-Nvidia researchers learned to train on smaller floating point bit lengths, which Nvidia raced to support. And great, well, done! A list of ironies though ... for example it's Google Deepmind that made the Turing generation of cards viable for nvidia. However, the new floating point formats train has arrived at it's last station, the NXFP4 station. There is no FP3 and no FP2 to go to. Is there a new train to get on? I'm not aware of one.
Nvidia's argument is "Blackwell easily doubles Ada performance!" ... but that is deceptive. The actual improvement is that Blackwell NXFP4 (4-bits) is more than double Ada FP8 (8-bit) performance in ops. That's the train that's arrived at its last station. Go back further and the same is true, just with larger and larger FP formats, starting at FP32 (single precision). Aside from a small FP64 detour, and a few "oopses" in some of the format they chose turning out useless or unstable, all quickly abandoned that's the story of nvidia in ML.
Comparing, for example, FP32 you don't see big improvements: e.g. 4090: 83 FP32 TFLOPS, 5090: 104 FP32 TFLOPS. Given the power requirements involved that's actually a regression. If you're stuck at 8 bits, nvidia's story breaks down and Ada cards beat Blackwell cards in performance per watt: 4090: 5.44 Watt/FP32 TFLOP, 5090: 5.5 Watt/FP32 TFLOP. Or, FP8, same story: 4090 is 0.681 Watt/FP8 TFLOP, 5090 is 0.686 Watt/FP8 TFLOP. Now effectively the new memory still buys some improvement but not much.
Will the next generation after Blackwell, with the same floating format as the previous generation be a 10% improvement and subject to further diminishing returns and stuck there until ... well, until we find something better than silicon? I should point out 10% is generous, because for FP8, Blackwell is actually not an improvement at all over Ada, on a per-watt basis for equivalent floating point lengths.
Plus Blackwell is ahead of the competition ... but only 1 generation. If nvidia doesn't get on a new train, the next generation of AMD cards will match the current nvidia generation. Then the next TPU generation will match nvidia.","created_at":"2025-12-09T08:29:28Z","created_at_i":1765268968,"objectID":"46202603","parent_id":46200866,"story_id":46188451,"story_title":"Palantir could be the most overvalued company that ever existed","story_url":"https://247wallst.com/investing/2025/11/25/palantir-could-be-the-most-overvalued-company-that-ever-existed/","updated_at":"2026-03-05T23:06:56Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"spwa4"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["fp4","nvfp4","blackwell"],"value":"You could just give the TLDR: by far the biggest improvement in the different generations of nVidia chips is calculating faster at half the accuracy. For blackwell vs hopper it was "double performance". By which they mean blackwell can calculate with NXFP4 at twice the rate hopper can calculate at FP8. Then go back generations all the way until you arrive at FP64, where we started. They even made a slight detour to "FP128".
Decide for yourself if this is a real improvement. You should probably consider that nVidia did not just give the new chips, but also demonstrated training a neural net with NXFP4.
It's not the only improvement, but it is by far the biggest.
As for the future: nobody's gotten FP2 to work satisfactorily yet. But hey, maybe at nVidia's next conference. But, even NXFP4 is not actually 4 bits (meaning various parts of the computation don't actually happen at 4 bits), and neither was FP8 (you could use it like that but people didn't)"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"OpenAI is walking away from expanding its Stargate data center with Oracle"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://www.cnbc.com/2026/03/09/oracle-is-building-yesterdays-data-centers-with-tomorrows-debt.html"}},"_tags":["comment","author_spwa4","story_47315128"],"author":"spwa4","comment_text":"You could just give the TLDR: by far the biggest improvement in the different generations of nVidia chips is calculating faster at half the accuracy. For blackwell vs hopper it was "double performance". By which they mean blackwell can calculate with NXFP4 at twice the rate hopper can calculate at FP8. Then go back generations all the way until you arrive at FP64, where we started. They even made a slight detour to "FP128".
Decide for yourself if this is a real improvement. You should probably consider that nVidia did not just give the new chips, but also demonstrated training a neural net with NXFP4.
It's not the only improvement, but it is by far the biggest.
As for the future: nobody's gotten FP2 to work satisfactorily yet. But hey, maybe at nVidia's next conference. But, even NXFP4 is not actually 4 bits (meaning various parts of the computation don't actually happen at 4 bits), and neither was FP8 (you could use it like that but people didn't)","created_at":"2026-03-11T11:58:41Z","created_at_i":1773230321,"objectID":"47334470","parent_id":47321259,"story_id":47315128,"story_title":"OpenAI is walking away from expanding its Stargate data center with Oracle","story_url":"https://www.cnbc.com/2026/03/09/oracle-is-building-yesterdays-data-centers-with-tomorrows-debt.html","updated_at":"2026-03-24T22:19:31Z"}],"hitsPerPage":20,"nbHits":16,"nbPages":1,"page":0,"params":"query=FP4+NVFP4+Blackwell&advancedSyntax=true&analyticsTags=backend","processingTimeMS":14,"processingTimingsMS":{"_request":{"roundTrip":18},"afterFetch":{"format":{"highlighting":1,"total":1},"merge":{"mergeLoop":{"prepareNextHit":1,"total":1},"total":1},"total":1},"fetch":{"query":10,"scanning":1,"total":12},"total":14},"query":"FP4 NVFP4 Blackwell","serverTimeMS":16}