Backed by top investors (Khosla, Lightspeed, Nat Friedman, etc.), currently ~60 employees.
Hiring for:\n- platform engineers\n- swe in general\n- front end swe / backend swe
If interested, please reach out (email in profile) and apply here:
- https://jobs.ashbyhq.com/Distyl?utm_source=PO3ZojlojQ"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Ask HN: Who is hiring? (April 2025)"}},"_tags":["comment","author_VitoriaLima","story_43547611"],"author":"VitoriaLima","children":[43731289],"comment_text":"Distyl AI | NYC or SF in person | Full Time | https://distyl.ai/
Backed by top investors (Khosla, Lightspeed, Nat Friedman, etc.), currently ~60 employees.
Hiring for:\n- platform engineers\n- swe in general\n- front end swe / backend swe
If interested, please reach out (email in profile) and apply here:
- https://jobs.ashbyhq.com/Distyl?utm_source=PO3ZojlojQ","created_at":"2025-04-17T21:47:07Z","created_at_i":1744926427,"objectID":"43722590","parent_id":43547611,"story_id":43547611,"story_title":"Ask HN: Who is hiring? (April 2025)","updated_at":"2025-04-21T20:37:45Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"jasikpark"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"Distill: a modern machine learning journal"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"http://distill.pub/about/"}},"_tags":["story","author_jasikpark","story_13915808"],"author":"jasikpark","children":[13915938,13916026,13916220,13916331,13916393,13916408,13916424,13916442,13916660,13916704,13916849,13916956,13917054,13917111,13917202,13917366,13917449,13917498,13917545,13917767,13917921,13917993,13918578,13918630,13919748,13920350,13920493,13920851,13921015,13921054,13921787],"created_at":"2017-03-20T17:08:23Z","created_at_i":1490029703,"num_comments":105,"objectID":"13915808","points":930,"story_id":13915808,"title":"Distill: a modern machine learning journal","updated_at":"2024-12-20T09:08:33Z","url":"http://distill.pub/about/"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"HenryNdubuaku"},"story_text":{"matchLevel":"none","matchedWords":[],"value":"Hey HN, Henry here from Cactus. We open-sourced Needle, a 26M parameter function-calling (tool use) model. It runs at 6000 tok/s prefill and 1200 tok/s decode on consumer devices.
We were always frustrated by the little effort made towards building agentic models that run on budget phones, so we conducted investigations that led to an observation: agentic experiences are built upon tool calling, and massive models are overkill for it. Tool calling is fundamentally retrieval-and-assembly (match query to tool name, extract argument values, emit JSON), not reasoning. Cross-attention is the right primitive for this, and FFN parameters are wasted at this scale.
Simple Attention Networks: the entire model is just attention and gating, no MLPs anywhere. Needle is an experimental run for single-shot function calling for consumer devices (phones, watches, glasses...).
Training:\n- Pretrained on 200B tokens across 16 TPU v6e (27 hours)\n- Post-trained on 2B tokens of synthesized function-calling data (45 minutes)\n- Dataset synthesized via Gemini with 15 tool categories (timers, messaging, navigation, smart home, etc.)
You can test it right now and finetune on your Mac/PC: https://github.com/cactus-compute/needle
The full writeup on the architecture is here: https://github.com/cactus-compute/needle/blob/main/docs/simp...
We found that the "no FFN" finding generalizes beyond function calling to any task where the model has access to external structured knowledge (RAG, tool use, retrieval-augmented generation). The model doesn't need to memorize facts in FFN weights if the facts are provided in the input. Experimental results to published.
While it beats FunctionGemma-270M, Qwen-0.6B, Granite-350M, LFM2.5-350M on single-shot function calling, those models have more scope/capacity and excel in conversational settings. We encourage you to test on your own tools via the playground and finetune accordingly.
This is part of our broader work on Cactus (https://github.com/cactus-compute/cactus), an inference engine built from scratch for mobile, wearables and custom hardware. We wrote about Cactus here previously: https://news.ycombinator.com/item?id=44524544
Everything is MIT licensed. Weights: https://huggingface.co/Cactus-Compute/needle\nGitHub: https://github.com/cactus-compute/needle"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model"},"url":{"matchLevel":"none","matchedWords":[],"value":"https://github.com/cactus-compute/needle"}},"_tags":["story","author_HenryNdubuaku","story_48111896","show_hn"],"author":"HenryNdubuaku","children":[48112181,48112914,48113166,48113335,48113360,48113387,48113596,48113615,48113889,48114111,48114163,48114180,48114350,48114414,48114445,48114709,48114828,48114927,48115116,48115161,48115220,48115290,48115334,48115385,48115470,48115524,48115552,48115928,48115978,48116171,48116260,48116333,48116373,48116454,48117417,48117521,48117647,48117773,48117879,48117970,48117983,48118182,48118236,48118253,48118588,48118696,48118796,48119190,48119554,48120546,48121382,48122076,48122630,48123040,48123144,48123148,48123669,48124234,48124608,48125737,48126882,48127269,48127573,48128500,48128602,48130433,48130724,48130920,48131437,48132122,48132703,48133248,48136452,48137173,48140939,48143906,48150018,48157445,48159047,48178635,48179510,48222819,48276202],"created_at":"2026-05-12T18:03:11Z","created_at_i":1778608991,"num_comments":211,"objectID":"48111896","points":776,"story_id":48111896,"story_text":"Hey HN, Henry here from Cactus. We open-sourced Needle, a 26M parameter function-calling (tool use) model. It runs at 6000 tok/s prefill and 1200 tok/s decode on consumer devices.
We were always frustrated by the little effort made towards building agentic models that run on budget phones, so we conducted investigations that led to an observation: agentic experiences are built upon tool calling, and massive models are overkill for it. Tool calling is fundamentally retrieval-and-assembly (match query to tool name, extract argument values, emit JSON), not reasoning. Cross-attention is the right primitive for this, and FFN parameters are wasted at this scale.
Simple Attention Networks: the entire model is just attention and gating, no MLPs anywhere. Needle is an experimental run for single-shot function calling for consumer devices (phones, watches, glasses...).
Training:\n- Pretrained on 200B tokens across 16 TPU v6e (27 hours)\n- Post-trained on 2B tokens of synthesized function-calling data (45 minutes)\n- Dataset synthesized via Gemini with 15 tool categories (timers, messaging, navigation, smart home, etc.)
You can test it right now and finetune on your Mac/PC: https://github.com/cactus-compute/needle
The full writeup on the architecture is here: https://github.com/cactus-compute/needle/blob/main/docs/simp...
We found that the "no FFN" finding generalizes beyond function calling to any task where the model has access to external structured knowledge (RAG, tool use, retrieval-augmented generation). The model doesn't need to memorize facts in FFN weights if the facts are provided in the input. Experimental results to published.
While it beats FunctionGemma-270M, Qwen-0.6B, Granite-350M, LFM2.5-350M on single-shot function calling, those models have more scope/capacity and excel in conversational settings. We encourage you to test on your own tools via the playground and finetune accordingly.
This is part of our broader work on Cactus (https://github.com/cactus-compute/cactus), an inference engine built from scratch for mobile, wearables and custom hardware. We wrote about Cactus here previously: https://news.ycombinator.com/item?id=44524544
Everything is MIT licensed. Weights: https://huggingface.co/Cactus-Compute/needle\nGitHub: https://github.com/cactus-compute/needle","title":"Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model","updated_at":"2026-09-28T12:49:50Z","url":"https://github.com/cactus-compute/needle"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"Anon84"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"Embarrassingly simple self-distillation improves code generation"},"url":{"matchLevel":"none","matchedWords":[],"value":"https://arxiv.org/abs/2604.01193"}},"_tags":["story","author_Anon84","story_47637757"],"author":"Anon84","children":[47638176,47638183,47638197,47638203,47638217,47638237,47638249,47638254,47638287,47638349,47638360,47638385,47638432,47638603,47638676,47638684,47638935,47638958,47639009,47639104,47639135,47639237,47639362,47639476,47639515,47639674,47639913,47639942,47640352,47640408,47640576,47641142,47641241,47641269,47641568,47642155,47642375,47642583,47642806,47643201,47643509,47643722,47643881,47644058,47644622,47644633,47644784,47644822,47644985,47645170,47645200,47645672,47647147,47647589,47647685,47670007],"created_at":"2026-04-04T10:26:21Z","created_at_i":1775298381,"num_comments":201,"objectID":"47637757","points":658,"story_id":47637757,"title":"Embarrassingly simple self-distillation improves code generation","updated_at":"2026-09-07T17:05:44Z","url":"https://arxiv.org/abs/2604.01193"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"t-3"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"US appeals court declares 158-year-old home distilling ban unconstitutional"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"https://nypost.com/2026/04/11/us-news/us-appeals-court-declares-158-year-old-home-distilling-ban-unconstitutional/"}},"_tags":["story","author_t-3","story_47751781"],"author":"t-3","children":[47752075,47752706,47752810,47753084,47753132,47753138,47753269,47753304,47753397,47753531,47753532,47753578,47753693,47754205,47754837,47755480,47755590,47756083,47756399,47757480,47758038,47759195,47759881,47761733,47763983,47765221,47765508],"created_at":"2026-04-13T13:37:33Z","created_at_i":1776087453,"num_comments":338,"objectID":"47751781","points":461,"story_id":47751781,"title":"US appeals court declares 158-year-old home distilling ban unconstitutional","updated_at":"2026-10-01T12:00:00Z","url":"https://nypost.com/2026/04/11/us-news/us-appeals-court-declares-158-year-old-home-distilling-ban-unconstitutional/"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"TheJCDenton"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"Garry Tan wants US open-weight AI labs to 'distill' frontier models, too"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"https://techcrunch.com/2026/09/11/y-combinators-garry-tan-wants-u-s-open-weight-ai-labs-to-distill-frontier-models-too/"}},"_tags":["story","author_TheJCDenton","story_49685253"],"author":"TheJCDenton","children":[49685622,49685724,49685908,49685929,49685940,49686011,49686022,49686041,49686043,49686102,49686136,49686182,49686231,49686263,49686273,49686330,49686353,49686435,49686661,49686725,49686779,49686790,49686939,49687024,49687311,49687421,49687454,49687467,49687494,49687562,49687563,49687742,49687917,49688159,49688368,49688663,49688933,49689310,49689353,49689480,49690381,49690582,49690749,49691641,49695176],"created_at":"2026-09-13T15:44:38Z","created_at_i":1789314278,"num_comments":240,"objectID":"49685253","points":418,"story_id":49685253,"title":"Garry Tan wants US open-weight AI labs to 'distill' frontier models, too","updated_at":"2026-10-02T20:01:05Z","url":"https://techcrunch.com/2026/09/11/y-combinators-garry-tan-wants-u-s-open-weight-ai-labs-to-distill-frontier-models-too/"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"chmaynard"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"Go Concurrency Distilled"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"https://antonz.org/go-concurrency-distilled/"}},"_tags":["story","author_chmaynard","story_49856988"],"author":"chmaynard","children":[49862080,49862677,49862836,49862908,49863303,49863393,49863668,49864443,49864467,49864704,49865837,49867182,49867371,49867769,49869598],"created_at":"2026-09-26T14:34:49Z","created_at_i":1790433289,"num_comments":187,"objectID":"49856988","points":402,"story_id":49856988,"title":"Go Concurrency Distilled","updated_at":"2026-10-05T23:21:29Z","url":"https://antonz.org/go-concurrency-distilled/"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"apsec112"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"I've decided to move on to Distill"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"http://colah.github.io/posts/2017-03-Distill/"}},"_tags":["story","author_apsec112","story_13915865"],"author":"apsec112","children":[13916082,13916205,13916327,13916418,13916941,13917952,13918846],"created_at":"2017-03-20T17:14:10Z","created_at_i":1490030050,"num_comments":46,"objectID":"13915865","points":353,"story_id":13915865,"title":"I've decided to move on to Distill","updated_at":"2024-09-20T00:32:20Z","url":"http://colah.github.io/posts/2017-03-Distill/"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"pkaeding"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"TL;DW: Too Long; Didn't Watch Distill YouTube Videos to the Relevant Information"},"url":{"matchLevel":"none","matchedWords":[],"value":"https://tldw.tube/"}},"_tags":["story","author_pkaeding","story_43021044"],"author":"pkaeding","children":[43028190,43055895,43055925,43055983,43056079,43056085,43056120,43056167,43056485,43056505,43056510,43056540,43056551,43056557,43056568,43056688,43056728,43056799,43056984,43057049,43057067,43057243,43057254,43057257,43057487,43057572,43057604,43057700,43057796,43057851,43058002,43058003,43058119,43058158,43058366,43058860,43059069,43059165,43059346,43059469,43059530,43059617,43059738,43059997,43060168,43060232,43060308,43061063,43061287,43061614,43062579,43063340,43063433,43063501,43064942,43064955,43065105,43065154,43067649,43073453,43083779,43083799],"created_at":"2025-02-12T02:15:17Z","created_at_i":1739326517,"num_comments":192,"objectID":"43021044","points":328,"story_id":43021044,"title":"TL;DW: Too Long; Didn't Watch Distill YouTube Videos to the Relevant Information","updated_at":"2026-02-26T03:34:02Z","url":"https://tldw.tube/"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"b4rtazz"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5"},"url":{"matchLevel":"none","matchedWords":[],"value":"https://github.com/b4rtaz/distributed-llama/discussions/162"}},"_tags":["story","author_b4rtazz","story_43059579"],"author":"b4rtazz","children":[43059793,43059832,43059858,43059885,43059899,43059934,43059942,43060110,43060285,43060342,43060683,43060693,43060700,43060775,43061219,43061835,43061871,43063898,43074451],"created_at":"2025-02-15T16:11:29Z","created_at_i":1739635889,"num_comments":156,"objectID":"43059579","points":306,"story_id":43059579,"title":"Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5","updated_at":"2026-04-22T15:34:54Z","url":"https://github.com/b4rtaz/distributed-llama/discussions/162"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"omarfarooq"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"Distil-Whisper: distilled version of Whisper that is 6 times faster, 49% smaller"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"https://github.com/huggingface/distil-whisper"}},"_tags":["story","author_omarfarooq","story_38093353"],"author":"omarfarooq","children":[38095208,38095639,38095773,38095782,38095861,38096195,38096244,38096585,38096653,38096699,38096957,38097028,38097192,38098299,38098947,38100273,38125954],"created_at":"2023-11-01T00:55:41Z","created_at_i":1698800141,"num_comments":83,"objectID":"38093353","points":277,"story_id":38093353,"title":"Distil-Whisper: distilled version of Whisper that is 6 times faster, 49% smaller","updated_at":"2025-03-08T18:36:44Z","url":"https://github.com/huggingface/distil-whisper"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"babel16"},"story_text":{"matchLevel":"none","matchedWords":[],"value":"Number Gacha is a half-parody, half-real gacha game where you roll, unwrap, and battle numbers. Play on Desktop for the best experience!"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"Show HN: Number Gacha, a gacha game distilled to its essence"},"url":{"matchLevel":"none","matchedWords":[],"value":"https://isabisabel.com/gacha/"}},"_tags":["story","author_babel16","story_48123359","show_hn"],"author":"babel16","children":[48187548,48187704,48187897,48187906,48187985,48188031,48188035,48188215,48188290,48188360,48188407,48188463,48188472,48188531,48188563,48188573,48188625,48188655,48188657,48188742,48188891,48189154,48189270,48189326,48189433,48189438,48189514,48189641,48189679,48189702,48189740,48189749,48189971,48190452,48190577,48190581,48190693,48190717,48190810,48191109,48191412,48191554,48191606,48191740,48192500,48192560,48192706,48193307,48193373,48193381,48194922,48195255,48195494,48196142,48196151,48196945,48198143,48201393,48204624,48207711,48207743,48208050,48223134,48261795,48263605],"created_at":"2026-05-13T15:39:41Z","created_at_i":1778686781,"num_comments":143,"objectID":"48123359","points":268,"story_id":48123359,"story_text":"Number Gacha is a half-parody, half-real gacha game where you roll, unwrap, and battle numbers. Play on Desktop for the best experience!","title":"Show HN: Number Gacha, a gacha game distilled to its essence","updated_at":"2026-07-26T15:31:44Z","url":"https://isabisabel.com/gacha/"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"softwaredoug"},"story_text":{"matchLevel":"none","matchedWords":[],"value":"https://xcancel.com/mkratsios47/status/2079933645888880708"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"\u201cWe have information that Moonshot distilled Fable for the development of K3\u201d"},"url":{"matchLevel":"none","matchedWords":[],"value":"https://twitter.com/mkratsios47/status/2079933645888880708"}},"_tags":["story","author_softwaredoug","story_49007610"],"author":"softwaredoug","children":[49007792,49009232,49009285,49009338,49009379,49009388,49009399,49009427,49009501,49009527,49009531,49009547,49009614,49009700,49009729,49009766,49009768,49009776,49009786,49009831,49009835,49009914,49009978,49010008,49010011,49010023,49010025,49010026,49010033,49010046,49010063,49010065,49010069,49010087,49010094,49010096,49010124,49010144,49010154,49010188,49010238,49010248,49010317,49010325,49010339,49010414,49010473,49010479,49010500,49010526,49010545,49010579,49010604,49010619,49010645,49010772,49010785,49010795,49010848,49010853,49010950,49010953,49011102,49011107,49011179,49011189,49011232,49011235,49011304,49011330,49011360,49011368,49011379,49011413,49011478,49011533,49011538,49011558,49011560,49011577,49011583,49011615,49011670,49011693,49011701,49011706,49011711,49011714,49011744,49011770,49011894,49011897,49011930,49011960,49012027,49012065,49012192,49012238,49012424,49012430,49012434,49012468,49012523,49012530,49012572,49012617,49012643,49012645,49012697,49012703,49012706,49012721,49012746,49012851,49012862,49012879,49012881,49012989,49013008,49013034,49013115,49013136,49013219,49013229,49013267,49013292,49013323,49013394,49013461,49013544,49013621,49013622,49013643,49013787,49013799,49013856,49013896,49014078,49014117,49014133,49014473,49014544,49014578,49014613,49014614,49014976,49015163,49015246,49015265,49015818,49015934,49015959,49016027,49016541,49016778,49016970,49017009,49017464,49017489,49017862,49018127,49018291,49018656,49018775,49018900,49019397,49030222,49031324,49031576,49046087],"created_at":"2026-07-22T14:42:08Z","created_at_i":1784731328,"num_comments":695,"objectID":"49007610","points":254,"story_id":49007610,"story_text":"https://xcancel.com/mkratsios47/status/2079933645888880708","title":"\u201cWe have information that Moonshot distilled Fable for the development of K3\u201d","updated_at":"2026-10-03T10:56:35Z","url":"https://twitter.com/mkratsios47/status/2079933645888880708"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"Jimmc414"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"US appeals court declares 158-year-old home distilling ban unconstitutional"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"https://www.theguardian.com/law/2026/apr/11/appeals-court-ruling-home-distilling-ban-unconstitutional"}},"_tags":["story","author_Jimmc414","story_47736298"],"author":"Jimmc414","children":[47736354,47736355,47736365,47736367,47736397,47736505,47736670,47736862,47736906,47736953,47736996,47737230,47737900,47738317,47738802,47738817,47738880,47753160],"created_at":"2026-04-12T05:08:03Z","created_at_i":1775970483,"num_comments":316,"objectID":"47736298","points":246,"story_id":47736298,"title":"US appeals court declares 158-year-old home distilling ban unconstitutional","updated_at":"2026-08-06T20:54:24Z","url":"https://www.theguardian.com/law/2026/apr/11/appeals-court-ruling-home-distilling-ban-unconstitutional"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"jboynyc"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"SICP Distilled: An idiosyncratic tour of SICP in Clojure"},"url":{"matchLevel":"none","matchedWords":[],"value":"http://www.sicpdistilled.com/"}},"_tags":["story","author_jboynyc","story_10178544"],"author":"jboynyc","children":[10178674,10178711,10179806,10179811,10180287,10182641],"created_at":"2015-09-06T18:43:12Z","created_at_i":1441564992,"num_comments":20,"objectID":"10178544","points":243,"story_id":10178544,"title":"SICP Distilled: An idiosyncratic tour of SICP in Clojure","updated_at":"2024-09-19T22:15:09Z","url":"http://www.sicpdistilled.com/"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"alexamadoriml"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"Distilling knowledge from neural networks to build smaller and faster models"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"https://blog.floydhub.com/knowledge-distillation/"}},"_tags":["story","author_alexamadoriml","story_21547312"],"author":"alexamadoriml","children":[21548781,21548839,21550074,21551075],"created_at":"2019-11-15T18:23:58Z","created_at_i":1573842238,"num_comments":16,"objectID":"21547312","points":230,"story_id":21547312,"title":"Distilling knowledge from neural networks to build smaller and faster models","updated_at":"2024-09-20T05:09:50Z","url":"https://blog.floydhub.com/knowledge-distillation/"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"peterbotond"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"Calculus distilled into hours"},"url":{"matchLevel":"none","matchedWords":[],"value":"http://ocw.mit.edu/resources/res-18-005-highlights-of-calculus-spring-2010/"}},"_tags":["story","author_peterbotond","story_1861930"],"author":"peterbotond","children":[1862101,1862433,1862448,1862601,1862770,1862964,1863512,1863631,1863950,1864160,1864536,1864849],"created_at":"2010-11-02T19:29:25Z","created_at_i":1288726165,"num_comments":37,"objectID":"1861930","points":218,"story_id":1861930,"title":"Calculus distilled into hours","updated_at":"2023-09-06T20:14:43Z","url":"http://ocw.mit.edu/resources/res-18-005-highlights-of-calculus-spring-2010/"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"Philipp__"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"SICP Distilled \u2013 An Idiosyncratic Tour of SICP in Clojure"},"url":{"matchLevel":"none","matchedWords":[],"value":"http://www.sicpdistilled.com"}},"_tags":["story","author_Philipp__","story_17643452"],"author":"Philipp__","children":[17651091,17651396,17652149,17653695,17658178,17660236],"created_at":"2018-07-30T10:37:54Z","created_at_i":1532947074,"num_comments":12,"objectID":"17643452","points":211,"story_id":17643452,"title":"SICP Distilled \u2013 An Idiosyncratic Tour of SICP in Clojure","updated_at":"2026-03-19T18:10:22Z","url":"http://www.sicpdistilled.com"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"cgorlla"},"story_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"We recently used DeepSeek V4 Flash as a teacher for finance tasks with GPT-OSS-120B. Distillation works well on this problem. At a constrained 8k token budget, our self-distilled 120B scores 83.61% on FinanceReasoning, above Kimi K3 (81.93%) and Inkling (65.13%). We released the 20B open weights. With V4 as the teacher though, we realized it would be timely to measure if the censorship characteristic of it transferred to the distilled version of the base model. tl;dr it didn't, the teacher answered politically sensitive questions 7 SDs differently than expected, but the distilled model's behavior remained the same as its American base. You can try a couple queries yourself with no auth here: http://playground.ctgt.ai/
I will now dive in to the motivation, methodology and detailed results for those interested. The hard part of measuring this phenomena is isolating whether a model is reluctant to talk about sensitive things generally vs. a particular country's sensitive things. So we made 152 matched pairs where one prompt asked about a Chinese concept, and the other asked about a non-Chinese version of that concept. For example, the Great Leap Forward vs. the Holodomor. These were scored 0-100 by four LLM judges (Grok 4.20, Gemini 3.5 Flash, GPT-5 mini, Claude Sonnet 4.6), validated against 96 human scores at r=0.948. OpenRouter blocked some of these so we hosted the weights ourselves.
The teacher's gap on the core political set of pairs was +45.45 points, ~7 standard deviations from chance, and every distilled student was within 1 point of its base. Subliminal learning literature says this is expected when the initializations are not shared between teacher and student, which is true here. The distillation data also did not contain any China-sensitive content. The contribution here was to release the evaluation framework (LineageEval: https://github.com/CTGT-Inc/lineage-eval/) to elevate the discussion around this topic in DC and beyond. We are an interpretability lab working on high risk and regulated applications of AI, so we hear a lot of vagaries aimed at the supposed dangers of distilling Chinese models on American bases. We believe these conversations should be based on open, auditable frameworks and not feelings. We plan to test what happens with a Chinese teacher into a Chinese-lineage base like Qwen next.
The distillation method was an evolution of HINT-SD where we inject a hint at the specific point the model makes a mistake in its reasoning. Then we train on the corrected continuation with reverse KL over the next 100 toks of the rollout. As mentioned above 120B itself was efficacious as a teacher, and we ended up shipping this version. The self-distilled 120B scores 83.61% on FinanceReasoning, above Kimi K3 (81.93%) and Inkling (65.13%). Ours finishes 98.7% of problems in budget; the larger models truncate (90.76% and 71.01%) which score as incorrect. At 100k tokens big models gain (Kimi 89.92%). So for a finance task at a constrained (perhaps more realistic) budget a 120B on one H100 at ~$0.00026/query outpaced models running 62-160x more per query.
We put out the 20B finance model as open weights (64.71% to 74.79% at 8k on FinanceReasoning, 23% lower cost/query, runs on one 80GB GPU), the 120B in a playground with teacher and students side by side (a few queries, no auth), and LineageEval with all prompts, controls, rubric, and code.
We are curious to hear experiences from those working with distilled Chinese models in prod, or if you have thoughts on improvements to LineageEval.
https://huggingface.co/ctgt-inc/gpt-oss-20b-finance
https://github.com/CTGT-Inc/lineage-eval/
https://www.ctgt.ai/research/distillation-censorship-transfe..."},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"Show HN: Distilling DeepSeek into GPT-OSS doesn't transfer censorship. Try it"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"https://www.ctgt.ai/research/distillation-censorship-transfer"}},"_tags":["story","author_cgorlla","story_49113599","show_hn"],"author":"cgorlla","children":[49114663,49114771,49114791,49114863,49114868,49114962,49115076,49115263,49115591,49115633,49115745,49116077,49116127,49116173,49116247,49116321,49116580,49117024,49117092,49117331,49117714,49117807,49117873,49117888,49117920,49121490,49123876,49124031],"created_at":"2026-07-30T18:13:06Z","created_at_i":1785435186,"num_comments":73,"objectID":"49113599","points":170,"story_id":49113599,"story_text":"We recently used DeepSeek V4 Flash as a teacher for finance tasks with GPT-OSS-120B. Distillation works well on this problem. At a constrained 8k token budget, our self-distilled 120B scores 83.61% on FinanceReasoning, above Kimi K3 (81.93%) and Inkling (65.13%). We released the 20B open weights. With V4 as the teacher though, we realized it would be timely to measure if the censorship characteristic of it transferred to the distilled version of the base model. tl;dr it didn't, the teacher answered politically sensitive questions 7 SDs differently than expected, but the distilled model's behavior remained the same as its American base. You can try a couple queries yourself with no auth here: http://playground.ctgt.ai/
I will now dive in to the motivation, methodology and detailed results for those interested. The hard part of measuring this phenomena is isolating whether a model is reluctant to talk about sensitive things generally vs. a particular country's sensitive things. So we made 152 matched pairs where one prompt asked about a Chinese concept, and the other asked about a non-Chinese version of that concept. For example, the Great Leap Forward vs. the Holodomor. These were scored 0-100 by four LLM judges (Grok 4.20, Gemini 3.5 Flash, GPT-5 mini, Claude Sonnet 4.6), validated against 96 human scores at r=0.948. OpenRouter blocked some of these so we hosted the weights ourselves.
The teacher's gap on the core political set of pairs was +45.45 points, ~7 standard deviations from chance, and every distilled student was within 1 point of its base. Subliminal learning literature says this is expected when the initializations are not shared between teacher and student, which is true here. The distillation data also did not contain any China-sensitive content. The contribution here was to release the evaluation framework (LineageEval: https://github.com/CTGT-Inc/lineage-eval/) to elevate the discussion around this topic in DC and beyond. We are an interpretability lab working on high risk and regulated applications of AI, so we hear a lot of vagaries aimed at the supposed dangers of distilling Chinese models on American bases. We believe these conversations should be based on open, auditable frameworks and not feelings. We plan to test what happens with a Chinese teacher into a Chinese-lineage base like Qwen next.
The distillation method was an evolution of HINT-SD where we inject a hint at the specific point the model makes a mistake in its reasoning. Then we train on the corrected continuation with reverse KL over the next 100 toks of the rollout. As mentioned above 120B itself was efficacious as a teacher, and we ended up shipping this version. The self-distilled 120B scores 83.61% on FinanceReasoning, above Kimi K3 (81.93%) and Inkling (65.13%). Ours finishes 98.7% of problems in budget; the larger models truncate (90.76% and 71.01%) which score as incorrect. At 100k tokens big models gain (Kimi 89.92%). So for a finance task at a constrained (perhaps more realistic) budget a 120B on one H100 at ~$0.00026/query outpaced models running 62-160x more per query.
We put out the 20B finance model as open weights (64.71% to 74.79% at 8k on FinanceReasoning, 23% lower cost/query, runs on one 80GB GPU), the 120B in a playground with teacher and students side by side (a few queries, no auth), and LineageEval with all prompts, controls, rubric, and code.
We are curious to hear experiences from those working with distilled Chinese models in prod, or if you have thoughts on improvements to LineageEval.
https://huggingface.co/ctgt-inc/gpt-oss-20b-finance
https://github.com/CTGT-Inc/lineage-eval/
https://www.ctgt.ai/research/distillation-censorship-transfe...","title":"Show HN: Distilling DeepSeek into GPT-OSS doesn't transfer censorship. Try it","updated_at":"2026-09-04T11:21:19Z","url":"https://www.ctgt.ai/research/distillation-censorship-transfer"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"nickmain"},"story_text":{"matchLevel":"none","matchedWords":[],"value":""},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["distyl"],"value":"Clojure Distilled"},"url":{"matchLevel":"none","matchedWords":[],"value":"http://yogthos.github.io/ClojureDistilled.html"}},"_tags":["story","author_nickmain","story_8374263"],"author":"nickmain","children":[8374439,8374452,8374567,8374572,8374607,8374951,8375042,8375216,8375337,8375380,8375437,8376094,8376605,8376743],"created_at":"2014-09-26T18:58:49Z","created_at_i":1411757929,"num_comments":51,"objectID":"8374263","points":168,"story_id":8374263,"story_text":"","title":"Clojure Distilled","updated_at":"2023-09-06T22:28:33Z","url":"http://yogthos.github.io/ClojureDistilled.html"}],"hitsPerPage":20,"nbHits":23398,"nbPages":50,"page":0,"params":"query=Distyl&advancedSyntax=true&analyticsTags=backend","processingTimeMS":12,"processingTimingsMS":{"_request":{"roundTrip":16},"afterFetch":{"format":{"highlighting":1,"total":1}},"fetch":{"query":6,"scanning":4,"total":11},"total":12},"query":"Distyl","serverTimeMS":14}