https://www.youtube.com/watch?v=d6iQrh2TK98","title":null,"type":"comment","url":null}],"created_at":"2026-09-23T16:23:37.000Z","created_at_i":1790180617,"id":49818466,"options":[],"parent_id":49818434,"points":null,"story_id":49816899,"text":"Humans give way more random answers than LLMs to questions like "give me a random number between 1-100" (when not giving the LLM any tool calls).","title":null,"type":"comment","url":null}],"created_at":"2026-09-23T16:21:37.000Z","created_at_i":1790180497,"id":49818434,"options":[],"parent_id":49817305,"points":null,"story_id":49816899,"text":"Did you expect it to be good at it?
Humans also don't give a perfect 1/n probability when asked for a random number.","title":null,"type":"comment","url":null},{"author":"drtz","children":[{"author":"tomrod","children":[],"created_at":"2026-09-23T16:32:07.000Z","created_at_i":1790181127,"id":49818634,"options":[],"parent_id":49818589,"points":null,"story_id":49816899,"text":"The value of grandparent comment is that it identifies an edge case to keep in mind and make well-defined -- keeps us from blindly trusting.","title":null,"type":"comment","url":null}],"created_at":"2026-09-23T16:29:58.000Z","created_at_i":1790180998,"id":49818589,"options":[],"parent_id":49817305,"points":null,"story_id":49816899,"text":"In the early Gemini 2 days (don't remember which version exactly) I had Gemini running as a voice assistant in my kitchen, and asked it to flip a coin and tell me if it was heads or tails. It responded with "heads". I was curious if it was actually doing something to simulate randomness, so I asked a few more times and saw a pattern: "tails", "heads", "tails", "heads"...
It continued alternating between the two until I got bored (around a dozen turns).
Unless your specific test is baked into its training, real probabilities require math and rough approximation at a minimum needs reasoning to sanity-check. Jev does neither. This isn't a new problem or anything unique to Jev.","title":null,"type":"comment","url":null},{"author":"lesam","children":[{"author":"kantahayashi","children":[],"created_at":"2026-09-23T16:56:45.000Z","created_at_i":1790182605,"id":49819065,"options":[],"parent_id":49818812,"points":null,"story_id":49816899,"text":"Yes, and TypeSafe itself says Jev returns "calibrated probabilities", which is the former.
From TypeSafe docs:
"Higher probability should correspond to a greater chance that the answer is correct."
"Outcomes assigned a probability of 0.2 should occur about 20% of the time."
https://docs.typesafe.ai/introduction/machine-learning-prime...","title":null,"type":"comment","url":null}],"created_at":"2026-09-23T16:42:20.000Z","created_at_i":1790181740,"id":49818812,"options":[],"parent_id":49817305,"points":null,"story_id":49816899,"text":"So you want the probability that the answer is correct, but Jev is providing the probability that its answer is optimal?","title":null,"type":"comment","url":null},{"author":"esperent","children":[{"author":"bee_rider","children":[],"created_at":"2026-09-23T17:38:33.000Z","created_at_i":1790185113,"id":49819721,"options":[],"parent_id":49819174,"points":null,"story_id":49816899,"text":"As someone who hasn\u2019t used Jev, this seems reasonable in the sense that \u201cprobability that I\u2019ve given the best answer\u201d seems like a useful thing to provide.
Anyway, if it is \u201cprobability that I gave the best answer\u201d why 83%? I mean, the question is a sort of a textbook-style probability question in the sense that everything is well defined. Shouldn\u2019t it be basically 100%?","title":null,"type":"comment","url":null},{"author":"kantahayashi","children":[],"created_at":"2026-09-23T17:44:48.000Z","created_at_i":1790185488,"id":49819804,"options":[],"parent_id":49819174,"points":null,"story_id":49816899,"text":"TypeSafe defines the probabilities Jev returns as "calibrated probabilities". "Probability" here means the probability of the answer being correct. If the probability is 10%, the choice should be correct about one time in ten. So, when Jev returns 83% probability it should be correct about 83 times out of 100, but the choices were only correct about 19 times out of 100, and the true probability is 1/6.
"Higher probability should correspond to a greater chance that the answer is correct."
https://docs.typesafe.ai/introduction/machine-learning-prime...","title":null,"type":"comment","url":null}],"created_at":"2026-09-23T17:03:29.000Z","created_at_i":1790183009,"id":49819174,"options":[],"parent_id":49817305,"points":null,"story_id":49816899,"text":"> The true probability of face 1 is 1/6, but Jev always chose face 1 and the probability it returned was about 83%
I think this is a misunderstanding of what that 83% probability means. You think you're asking Jev to assign a probability to a dice roll, and if so then obviously it should pick each answer 1/6 of the time.
However, what Jev is doing is assigning a probability to the answer it should choose. And it's choosing 1 with 83% probability, as the correct answer to give, when there's no additional knowledge available.
1 is a perfectly valid answer here. In the absence of other data there's no more (nor less) valid answer than that. and I assume that just like LLMs, the token prediction must assign higher probability to one outcome.
If you keep feeding the previous answers back in (if you can do that with Jev) I assume it would start giving other numbers because it's no longer operating from zero info. Maybe that would even reach ~1/6 after enough rounds, although maybe not.","title":null,"type":"comment","url":null}],"created_at":"2026-09-23T15:06:57.000Z","created_at_i":1790176017,"id":49817305,"options":[],"parent_id":49816899,"points":null,"story_id":49816899,"text":"I tested Jev with a fair die 400 times without telling it the die result. The true probability of face 1 is 1/6, but Jev always chose face 1 and the probability it returned was about 83%. I also tested with a fair coin 200 times and got 0.92 probability.
I did several tests and I think Jev is good at problems with a correct answer but weak at problems about actual probabilities whose answers can't be known at all.
Write-up: "Jev Does Not Play Dice"\nhttps://kantahayashiai.github.io/posts/jev-does-not-play-dic...","title":null,"type":"comment","url":null},{"author":"conmod278","children":[],"created_at":"2026-09-23T15:55:02.000Z","created_at_i":1790178902,"id":49818015,"options":[],"parent_id":49816899,"points":null,"story_id":49816899,"text":"In future, we will see intiatives similar to OpenStreetMap for Textual data or Web similar to high quality non-contaminated steel.","title":null,"type":"comment","url":null},{"author":"daft_pink","children":[{"author":"oreoftw","children":[],"created_at":"2026-09-23T17:06:29.000Z","created_at_i":1790183189,"id":49819227,"options":[],"parent_id":49818079,"points":null,"story_id":49816899,"text":"Jev\u2019s context should have signals/features to operate on. Same way LLMs can use CV & code to analyze an image.","title":null,"type":"comment","url":null}],"created_at":"2026-09-23T15:58:52.000Z","created_at_i":1790179132,"id":49818079,"options":[],"parent_id":49816899,"points":null,"story_id":49816899,"text":"I'm really looking for a multi-modal image capable version of Jev.
If we could get machine learning type results on images without training, that would be fantastic.","title":null,"type":"comment","url":null},{"author":"simonw","children":[{"author":"alexmolas","children":[],"created_at":"2026-09-23T16:15:51.000Z","created_at_i":1790180151,"id":49818345,"options":[],"parent_id":49818140,"points":null,"story_id":49816899,"text":"If you want calibrated probabilities you'll be forced to fine-tune it","title":null,"type":"comment","url":null},{"author":"edot","children":[],"created_at":"2026-09-23T16:41:37.000Z","created_at_i":1790181697,"id":49818799,"options":[],"parent_id":49818140,"points":null,"story_id":49816899,"text":"But why? Jev-style models seem useful for "I have no clue what my incoming distribution looks like but I need to give some sort of answer". If I know what my incoming distribution looks like I'll just upload a CSV of that into ChatGPT and ask it to fit a basic ML model on my data.","title":null,"type":"comment","url":null}],"created_at":"2026-09-23T16:03:28.000Z","created_at_i":1790179408,"id":49818140,"options":[],"parent_id":49816899,"points":null,"story_id":49816899,"text":"Fine tuning LLMs has turned out to be mostly not worth the effort, but I wonder if fine tuning Jev-style models will turn out to be a whole lot more useful.","title":null,"type":"comment","url":null},{"author":"a_c","children":[{"author":"phainopepla2","children":[{"author":"CaptainNegative","children":[],"created_at":"2026-09-23T16:25:28.000Z","created_at_i":1790180728,"id":49818503,"options":[],"parent_id":49818250,"points":null,"story_id":49816899,"text":"I asked Jev and it said 41% yes 59% no.","title":null,"type":"comment","url":null}],"created_at":"2026-09-23T16:10:32.000Z","created_at_i":1790179832,"id":49818250,"options":[],"parent_id":49818191,"points":null,"story_id":49816899,"text":"Is Jev good at detecting AI-generated text?","title":null,"type":"comment","url":null},{"author":"hbrn","children":[],"created_at":"2026-09-23T16:44:37.000Z","created_at_i":1790181877,"id":49818853,"options":[],"parent_id":49818191,"points":null,"story_id":49816899,"text":"I suspect that it's as handy as a coin flip.
And if I'm being charitable to Jev (which is nearly impossible at this point), detecting whether blob of text is AI generated is not a "system one" question.","title":null,"type":"comment","url":null}],"created_at":"2026-09-23T16:06:57.000Z","created_at_i":1790179617,"id":49818191,"options":[],"parent_id":49816899,"points":null,"story_id":49816899,"text":"I\u2019m using Jev to classify a blob of text I see in browser with an extension. Calibrated? No. But it\u2019s handy enough. If a large blob of text is likely AI generated, I\u2019m very likely to skip it","title":null,"type":"comment","url":null},{"author":"empath75","children":[{"author":"alexmolas","children":[],"created_at":"2026-09-23T16:30:14.000Z","created_at_i":1790181014,"id":49818597,"options":[],"parent_id":49818441,"points":null,"story_id":49816899,"text":""Not especially good at classifying" does sound like a serious drawback for a classifier","title":null,"type":"comment","url":null},{"author":"0123456789ABCDE","children":[],"created_at":"2026-09-23T16:49:13.000Z","created_at_i":1790182153,"id":49818942,"options":[],"parent_id":49818441,"points":null,"story_id":49816899,"text":"do you have a benchmark name and results we can compare to other models?","title":null,"type":"comment","url":null}],"created_at":"2026-09-23T16:22:25.000Z","created_at_i":1790180545,"id":49818441,"options":[],"parent_id":49816899,"points":null,"story_id":49816899,"text":"I spent a couple of days with it and it is fast and cheap but not especially good at classifying.","title":null,"type":"comment","url":null},{"author":"abhgh","children":[{"author":"edot","children":[{"author":"abhgh","children":[],"created_at":"2026-09-23T16:44:15.000Z","created_at_i":1790181855,"id":49818846,"options":[],"parent_id":49818815,"points":null,"story_id":49816899,"text":"Thank you!","title":null,"type":"comment","url":null}],"created_at":"2026-09-23T16:42:29.000Z","created_at_i":1790181749,"id":49818815,"options":[],"parent_id":49818461,"points":null,"story_id":49816899,"text":"It's on OpenRouter if you want to try it.","title":null,"type":"comment","url":null},{"author":"danielmarkbruce","children":[],"created_at":"2026-09-23T17:21:34.000Z","created_at_i":1790184094,"id":49819473,"options":[],"parent_id":49818461,"points":null,"story_id":49816899,"text":"While I don't believe they are doing the following: you can calibrate by inspecting the reasoning traces. That is the relevant distribution. If you ask someone to explain how/why they are classifying something one way v another, you can get a reasonably good understanding of their confidence level.","title":null,"type":"comment","url":null}],"created_at":"2026-09-23T16:23:24.000Z","created_at_i":1790180604,"id":49818461,"options":[],"parent_id":49816899,"points":null,"story_id":49816899,"text":"I like this post. I haven't had time to dig into Jev (they aren't accepting new signups), but calibrated probabilities is one of their pitches that caught my attention. And I was wondering how does one offer them on user data. Standard calibration essentially ensures that if a score of 0.8 accompanies a positive prediction (assuming the simple case of binary classification), then if you gathered together all predictions with a score of 0.8, around 80% will be correct.
If you have just one example you're sending to a model, how would they guarantee 80% over your data?
FYI, for an overview, scikit's page on calibration is great [1], and my answer on Quora from a long time ago covers a specific type [2].
[1] https://scikit-learn.org/stable/modules/calibration.html
[2] https://www.quora.com/How-is-isotonic-regression-used-in-pra...","title":null,"type":"comment","url":null},{"author":"bnbn88","children":[{"author":"jrochkind1","children":[],"created_at":"2026-09-23T16:25:41.000Z","created_at_i":1790180741,"id":49818509,"options":[],"parent_id":49818492,"points":null,"story_id":49816899,"text":"They seem to have a really good social media astroturf marketing campaign.","title":null,"type":"comment","url":null},{"author":"softwaredoug","children":[{"author":"Tostino","children":[],"created_at":"2026-09-23T17:02:48.000Z","created_at_i":1790182968,"id":49819159,"options":[],"parent_id":49818856,"points":null,"story_id":49816899,"text":"The available API shapes have limited so much over the years. It's really hard to come up with a new one and have it used, so everyone just tries to fit their work into the existing APIs (chat/completions).
Good job to them for putting out something that does seem quite nice to use, and will likely get a bit of wider traction.","title":null,"type":"comment","url":null}],"created_at":"2026-09-23T16:44:45.000Z","created_at_i":1790181885,"id":49818856,"options":[],"parent_id":49818492,"points":null,"story_id":49816899,"text":"It\u2019s not just price and speed, the API is very well designed for classification. Going beyond the current structured outputs.","title":null,"type":"comment","url":null}],"created_at":"2026-09-23T16:24:53.000Z","created_at_i":1790180693,"id":49818492,"options":[],"parent_id":49816899,"points":null,"story_id":49816899,"text":"This hype is caused by the price and the speed since most people don't know about small fast models and use big models for everything.","title":null,"type":"comment","url":null},{"author":"jackb4040","children":[],"created_at":"2026-09-23T16:32:47.000Z","created_at_i":1790181167,"id":49818647,"options":[],"parent_id":49816899,"points":null,"story_id":49816899,"text":"This is why I don't understand why everyone's freaking out about it. By far the biggest problem with LLM classifiers is that they treat every individual business as the blurry average of all businesses in their training data. Being lighter is fine if you control for everything else, but at least at my company we would actually have room for a significantly more expensive / slower classifier if it were demonstrably better at following instructions.","title":null,"type":"comment","url":null},{"author":"sreekanth850","children":[],"created_at":"2026-09-23T16:51:15.000Z","created_at_i":1790182275,"id":49818979,"options":[],"parent_id":49816899,"points":null,"story_id":49816899,"text":"People on X will use JEV for self driving cars too. That is the height of stupidity. I\u2019ve seen this in so many tweets.","title":null,"type":"comment","url":null},{"author":"apwheele","children":[],"created_at":"2026-09-23T17:04:41.000Z","created_at_i":1790183081,"id":49819197,"options":[],"parent_id":49816899,"points":null,"story_id":49816899,"text":"While this is true, it is still possible to take the probabilities and make determinations of recall and false positives using conformal sets, https://crimede-coder.com/blogposts/2026/ConfClassification
(Or just use a model to re-calibrate the probabilities, I like the conformal approach though as those rates are what I often care about.)
You just need some labelled data to generate the "corrected" probabilities (or thresholds to meet the specified error rates).","title":null,"type":"comment","url":null},{"author":"time0ut","children":[],"created_at":"2026-09-23T17:14:33.000Z","created_at_i":1790183673,"id":49819355,"options":[],"parent_id":49816899,"points":null,"story_id":49816899,"text":"I have been running a series of experiments on Jev since its release targeted at understanding it, seeing how it handles real use cases I have, and maybe figure out what it is inside.
Some of my tests do point towards what this post says. I was not successful in getting it's score to align with an existing rubric I had. It 'worked' but it was off and compressed from where I wanted it to be. Not a bad starting point, but I couldn't get it to move to where I intended the rubric to be. It wasn't the most robust test and I didn't spend a lot of time trying, but it wasn't just instantly magical.
However, it does seem genuinely useful just by being fast and cheap and good enough, so I am still a bit hyped.","title":null,"type":"comment","url":null}],"created_at":"2026-09-23T14:39:57.000Z","created_at_i":1790174397,"id":49816899,"options":[],"parent_id":null,"points":36,"story_id":49816899,"text":null,"title":"Jev Can't Be Calibrated","type":"story","url":"https://www.alexmolas.com/2026/09/23/jev-cant-be-calibrated.html"}