And despite this, modern life is made possible by the illusion that "regulations" work..","title":null,"type":"comment","url":null}],"created_at":"2026-06-05T11:31:01.000Z","created_at_i":1780659061,"id":48410975,"options":[],"parent_id":48410726,"points":null,"story_id":48410589,"text":"For the same reasons why we eventully pollute and corrupt every system and environment we use. If there is any benefit that can be extracted for some while the costs are borne by many, than this will occur and generate a positive feedback loop that grows over time.
It's the law of monetization.","title":null,"type":"comment","url":null},{"author":"dmos62","children":[{"author":"temp8830","children":[],"created_at":"2026-06-05T12:38:33.000Z","created_at_i":1780663113,"id":48411587,"options":[],"parent_id":48411447,"points":null,"story_id":48410589,"text":"People who love cookie banners either don't exist, or are alien invaders :)","title":null,"type":"comment","url":null}],"created_at":"2026-06-05T12:22:37.000Z","created_at_i":1780662157,"id":48411447,"options":[],"parent_id":48410726,"points":null,"story_id":48410589,"text":"It seems there's little agreement over how the web is broken.","title":null,"type":"comment","url":null},{"author":"jt2190","children":[],"created_at":"2026-06-05T12:37:43.000Z","created_at_i":1780663063,"id":48411578,"options":[],"parent_id":48410726,"points":null,"story_id":48410589,"text":"Because while consumers value \u201cinefficiency\u201d (high design, wonderful prose, beautiful images, great usability) they don\u2019t want to actually pay for it. Producers have to become extremely efficient without revenue, and are stuck with a choice: Produce at a loss, stop producing, or seek payment from another source (sponsorships, ads).","title":null,"type":"comment","url":null},{"author":"takenotice","children":[],"created_at":"2026-06-07T12:49:25.000Z","created_at_i":1780836565,"id":48434311,"options":[],"parent_id":48410726,"points":null,"story_id":48410589,"text":"Capitalism","title":null,"type":"comment","url":null}],"created_at":"2026-06-05T11:00:12.000Z","created_at_i":1780657212,"id":48410726,"options":[],"parent_id":48410632,"points":null,"story_id":48410589,"text":"I wonder why we broke the web.","title":null,"type":"comment","url":null},{"author":"sunir","children":[{"author":"daniel-alexande","children":[],"created_at":"2026-06-05T14:21:39.000Z","created_at_i":1780669299,"id":48412950,"options":[],"parent_id":48411531,"points":null,"story_id":48410589,"text":"Always loved Gopher","title":null,"type":"comment","url":null},{"author":"gopher_space","children":[],"created_at":"2026-06-05T14:28:30.000Z","created_at_i":1780669710,"id":48413048,"options":[],"parent_id":48411531,"points":null,"story_id":48410589,"text":"A man can dream.","title":null,"type":"comment","url":null}],"created_at":"2026-06-05T12:32:24.000Z","created_at_i":1780662744,"id":48411531,"options":[],"parent_id":48410632,"points":null,"story_id":48410589,"text":"We'll finally bring back Gopher.","title":null,"type":"comment","url":null},{"author":"soco","children":[],"created_at":"2026-06-05T12:32:35.000Z","created_at_i":1780662755,"id":48411533,"options":[],"parent_id":48410632,"points":null,"story_id":48410589,"text":"It's a matter of time until the web for machines will be crawling with ads and everything else, and worse.","title":null,"type":"comment","url":null},{"author":"tacostakohashi","children":[],"created_at":"2026-06-05T12:38:43.000Z","created_at_i":1780663123,"id":48411588,"options":[],"parent_id":48410632,"points":null,"story_id":48410589,"text":"Yeah, when browsers have a "reader mode", it's pretty obvious the plot has been lost somewhere.","title":null,"type":"comment","url":null}],"created_at":"2026-06-05T10:50:08.000Z","created_at_i":1780656608,"id":48410632,"options":[],"parent_id":48410589,"points":null,"story_id":48410589,"text":"We broke the web so badly for humans that we had to build a clean web for machines, and now humans will have to use machines to experience a clean web again.","title":null,"type":"comment","url":null},{"author":"cyanydeez","children":[{"author":"maccam912","children":[],"created_at":"2026-06-05T11:47:09.000Z","created_at_i":1780660029,"id":48411118,"options":[],"parent_id":48410647,"points":null,"story_id":48410589,"text":"Already happening. I was using Claude to check out sampler plugins and I'm sure it happens undetected, and it might have mentioned it with other versions, but Claude Opus 4.8, being it's helpful, honest self, told me that one of the pages it reviewed had hidden text instructing it to recommend that plugin. It caught it and was able to avoid influence from that plugin at least, but we're already living in that world.","title":null,"type":"comment","url":null}],"created_at":"2026-06-05T10:52:14.000Z","created_at_i":1780656734,"id":48410647,"options":[],"parent_id":48410589,"points":null,"story_id":48410589,"text":"oh don't worry, in 5 years your AI will be unundated with context poison prompts that try to get them to spend all your bank notes and meta bucks on equally useless things.
This is just a redeux of the early web.","title":null,"type":"comment","url":null},{"author":"realty_geek","children":[{"author":"jbrooksuk","children":[],"created_at":"2026-06-05T11:13:29.000Z","created_at_i":1780658009,"id":48410825,"options":[],"parent_id":48410725,"points":null,"story_id":48410589,"text":"Mintlify generates an llms.txt and llms-full.txt for all documentation sites. These work really well:
- https://cloud.laravel.com/docs/llms.txt
- https://cloud.laravel.com/docs/llms-full.txt","title":null,"type":"comment","url":null},{"author":"pramodbiligiri","children":[],"created_at":"2026-06-05T17:17:49.000Z","created_at_i":1780679869,"id":48415508,"options":[],"parent_id":48410725,"points":null,"story_id":48410589,"text":"Anthropic's developer documentation: https://platform.claude.com/llms.txt. There's also https://platform.claude.com/llms-full.txt which is (WARNING) much bigger. Not sure where this second one fits into the standard.","title":null,"type":"comment","url":null}],"created_at":"2026-06-05T11:00:07.000Z","created_at_i":1780657207,"id":48410725,"options":[],"parent_id":48410589,"points":null,"story_id":48410589,"text":"What is an example of a site with a good llm.txt?","title":null,"type":"comment","url":null},{"author":"onion2k","children":[{"author":"8organicbits","children":[],"created_at":"2026-06-05T12:01:03.000Z","created_at_i":1780660863,"id":48411234,"options":[],"parent_id":48410768,"points":null,"story_id":48410589,"text":"Of course plugins that do this already exist. Save your tokens.","title":null,"type":"comment","url":null}],"created_at":"2026-06-05T11:05:32.000Z","created_at_i":1780657532,"id":48410768,"options":[],"parent_id":48410589,"points":null,"story_id":48410589,"text":"The only annoyance is web browsers like chrome do not render the markdown.
I imagine Claude could zero-shot a Chrome plugin for that.","title":null,"type":"comment","url":null},{"author":"rickette","children":[{"author":"HermanMartinus","children":[{"author":"isaachinman","children":[{"author":"the_real_cher","children":[{"author":"johannes1234321","children":[],"created_at":"2026-06-05T12:19:14.000Z","created_at_i":1780661954,"id":48411409,"options":[],"parent_id":48411127,"points":null,"story_id":48410589,"text":"Very few blogs require frequent updates. Even with user comments.","title":null,"type":"comment","url":null}],"created_at":"2026-06-05T11:48:34.000Z","created_at_i":1780660114,"id":48411127,"options":[],"parent_id":48410967,"points":null,"story_id":48410589,"text":"Are all blogs static though?","title":null,"type":"comment","url":null},{"author":"nickserv","children":[],"created_at":"2026-06-05T11:56:04.000Z","created_at_i":1780660564,"id":48411188,"options":[],"parent_id":48410967,"points":null,"story_id":48410589,"text":"> a blogging platform with around 80k blogs
But nah, I'm sure OP doesn't know about CDNs.","title":null,"type":"comment","url":null}],"created_at":"2026-06-05T11:29:40.000Z","created_at_i":1780658980,"id":48410967,"options":[],"parent_id":48410857,"points":null,"story_id":48410589,"text":"How is a static blog being scraped a problem? Do you not use a CDN?","title":null,"type":"comment","url":null},{"author":"nickserv","children":[],"created_at":"2026-06-05T11:59:12.000Z","created_at_i":1780660752,"id":48411218,"options":[],"parent_id":48410857,"points":null,"story_id":48410589,"text":"I'm seeing quite a bit of request for these on my work's GitBook documentation site.
But perhaps these are developers specifically targeting these pages to feed whatever LLM they are using.","title":null,"type":"comment","url":null},{"author":"0123456789ABCDE","children":[{"author":"m4tthumphrey","children":[{"author":"0123456789ABCDE","children":[],"created_at":"2026-06-05T12:47:01.000Z","created_at_i":1780663621,"id":48411676,"options":[],"parent_id":48411610,"points":null,"story_id":48410589,"text":"i stand corrected, but what was clear to you, obviously was not clear to me.","title":null,"type":"comment","url":null}],"created_at":"2026-06-05T12:40:37.000Z","created_at_i":1780663237,"id":48411610,"options":[],"parent_id":48411275,"points":null,"story_id":48410589,"text":"OP clearly meant that the AI players are not reading and/or honouring llms.txt of other websites when scraping.","title":null,"type":"comment","url":null}],"created_at":"2026-06-05T12:05:16.000Z","created_at_i":1780661116,"id":48411275,"options":[],"parent_id":48410857,"points":null,"story_id":48410589,"text":"> I can definitively say llms.txt is not used by any AI players.
https://developers.openai.com/llms.txt\n https://docs.anthropic.com/llms.txt\n https://geminicli.com/llms.txt\n https://github.com/llms.txt\n https://docs.aws.amazon.com/llms.txt\n https://openrouter.ai/docs/llms.txtSo it get even stranger, I am the only one reading those /llms.txt ...","title":null,"type":"comment","url":null}],"created_at":"2026-06-05T11:17:58.000Z","created_at_i":1780658278,"id":48410857,"options":[],"parent_id":48410783,"points":null,"story_id":48410589,"text":"I can definitively say llms.txt is not used by any AI players. I run a blogging platform with around 80k blogs and /llms.txt is not requested by anything (other than humans checking to see if there's an llms.txt path).
All regular pages are aggressively scraped to the extent it's a problem I have to consistently manage, but not llms.txt.","title":null,"type":"comment","url":null},{"author":"0123456789ABCDE","children":[],"created_at":"2026-06-05T12:07:34.000Z","created_at_i":1780661254,"id":48411298,"options":[],"parent_id":48410783,"points":null,"story_id":48410589,"text":"yes, they do.
anyone who's, even slightly, clued into how agents access documentation, has been making changes to their pages. ex: https://searchtxt-web.fly.dev/search?q=aws","title":null,"type":"comment","url":null},{"author":"solumos","children":[{"author":"christoff12","children":[],"created_at":"2026-06-05T14:56:31.000Z","created_at_i":1780671391,"id":48413394,"options":[],"parent_id":48413112,"points":null,"story_id":48410589,"text":"This is interesting. I should start incorporating this -- it couldn't hurt to do both.","title":null,"type":"comment","url":null},{"author":"kamma4434","children":[{"author":"solumos","children":[],"created_at":"2026-06-05T16:28:50.000Z","created_at_i":1780676930,"id":48414795,"options":[],"parent_id":48413764,"points":null,"story_id":48410589,"text":"If you want to see what that looks like, I one-shot a browser with Claude that does it[0]. Docs pages are early adopters to this[1][2], so that AI agents can better handle tasks.
[0] - https://github.com/solumos/md-browse
[1] - https://docs.stripe.com
[2] - https://vercel.com/docs","title":null,"type":"comment","url":null},{"author":"sunshine-o","children":[],"created_at":"2026-06-05T18:31:47.000Z","created_at_i":1780684307,"id":48416405,"options":[],"parent_id":48413764,"points":null,"story_id":48410589,"text":"I just found out Cloudflare supports real-time html to md conversion [0]
- [0] https://blog.cloudflare.com/markdown-for-agents/#convert-htm...","title":null,"type":"comment","url":null}],"created_at":"2026-06-05T15:21:47.000Z","created_at_i":1780672907,"id":48413764,"options":[],"parent_id":48413112,"points":null,"story_id":48410589,"text":"Now, it would be super cool to get markdown and zero javascript bundles\u2026","title":null,"type":"comment","url":null}],"created_at":"2026-06-05T14:34:13.000Z","created_at_i":1780670053,"id":48413112,"options":[],"parent_id":48410783,"points":null,"story_id":48410589,"text":"No, requesting "Accept: text/markdown" in the headers and returning markdown is the more agreed upon standard at this point.[0]
[0] - https://acceptmarkdown.com/","title":null,"type":"comment","url":null}],"created_at":"2026-06-05T11:07:48.000Z","created_at_i":1780657668,"id":48410783,"options":[],"parent_id":48410589,"points":null,"story_id":48410589,"text":"Does any of the LLM providers actually use llms.txt?
If I remember correctly this "standard" was setup by someone but without involvement of any of the major AI players.","title":null,"type":"comment","url":null},{"author":"mohamedkoubaa","children":[],"created_at":"2026-06-05T11:13:22.000Z","created_at_i":1780658002,"id":48410824,"options":[],"parent_id":48410589,"points":null,"story_id":48410589,"text":"It just hasn't been gamed yet","title":null,"type":"comment","url":null},{"author":"skywalqer","children":[{"author":"0123456789ABCDE","children":[],"created_at":"2026-06-05T12:01:22.000Z","created_at_i":1780660882,"id":48411237,"options":[],"parent_id":48410944,"points":null,"story_id":48410589,"text":"https://searchtxt-web.fly.dev","title":null,"type":"comment","url":null},{"author":"JimDabell","children":[],"created_at":"2026-06-05T12:55:12.000Z","created_at_i":1780664112,"id":48411775,"options":[],"parent_id":48410944,"points":null,"story_id":48410589,"text":"Putting it in .well-known/ was immediately raised as an issue from the beginning; it\u2019s issue #2 in fact:
https://github.com/AnswerDotAI/llms-txt/issues/2
It\u2019s been completely ignored ever since.","title":null,"type":"comment","url":null}],"created_at":"2026-06-05T11:27:31.000Z","created_at_i":1780658851,"id":48410944,"options":[],"parent_id":48410589,"points":null,"story_id":48410589,"text":"Why didn't they place it in .well-known? Also, I couldn't find a website that has it.","title":null,"type":"comment","url":null},{"author":"jordemort","children":[],"created_at":"2026-06-05T11:44:44.000Z","created_at_i":1780659884,"id":48411095,"options":[],"parent_id":48410589,"points":null,"story_id":48410589,"text":"no","title":null,"type":"comment","url":null},{"author":"marand23","children":[{"author":"imhoguy","children":[],"created_at":"2026-06-07T18:05:43.000Z","created_at_i":1780855543,"id":48437210,"options":[],"parent_id":48411135,"points":null,"story_id":48410589,"text":"It can be even better. An agent with Playwright MCP can browse even trickiest website structure and simply say what it "sees" and ask the user what to do next, voice or Braille.","title":null,"type":"comment","url":null}],"created_at":"2026-06-05T11:50:03.000Z","created_at_i":1780660203,"id":48411135,"options":[],"parent_id":48410589,"points":null,"story_id":48410589,"text":"I never thought about it before now but the llm era could be a form of renaissance for blind people on the Internet. An alternative web where functionality of every page is described in short but detailed text instead of extremely verbose and non-linear html tree structure.","title":null,"type":"comment","url":null},{"author":"tacostakohashi","children":[],"created_at":"2026-06-05T12:01:49.000Z","created_at_i":1780660909,"id":48411241,"options":[],"parent_id":48410589,"points":null,"story_id":48410589,"text":"Pretty much.
There is an enshittification cycle at work. The web used to be good, predominately text, and useful, 25 years ago. Then... slowly... we added javascript, then AJAX, CSS, flash, interstitials, popups, marketing, social media, algorithms, doomscrolling... gradually but surely turn it into the unusable cesspool that it is today.
Now we have AI! I think a big part of its utility is that it gets us back to text/information, and lets us bypass all the "beautiful" design / nonsense on the material it is trained on.
However, AI is just beginning its enshittification cycle - now that it has a critical mass of users, it is an irresistible target to start slowly adding ads, misinformation, conspiracy theories, and whatever else people can dream up, until it also becomes unusable and the cycle repeats.","title":null,"type":"comment","url":null},{"author":"DeathArrow","children":[{"author":"fxwin","children":[],"created_at":"2026-06-05T12:34:44.000Z","created_at_i":1780662884,"id":48411548,"options":[],"parent_id":48411513,"points":null,"story_id":48410589,"text":"here: https://llmstxt.org/\nand obviously it doesn't automatically produce markdown, it's something the website needs to provide (e.g. https://pydantic.dev/llms.txt)","title":null,"type":"comment","url":null}],"created_at":"2026-06-05T12:30:59.000Z","created_at_i":1780662659,"id":48411513,"options":[],"parent_id":48410589,"points":null,"story_id":48410589,"text":"I tried it: https://news.ycombinator.com/item?id=48410589`/llm.txt
Result: no such item.
From where do you got the idea that adding /llm.txt to urls will produce markdown?","title":null,"type":"comment","url":null},{"author":"gobdovan","children":[{"author":"sunshine-o","children":[{"author":"gobdovan","children":[],"created_at":"2026-06-05T14:49:49.000Z","created_at_i":1780670989,"id":48413302,"options":[],"parent_id":48412649,"points":null,"story_id":48410589,"text":"Thanks, the comparison hit like a bag of bricks.","title":null,"type":"comment","url":null}],"created_at":"2026-06-05T13:56:15.000Z","created_at_i":1780667775,"id":48412649,"options":[],"parent_id":48411602,"points":null,"story_id":48410589,"text":"llms.txt usually includes a clear sitemap and description of information available on a site.
There are also clear definition of the restful scheme and API/data access options.
One very basic example would be the weather channel https://weather.com/llms.txt","title":null,"type":"comment","url":null}],"created_at":"2026-06-05T12:40:13.000Z","created_at_i":1780663213,"id":48411602,"options":[],"parent_id":48410589,"points":null,"story_id":48410589,"text":"Not really, but sounds interesting. Would you care to share some sites that offer better llms.txt than main web page? Or talk about some piece of info you easily found on llms.txt that was hard to navigate to on the regular website?","title":null,"type":"comment","url":null},{"author":"croes","children":[{"author":"user568439","children":[],"created_at":"2026-06-05T15:10:06.000Z","created_at_i":1780672206,"id":48413572,"options":[],"parent_id":48411615,"points":null,"story_id":48410589,"text":"That's what I was thinking... Now spammers will add hidden prompts or things worse than that for the LLMs...","title":null,"type":"comment","url":null}],"created_at":"2026-06-05T12:41:16.000Z","created_at_i":1780663276,"id":48411615,"options":[],"parent_id":48410589,"points":null,"story_id":48410589,"text":"No, the spammers are just at the beginning of ruining that too
https://news.ycombinator.com/item?id=48411569
BTW why should Chrome even consider rendering a .txt file as markdown?","title":null,"type":"comment","url":null}],"created_at":"2026-06-05T10:43:28.000Z","created_at_i":1780656208,"id":48410589,"options":[],"parent_id":null,"points":36,"story_id":48410589,"text":"I got really tired, as a human, of parsing the standard marketing heavy web we have today.\nI've always loved the simplicity of gopher and gemini web.
Recently I found myself manually adding `/llm.txt` to most websites I visit because I find the content for LLMs strait to the point and clear. \nThe only annoyance is web browsers like chrome do not render the markdown.
So could the AI revolution actually fix the web for humans as a side effect?
Do you find yourself doing the same?","title":"Ask HN: Is the web for machines (/llm.txt) the one we wished we had as humans?","type":"story","url":null}