A comprehensive, community-driven resource dedicated to Crawl4AI, the leading open-source LLM-friendly web crawler. This site provides production-ready integration guides for Cursor MCP, n8n automation, and Docker deployments. It features detailed performance comparisons against proprietary alternatives like Firecrawl, helping developers build scalable, self-hosted data pipelines for AI agents and RAG systems."},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crawl4ai"],"value":"Show HN: Crawl4AI \u2013 Open-Source Web Crawler for LLMs and Structured Data"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crawl4ai"],"value":"https://crawl4ai.dev"}},"_tags":["story","author_medivhX","story_46646798","show_hn"],"author":"medivhX","children":[46646807],"created_at":"2026-01-16T14:39:50Z","created_at_i":1768574390,"num_comments":1,"objectID":"46646798","points":2,"story_id":46646798,"story_text":"Crawl4AI Unofficial Guides: Master High-Performance AI Web Scraping
A comprehensive, community-driven resource dedicated to Crawl4AI, the leading open-source LLM-friendly web crawler. This site provides production-ready integration guides for Cursor MCP, n8n automation, and Docker deployments. It features detailed performance comparisons against proprietary alternatives like Firecrawl, helping developers build scalable, self-hosted data pipelines for AI agents and RAG systems.","title":"Show HN: Crawl4AI \u2013 Open-Source Web Crawler for LLMs and Structured Data","updated_at":"2026-03-05T23:25:54Z","url":"https://crawl4ai.dev"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"synergy20"},"title":{"fullyHighlighted":true,"matchLevel":"full","matchedWords":["crawl4ai"],"value":"Crawl4AI"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crawl4ai"],"value":"https://github.com/unclecode/crawl4ai"}},"_tags":["story","author_synergy20","story_41747108"],"author":"synergy20","created_at":"2024-10-05T01:46:45Z","created_at_i":1728092805,"num_comments":0,"objectID":"41747108","points":2,"story_id":41747108,"title":"Crawl4AI","updated_at":"2024-10-05T05:52:39Z","url":"https://github.com/unclecode/crawl4ai"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"mpfect"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crawl4ai"],"value":"Crawl4AI: Open-source web crawler and scraper for LLMs"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crawl4ai"],"value":"https://github.com/unclecode/crawl4ai"}},"_tags":["story","author_mpfect","story_48857017"],"author":"mpfect","created_at":"2026-07-10T07:51:37Z","created_at_i":1783669897,"num_comments":0,"objectID":"48857017","points":1,"story_id":48857017,"title":"Crawl4AI: Open-source web crawler and scraper for LLMs","updated_at":"2026-07-10T07:53:02Z","url":"https://github.com/unclecode/crawl4ai"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"zoooey"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crawl4ai"],"value":"Crawl4AI: Open-Source LLM Friendly Web Crawler and Scraper"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crawl4ai"],"value":"https://github.com/unclecode/crawl4ai"}},"_tags":["story","author_zoooey","story_45963180"],"author":"zoooey","created_at":"2025-11-18T09:49:52Z","created_at_i":1763459392,"num_comments":0,"objectID":"45963180","points":1,"story_id":45963180,"title":"Crawl4AI: Open-Source LLM Friendly Web Crawler and Scraper","updated_at":"2026-03-05T23:01:55Z","url":"https://github.com/unclecode/crawl4ai"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"basic_banana"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crawl4ai"],"value":"Crawl4ai \u2013 open-source web crawler for your AI agent"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crawl4ai"],"value":"https://github.com/unclecode/crawl4ai"}},"_tags":["story","author_basic_banana","story_43048990"],"author":"basic_banana","created_at":"2025-02-14T15:00:35Z","created_at_i":1739545235,"num_comments":0,"objectID":"43048990","points":1,"story_id":43048990,"title":"Crawl4ai \u2013 open-source web crawler for your AI agent","updated_at":"2025-02-14T15:03:19Z","url":"https://github.com/unclecode/crawl4ai"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"schoblaska"},"story_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crawl4ai"],"value":"Hey HN! Over the weekend (leaning heavily on Opus 4.5) I wrote Jargon - an AI-managed zettelkasten that reads articles, papers, and YouTube videos, extracts the key ideas, and automatically links related concepts together.
Demo video: https://youtu.be/W7ejMqZ6EUQ
Repo: https://github.com/schoblaska/jargon
You can paste an article, PDF link, or YouTube video to parse, or ask questions directly and it'll find its own content. Sources get summarized, broken into insight cards, and embedded for semantic search. Similar ideas automatically cluster together. Each insight can spawn research threads - questions that trigger web searches to pull in related content, which flows through the same pipeline.
You can explore the graph of linked ideas directly, or ask questions and it'll RAG over your whole library plus fresh web results.
Jargon uses Rails + Hotwire with Falcon for async processing, pgvector for embeddings, Exa for neural web search, crawl4ai as a fallback scraper, and pdftotext for academic papers."},"title":{"matchLevel":"none","matchedWords":[],"value":"Show HN: An AI zettelkasten that extracts ideas from articles, videos, and PDFs"},"url":{"matchLevel":"none","matchedWords":[],"value":"https://github.com/schoblaska/jargon"}},"_tags":["story","author_schoblaska","story_46110897","show_hn"],"author":"schoblaska","children":[46114054,46114344,46114495,46116921,46118108,46119359,46130474,46172230],"created_at":"2025-12-01T18:20:46Z","created_at_i":1764613246,"num_comments":9,"objectID":"46110897","points":38,"story_id":46110897,"story_text":"Hey HN! Over the weekend (leaning heavily on Opus 4.5) I wrote Jargon - an AI-managed zettelkasten that reads articles, papers, and YouTube videos, extracts the key ideas, and automatically links related concepts together.
Demo video: https://youtu.be/W7ejMqZ6EUQ
Repo: https://github.com/schoblaska/jargon
You can paste an article, PDF link, or YouTube video to parse, or ask questions directly and it'll find its own content. Sources get summarized, broken into insight cards, and embedded for semantic search. Similar ideas automatically cluster together. Each insight can spawn research threads - questions that trigger web searches to pull in related content, which flows through the same pipeline.
You can explore the graph of linked ideas directly, or ask questions and it'll RAG over your whole library plus fresh web results.
Jargon uses Rails + Hotwire with Falcon for async processing, pgvector for embeddings, Exa for neural web search, crawl4ai as a fallback scraper, and pdftotext for academic papers.","title":"Show HN: An AI zettelkasten that extracts ideas from articles, videos, and PDFs","updated_at":"2026-03-05T23:04:46Z","url":"https://github.com/schoblaska/jargon"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"mxfeinberg"},"story_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crawl4ai"],"value":"As a long time data scientist and engineer, I've had to write a couple of quick and dirty scrapers and bots over the years using selenium and more recently playwright. I haven't really been tracking it, but I've also been reading about the crawl4ai project.
With the explosion of AI agents, I've been playing around with building agentic scrapers that can simply be given a prompt and a target site and are able to return structured data in a specified format. I've also been playing around with adding in steps that have a different model/step attempt to define the structured format dynamically.
However, as with most AI projects, the token consumption can scale pretty aggressively.
Has anyone else been working on similar projects? Would people realistically pay $0.025 to $0.03 per request?"},"title":{"matchLevel":"none","matchedWords":[],"value":"Ask HN: Is there a market for agentic scraping tools?"}},"_tags":["story","author_mxfeinberg","story_44474800","ask_hn"],"author":"mxfeinberg","children":[44474909,44475773],"created_at":"2025-07-05T19:07:46Z","created_at_i":1751742466,"num_comments":2,"objectID":"44474800","points":3,"story_id":44474800,"story_text":"As a long time data scientist and engineer, I've had to write a couple of quick and dirty scrapers and bots over the years using selenium and more recently playwright. I haven't really been tracking it, but I've also been reading about the crawl4ai project.
With the explosion of AI agents, I've been playing around with building agentic scrapers that can simply be given a prompt and a target site and are able to return structured data in a specified format. I've also been playing around with adding in steps that have a different model/step attempt to define the structured format dynamically.
However, as with most AI projects, the token consumption can scale pretty aggressively.
Has anyone else been working on similar projects? Would people realistically pay $0.025 to $0.03 per request?","title":"Ask HN: Is there a market for agentic scraping tools?","updated_at":"2026-07-22T17:14:57Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"dhruv_ahuja"},"story_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crawl4ai"],"value":"I am curious, what are people and organisations building with web crawlers, especially since now all of them seem to have LLM support for further refining the output from crawled pages.
Open source projects like crawl4ai[1] have a crazy amount of stars (~59k). This got me curious.
https://github.com/unclecode/crawl4ai"},"title":{"matchLevel":"none","matchedWords":[],"value":"Ask HN: What do you use (AI based) web crawlers for?"}},"_tags":["story","author_dhruv_ahuja","story_46733548","ask_hn"],"author":"dhruv_ahuja","created_at":"2026-01-23T15:21:22Z","created_at_i":1769181682,"num_comments":0,"objectID":"46733548","points":3,"story_id":46733548,"story_text":"I am curious, what are people and organisations building with web crawlers, especially since now all of them seem to have LLM support for further refining the output from crawled pages.
Open source projects like crawl4ai[1] have a crazy amount of stars (~59k). This got me curious.
https://github.com/unclecode/crawl4ai","title":"Ask HN: What do you use (AI based) web crawlers for?","updated_at":"2026-03-05T23:24:06Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"Roark66"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crawl4ai"],"value":"I've been using Qwen3.6 models locally for a couple of weeks. Both the A3B moe and the dense variant. The moe works well in Librechat combined with my local search/Web retrieval system. All components use open source projects such as SearXNG, Crawl4AI, MetaMCP, Jina rerank, but all needed quite a bit of coding to work nicely together.
I get 140 tok/s on short prompts on an rtx3090 on the qwen3.6 moe which makes is easily 4x the speed of Chatgpt or Claude doing Web research.
But it is a much simpler model. It is only good for simple queries, usually I search for cheapest product in stock in my country available online and stuff like that.
I use the dense model for planning and such, but on its own it is much inferior to for example opus. It needs careful pipelines that check facts and such and in such harness it can be used for mamy tasks."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"DeepSeek costs OpenCode Go user $1.14/day; dual DGX breaks even in 24 years"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://twitter.com/thdxr/status/2086599224674681242"}},"_tags":["comment","author_Roark66","story_49242728"],"author":"Roark66","comment_text":"I've been using Qwen3.6 models locally for a couple of weeks. Both the A3B moe and the dense variant. The moe works well in Librechat combined with my local search/Web retrieval system. All components use open source projects such as SearXNG, Crawl4AI, MetaMCP, Jina rerank, but all needed quite a bit of coding to work nicely together.
I get 140 tok/s on short prompts on an rtx3090 on the qwen3.6 moe which makes is easily 4x the speed of Chatgpt or Claude doing Web research.
But it is a much simpler model. It is only good for simple queries, usually I search for cheapest product in stock in my country available online and stuff like that.
I use the dense model for planning and such, but on its own it is much inferior to for example opus. It needs careful pipelines that check facts and such and in such harness it can be used for mamy tasks.","created_at":"2026-08-10T14:21:53Z","created_at_i":1786371713,"objectID":"49244098","parent_id":49243824,"story_id":49242728,"story_title":"DeepSeek costs OpenCode Go user $1.14/day; dual DGX breaks even in 24 years","story_url":"https://twitter.com/thdxr/status/2086599224674681242","updated_at":"2026-08-10T15:27:53Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"Roark66"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crawl4ai"],"value":"The author mentions lack of good Web search. I've been using slightly modified crawl4ai and searXNG together with firebase for the rare sites that insist on throwing wrenches in the works of my LLMs.
I also have my fork of metamcp that replaces firebase MCP spec with my own that tells the model to use crawl4ai and SearXNG instead.
I've been using this wia Librechat with every commercial and open weight model I tested.
The search is way better than OpenAI and what ClaudeCode uses, but Gemini is way faster. That will change soon as I'm planning to put these instances in a DC with gigabit pipe.
Firebase is not cheap, but it retrieves everything, bypasses captchas and so on.... As long as one uses it for 1% of Web queries the cost is manageable."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"GLM 5.2 and the coming AI margin collapse"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://martinalderson.com/posts/the-upcoming-ai-margin-collapse-part-1-glm-5-2/"}},"_tags":["comment","author_Roark66","story_48809877"],"author":"Roark66","comment_text":"The author mentions lack of good Web search. I've been using slightly modified crawl4ai and searXNG together with firebase for the rare sites that insist on throwing wrenches in the works of my LLMs.
I also have my fork of metamcp that replaces firebase MCP spec with my own that tells the model to use crawl4ai and SearXNG instead.
I've been using this wia Librechat with every commercial and open weight model I tested.
The search is way better than OpenAI and what ClaudeCode uses, but Gemini is way faster. That will change soon as I'm planning to put these instances in a DC with gigabit pipe.
Firebase is not cheap, but it retrieves everything, bypasses captchas and so on.... As long as one uses it for 1% of Web queries the cost is manageable.","created_at":"2026-07-07T13:54:26Z","created_at_i":1783432466,"objectID":"48817896","parent_id":48809877,"story_id":48809877,"story_title":"GLM 5.2 and the coming AI margin collapse","story_url":"https://martinalderson.com/posts/the-upcoming-ai-margin-collapse-part-1-glm-5-2/","updated_at":"2026-07-07T14:07:37Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"jordan_gibbs"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crawl4ai"],"value":"HyperResearch is a simple Claude Code skill harness that outperforms every deep research framework.
HyperResearch surpasses OpenAI, Google, and NVIDIA's offerings in the agentic search space based on DeepResearch Bench. It's open-source, installable with a single command, and uses your CC subscription, so you don't have to pay for OpenAI or Gemini Pro.
It uses a 16-step pipeline that creates a searchable, persistent knowledge store during each session that can be built upon in later searches. I designed it to align with the original user prompt as closely as possible, while incorporating built-in fact-checking, adversarial review, and breadth and depth-investigating capabilities.
This is a generalized framework; you can use it for any large-scale research task, from developing a trading strategy for a specific stock to analyzing competitor products to understanding the current state of the art in LLM architecture.
It uses crawl4ai (an open-source LLM search tool) to capture a wider breadth of information than the standard websearch tool is capable of. You can also configure authenticated sessions, meaning that LinkedIn, Twitter, etc., are now fair game for agentic search.
If anyone wants to port it to Codex, be my guest!"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Converting Claude Code into the top scoring deep research agent"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://github.com/jordan-gibbs/hyperresearch"}},"_tags":["comment","author_jordan_gibbs","story_47953372"],"author":"jordan_gibbs","comment_text":"HyperResearch is a simple Claude Code skill harness that outperforms every deep research framework.
HyperResearch surpasses OpenAI, Google, and NVIDIA's offerings in the agentic search space based on DeepResearch Bench. It's open-source, installable with a single command, and uses your CC subscription, so you don't have to pay for OpenAI or Gemini Pro.
It uses a 16-step pipeline that creates a searchable, persistent knowledge store during each session that can be built upon in later searches. I designed it to align with the original user prompt as closely as possible, while incorporating built-in fact-checking, adversarial review, and breadth and depth-investigating capabilities.
This is a generalized framework; you can use it for any large-scale research task, from developing a trading strategy for a specific stock to analyzing competitor products to understanding the current state of the art in LLM architecture.
It uses crawl4ai (an open-source LLM search tool) to capture a wider breadth of information than the standard websearch tool is capable of. You can also configure authenticated sessions, meaning that LinkedIn, Twitter, etc., are now fair game for agentic search.
If anyone wants to port it to Codex, be my guest!","created_at":"2026-04-29T19:39:30Z","created_at_i":1777491570,"objectID":"47953373","parent_id":47953372,"story_id":47953372,"story_title":"Converting Claude Code into the top scoring deep research agent","story_url":"https://github.com/jordan-gibbs/hyperresearch","updated_at":"2026-04-29T19:42:36Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"breed101"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crawl4ai"],"value":"replaces claude code's webfetch - claude code webfetch hangs indefinitely, does not have timeout. does not pierce dom js-rendered. the crawl4ai, firecrawl tools do not pierce shadow dom. the kiss solution, just select all copy paste the display text. this returns actions available, text offset / limits. great for the agent."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Because Webfetch Sucks"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://github.com/bxxd/mcp-fetch-ux"}},"_tags":["comment","author_breed101","story_47874773"],"author":"breed101","comment_text":"replaces claude code's webfetch - claude code webfetch hangs indefinitely, does not have timeout. does not pierce dom js-rendered. the crawl4ai, firecrawl tools do not pierce shadow dom. the kiss solution, just select all copy paste the display text. this returns actions available, text offset / limits. great for the agent.","created_at":"2026-04-23T12:07:56Z","created_at_i":1776946076,"objectID":"47874774","parent_id":47874773,"story_id":47874773,"story_title":"Because Webfetch Sucks","story_url":"https://github.com/bxxd/mcp-fetch-ux","updated_at":"2026-04-23T12:10:10Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"bredren"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crawl4ai"],"value":"Why not just write a skill and script that calls crawl4ai or similar and do this using Claude code?
You can store the page as markdown for future sessions, mash the data w other context, you name it.
The web Claude is incredibly limited both in capability and workflow integration. Doesn\u2019t matter if you\u2019re dealing with bids from arbor contractors or researching solutions for a DB problem."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Switch to Claude without starting over"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://claude.com/import-memory"}},"_tags":["comment","author_bredren","story_47204571"],"author":"bredren","children":[47209190,47209363],"comment_text":"Why not just write a skill and script that calls crawl4ai or similar and do this using Claude code?
You can store the page as markdown for future sessions, mash the data w other context, you name it.
The web Claude is incredibly limited both in capability and workflow integration. Doesn\u2019t matter if you\u2019re dealing with bids from arbor contractors or researching solutions for a DB problem.","created_at":"2026-03-01T17:52:50Z","created_at_i":1772387570,"objectID":"47208961","parent_id":47206490,"story_id":47204571,"story_title":"Switch to Claude without starting over","story_url":"https://claude.com/import-memory","updated_at":"2026-03-05T23:39:06Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"Roark66"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crawl4ai"],"value":"Cloudflare is ridiculous. I can't even open it using Cromite (privacy enhanced, but not over the top, android browser).
I get:
blog.adafruit.com\nVerifying you are human. This may take a few seconds.
blog.adafruit.com needs to review the security of your connection before proceeding.
And this hangs forever. What difference does it make if I access this site using a browser (blocked anyway) or I asked my LLM to fetch the content? I bet my LLM coukd get it anyway as I'm using basic local scraping with firecrawl for backup. So my LLM if it fails to retrieve using my basic local crawl4ai will use my paid firecrawl api and those guys can scrape EVERYTHING.
I do not understand why do you (as a site owner) care? Are these bots generating so much traffic? Can you set it up to serve text only version to them then?"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"New York\u2019s budget bill would require \u201cblocking technology\u201d on all 3D printers"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://blog.adafruit.com/2026/02/03/new-york-wants-to-ctrlaltdelete-your-3d-printer/"}},"_tags":["comment","author_Roark66","story_46872540"],"author":"Roark66","children":[46887624,46887964],"comment_text":"Cloudflare is ridiculous. I can't even open it using Cromite (privacy enhanced, but not over the top, android browser).
I get:
blog.adafruit.com\nVerifying you are human. This may take a few seconds.
blog.adafruit.com needs to review the security of your connection before proceeding.
And this hangs forever. What difference does it make if I access this site using a browser (blocked anyway) or I asked my LLM to fetch the content? I bet my LLM coukd get it anyway as I'm using basic local scraping with firecrawl for backup. So my LLM if it fails to retrieve using my basic local crawl4ai will use my paid firecrawl api and those guys can scrape EVERYTHING.
I do not understand why do you (as a site owner) care? Are these bots generating so much traffic? Can you set it up to serve text only version to them then?","created_at":"2026-02-04T16:02:21Z","created_at_i":1770220941,"objectID":"46887472","parent_id":46886301,"story_id":46872540,"story_title":"New York\u2019s budget bill would require \u201cblocking technology\u201d on all 3D printers","story_url":"https://blog.adafruit.com/2026/02/03/new-york-wants-to-ctrlaltdelete-your-3d-printer/","updated_at":"2026-06-02T20:39:09Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"medivhX"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crawl4ai"],"value":"Hi HN,
Crawl4AI is an amazing open-source library that solves many LLM-scraping headaches, but I found that new developers often struggle with production configurations\u2014specifically how to use it with MCP servers for Cursor, or how to bridge it with automation tools like n8n.
I built crawl4ai.dev as a community-driven documentation hub to fill these gaps. It includes:
One-click Docker setups for n8n/FastAPI.
Production-ready MCP server guides for Cursor & Claude.
Cost/performance benchmarks vs proprietary tools like Firecrawl.
The goal is to help everyone build self-hosted, affordable AI data pipelines. I'd love to hear your feedback on the guides or what other integrations you'd like to see documented!"},"story_title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crawl4ai"],"value":"Show HN: Crawl4AI \u2013 Open-Source Web Crawler for LLMs and Structured Data"},"story_url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crawl4ai"],"value":"https://crawl4ai.dev"}},"_tags":["comment","author_medivhX","story_46646798"],"author":"medivhX","comment_text":"Hi HN,
Crawl4AI is an amazing open-source library that solves many LLM-scraping headaches, but I found that new developers often struggle with production configurations\u2014specifically how to use it with MCP servers for Cursor, or how to bridge it with automation tools like n8n.
I built crawl4ai.dev as a community-driven documentation hub to fill these gaps. It includes:
One-click Docker setups for n8n/FastAPI.
Production-ready MCP server guides for Cursor & Claude.
Cost/performance benchmarks vs proprietary tools like Firecrawl.
The goal is to help everyone build self-hosted, affordable AI data pipelines. I'd love to hear your feedback on the guides or what other integrations you'd like to see documented!","created_at":"2026-01-16T14:41:09Z","created_at_i":1768574469,"objectID":"46646807","parent_id":46646798,"story_id":46646798,"story_title":"Show HN: Crawl4AI \u2013 Open-Source Web Crawler for LLMs and Structured Data","story_url":"https://crawl4ai.dev","updated_at":"2026-03-05T23:25:54Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"Roark66"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crawl4ai"],"value":"I'm glad the author clarified he wants to prevent his instance from crashing not simply "block robots and allow humans".
I think the idea that you can block bots and allow humans is fallacious.
We should focus on a specific behaviour that causes problems (like making a bajillion requests one for each commit, instead of cloning the repo). To fix this we should block clients that work in such ways. If these bots learn to request at a reasonable pace why cares if they are bots, humans, bots under a control of an individual human, bots owned by a huge company scraping for training data? Once you make your code (or anything else) public, then trying to limit access to only a certain class of consumers is a waste of effort.
Also, perhaps I'm biased, because I run a searXNG and Crawl4AI (and few ancillaries like jina rerank etc) in my homelab so I can tell my AI to perform live internet searches as well as it can get any website. For code it has a way to clone stuff, but for things like issues, discussions, PRs it goes mostly to GitHub.
I like that my AI can browse almost like me. I think this is the future way to consume a lot of the web (except sites like this one that are an actual pleasure to use).
The models sometimes hit sites they can't fetch. For this I use Firecrawl. I use MCP proxy that lets me rewrite the tool descriptions so my models get access to both my local Crawl4ai and hosted (and rather expensive)firecrawl, but they are told to use Firecrawl as last resort.
The more people use these kinds of solutions the more incentive there will be for sites not to block users that use automation. Of course they will have to rely on alternative monetisation methods, but I think eventually these stupid capchas will disappear and reasonable rate limiting will prevail."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"How I protect my Forgejo instance from AI web crawlers"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://her.esy.fun/posts/0031-how-i-protect-my-forgejo-instance-from-ai-web-crawlers/index.html"}},"_tags":["comment","author_Roark66","story_46345205"],"author":"Roark66","children":[46353065,46353158,46353924],"comment_text":"I'm glad the author clarified he wants to prevent his instance from crashing not simply "block robots and allow humans".
I think the idea that you can block bots and allow humans is fallacious.
We should focus on a specific behaviour that causes problems (like making a bajillion requests one for each commit, instead of cloning the repo). To fix this we should block clients that work in such ways. If these bots learn to request at a reasonable pace why cares if they are bots, humans, bots under a control of an individual human, bots owned by a huge company scraping for training data? Once you make your code (or anything else) public, then trying to limit access to only a certain class of consumers is a waste of effort.
Also, perhaps I'm biased, because I run a searXNG and Crawl4AI (and few ancillaries like jina rerank etc) in my homelab so I can tell my AI to perform live internet searches as well as it can get any website. For code it has a way to clone stuff, but for things like issues, discussions, PRs it goes mostly to GitHub.
I like that my AI can browse almost like me. I think this is the future way to consume a lot of the web (except sites like this one that are an actual pleasure to use).
The models sometimes hit sites they can't fetch. For this I use Firecrawl. I use MCP proxy that lets me rewrite the tool descriptions so my models get access to both my local Crawl4ai and hosted (and rather expensive)firecrawl, but they are told to use Firecrawl as last resort.
The more people use these kinds of solutions the more incentive there will be for sites not to block users that use automation. Of course they will have to rely on alternative monetisation methods, but I think eventually these stupid capchas will disappear and reasonable rate limiting will prevail.","created_at":"2025-12-22T10:46:50Z","created_at_i":1766400410,"objectID":"46353048","parent_id":46345205,"story_id":46345205,"story_title":"How I protect my Forgejo instance from AI web crawlers","story_url":"https://her.esy.fun/posts/0031-how-i-protect-my-forgejo-instance-from-ai-web-crawlers/index.html","updated_at":"2026-03-05T23:16:15Z"}],"hitsPerPage":20,"nbHits":10796,"nbPages":50,"page":0,"params":"query=crawl4ai&advancedSyntax=true&analyticsTags=backend","processingTimeMS":9,"processingTimingsMS":{"_request":{"roundTrip":20},"afterFetch":{"merge":{"mergeLoop":{"prepareNextHit":1,"total":1},"total":1},"total":1},"fetch":{"query":5,"scanning":1,"total":7},"total":9},"query":"crawl4ai","serverTimeMS":10}