With crul you can crawl and transform web pages into csv tables, explore and dynamically query APIs, filter and organize data, and push data sets to third party data lakes and analytics tools. Here\u2019s a demo video, we\u2019ve been told Nic sounds like John Mayer (lol) (https://www.crul.com/demo-video)
We\u2019ve personally struggled wrangling data from the web using puppeteer/playwright/selenium, jq or cobbling together python scripts, client libraries, and schedulers to consume APIs. The reality is that shit is hard, doesn\u2019t scale (classic blocking for-loop or async saturation), and comes with thorny maintenance/security issues. The tools we love to hate.
Crul\u2019s value prop is simple: Query any Webpage or API for free.
At its core, crul is based on the foundational linked nature of Web/API content. It consists of a purpose built map/expand/reduce engine for hierarchical Web/API content (kind of like postman but with a membership to Gold's Gym) with a familiar parser expression grammar that naturally gets the job done (and layered caching to make it quick to fix when it doesn\u2019t on the first try). There\u2019s a boatload of other features like domain policies, scheduler, checkpoints, templates, REST API, Web UI, vault, OAuth for third parties and 20+ stores to send your data to.
Our goal is to open source crul as time and resources permit. At the end of the day it\u2019s just the two of us trying to figure things out as we go! We\u2019re just getting started.
Crul is one bad mother#^@%*& and the web is finally yours!
Download crul for free as a Mac OS desktop application or as a Docker image (https://www.crul.com) and let us know if you love it or hate it. (https://forms.gle/5BXb5bLC1D5QG7i99) And come say hello to us on our slack channel - we\u2019re a friendly bunch! (https://crulinc.slack.com/)
Nic and Carl (https://www.crul.com/early-days)"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crul","show","hn"],"value":"Show HN: Crul \u2013 Query Any Webpage or API"},"url":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["crul"],"value":"https://www.crul.com/"}},"_tags":["story","author_portInit","story_34970917","show_hn"],"author":"portInit","children":[34970922,34971106,34972765,34972781,34973272,34973741,34974612,34974690,34974860,34975742,34975858,34976006,34977090,34977290,34977350,34978259,34978263,34979088,34979124,34979895,34980217,34986709,34990569],"created_at":"2023-02-28T16:12:58Z","created_at_i":1677600778,"num_comments":79,"objectID":"34970917","points":241,"story_id":34970917,"story_text":"Hi HN, we\u2019re Carl and Nic, the creators of crul (https://www.crul.com), and we\u2019ve been hard at work for the last year and a half building our dream of turning the web into a dataset. In a nutshell crul is a tool for querying and building web and api data feeds from anywhere to anywhere.
With crul you can crawl and transform web pages into csv tables, explore and dynamically query APIs, filter and organize data, and push data sets to third party data lakes and analytics tools. Here\u2019s a demo video, we\u2019ve been told Nic sounds like John Mayer (lol) (https://www.crul.com/demo-video)
We\u2019ve personally struggled wrangling data from the web using puppeteer/playwright/selenium, jq or cobbling together python scripts, client libraries, and schedulers to consume APIs. The reality is that shit is hard, doesn\u2019t scale (classic blocking for-loop or async saturation), and comes with thorny maintenance/security issues. The tools we love to hate.
Crul\u2019s value prop is simple: Query any Webpage or API for free.
At its core, crul is based on the foundational linked nature of Web/API content. It consists of a purpose built map/expand/reduce engine for hierarchical Web/API content (kind of like postman but with a membership to Gold's Gym) with a familiar parser expression grammar that naturally gets the job done (and layered caching to make it quick to fix when it doesn\u2019t on the first try). There\u2019s a boatload of other features like domain policies, scheduler, checkpoints, templates, REST API, Web UI, vault, OAuth for third parties and 20+ stores to send your data to.
Our goal is to open source crul as time and resources permit. At the end of the day it\u2019s just the two of us trying to figure things out as we go! We\u2019re just getting started.
Crul is one bad mother#^@%*& and the web is finally yours!
Download crul for free as a Mac OS desktop application or as a Docker image (https://www.crul.com) and let us know if you love it or hate it. (https://forms.gle/5BXb5bLC1D5QG7i99) And come say hello to us on our slack channel - we\u2019re a friendly bunch! (https://crulinc.slack.com/)
Nic and Carl (https://www.crul.com/early-days)","title":"Show HN: Crul \u2013 Query Any Webpage or API","updated_at":"2024-09-20T13:29:29Z","url":"https://www.crul.com/"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"jamesbhe"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crul","show","hn"],"value":"Show HN: cRUL-like network tool to test server timeouts with ease"},"url":{"matchLevel":"none","matchedWords":[],"value":"http://getmintbox.com/"}},"_tags":["story","author_jamesbhe","story_22784198","show_hn"],"author":"jamesbhe","created_at":"2020-04-05T06:55:23Z","created_at_i":1586069723,"num_comments":0,"objectID":"22784198","points":4,"story_id":22784198,"title":"Show HN: cRUL-like network tool to test server timeouts with ease","updated_at":"2024-09-20T05:57:14Z","url":"http://getmintbox.com/"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"docyes"},"story_text":{"matchLevel":"none","matchedWords":[],"value":"Semantically search Hacker News comments, assign a sentiment score to each comment and calculate the sentiment frequency distribution."},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crul","show","hn"],"value":"Show HN: Web/API retrieval for LLM's using Crul"},"url":{"matchLevel":"none","matchedWords":[],"value":"https://www.loom.com/share/51c60672021c407c90aeb8aed1dd9ade"}},"_tags":["story","author_docyes","story_40462930","show_hn"],"author":"docyes","created_at":"2024-05-24T04:58:14Z","created_at_i":1716526694,"num_comments":0,"objectID":"40462930","points":2,"story_id":40462930,"story_text":"Semantically search Hacker News comments, assign a sentiment score to each comment and calculate the sentiment frequency distribution.","title":"Show HN: Web/API retrieval for LLM's using Crul","updated_at":"2024-09-20T17:06:01Z","url":"https://www.loom.com/share/51c60672021c407c90aeb8aed1dd9ade"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"mdaniel"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["crul","show","hn"],"value":"I'm sorry to hear that your searches for that very specific name didn't provide the information you were looking for
its show hn: https://news.ycombinator.com/item?id=34970917
tfl: https://www.crul.com/"},"story_title":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["show","hn"],"value":"Show HN: Flyscrape \u2013 A standalone and scriptable web scraper in Go"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://github.com/philippta/flyscrape"}},"_tags":["comment","author_mdaniel","story_38230476"],"author":"mdaniel","comment_text":"I'm sorry to hear that your searches for that very specific name didn't provide the information you were looking for
its show hn: https://news.ycombinator.com/item?id=34970917
tfl: https://www.crul.com/","created_at":"2023-11-11T19:40:50Z","created_at_i":1699731650,"objectID":"38233755","parent_id":38232425,"story_id":38230476,"story_title":"Show HN: Flyscrape \u2013 A standalone and scriptable web scraper in Go","story_url":"https://github.com/philippta/flyscrape","updated_at":"2024-09-20T15:36:26Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"bhalp1"},"story_text":{"matchLevel":"none","matchedWords":[],"value":""},"title":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["show","hn"],"value":"Show HN: Same Picture of Dave Coulier Chrome Extension for pranking"},"url":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["crul"],"value":"https://chrome.google.com/webstore/detail/same-picture-of-dave-coul/cgeabdckhppkgainjcafbhljmemhaaek"}},"_tags":["story","author_bhalp1","story_7543213","show_hn"],"author":"bhalp1","created_at":"2014-04-06T21:53:37Z","created_at_i":1396821217,"num_comments":0,"objectID":"7543213","points":2,"story_id":7543213,"story_text":"","title":"Show HN: Same Picture of Dave Coulier Chrome Extension for pranking","updated_at":"2023-09-06T22:37:09Z","url":"https://chrome.google.com/webstore/detail/same-picture-of-dave-coul/cgeabdckhppkgainjcafbhljmemhaaek"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"mittermayr"},"story_text":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["crul","show"],"value":"My partner was let go after 15 years working tirelessly for one of the big five. She's probably the most resilient, rewarded and liked person that has survived and fought through this part of the business in the past decade, and while I am clearly biased, all her colleagues would most certainly agree. She's moved from IC to managing over 30 people and barely manages to use vacation days because she's always working.
She was let go not because of her own incompetence, failure or not fitting in, but purely out of rushed budget cuts (budgets were finalized, finalized again, and then they realized it's still not good enough).
The reason for those cuts, in this particular instance, was not the overwhelming rise of AI and/or getting more firing power for data centers, it was very clearly a long-visible path of failure two levels above, where one particular person continued to wreak havoc up over the past few years, and, ultimately, caused this cul-de-sac disaster.
The person at the center of this all is an older dude who consistenty has been on the wrong side of any decision he's made. I've only heard of a few things as it is (literally) none of my business, but it's been nothing but chaotic and ill-advised choices. He moved into this role from failing elsewhere, leaving the chaos there behind him, and over the past two years, now has caused yet another disaster in this new role.
My question is, and sort of the point of this post (which I understand is entirely philosophical): how do these people consistently survive their self-inflicted disasters? While her entire team (30+) has been let go now as a result of it all, the person who ran this into the ground continues onwards in the business?
They even made her fire and break the news to most of her team in weekly chunks recently, before telling her she's of course, also done.
I used to work for a big five too, but have been self-employed for over a decade now, so I may be too far out to understand, but it just puzzles me how certain people always manage to stay untainted admist the chaos they've caused, at the cost of everyone else. This used to be a thing I've heard over and over again from friends at Google, but it really seems to be very consistent thing elsewhere, too.
They're clearly not smart people, is it ruthlessness? Are they aware of the disaster they're navigating into and plan well ahead? It's a mystery to me, and it just can't be healthy for these companies to allow such folks to continue onwards."},"title":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["hn"],"value":"Ask HN: Why do corporate failures always seem to punish the wrong people?"}},"_tags":["story","author_mittermayr","story_49418675","ask_hn"],"author":"mittermayr","children":[49418955,49419013,49419065,49419093,49419116,49419136,49419138,49419238,49419240,49419241,49419255,49419362,49419400,49419410,49419415,49419427,49419429,49419445,49419558,49419567,49419583,49419592,49419653,49419676,49419690,49419708,49419811,49419979,49420127,49420183,49420204,49420343,49420380,49420468,49420595,49420607,49420844,49420974,49421037,49421164,49421366,49421718,49424968,49429082,49431078,49479178],"created_at":"2026-08-24T12:18:56Z","created_at_i":1787573936,"num_comments":125,"objectID":"49418675","points":122,"story_id":49418675,"story_text":"My partner was let go after 15 years working tirelessly for one of the big five. She's probably the most resilient, rewarded and liked person that has survived and fought through this part of the business in the past decade, and while I am clearly biased, all her colleagues would most certainly agree. She's moved from IC to managing over 30 people and barely manages to use vacation days because she's always working.
She was let go not because of her own incompetence, failure or not fitting in, but purely out of rushed budget cuts (budgets were finalized, finalized again, and then they realized it's still not good enough).
The reason for those cuts, in this particular instance, was not the overwhelming rise of AI and/or getting more firing power for data centers, it was very clearly a long-visible path of failure two levels above, where one particular person continued to wreak havoc up over the past few years, and, ultimately, caused this cul-de-sac disaster.
The person at the center of this all is an older dude who consistenty has been on the wrong side of any decision he's made. I've only heard of a few things as it is (literally) none of my business, but it's been nothing but chaotic and ill-advised choices. He moved into this role from failing elsewhere, leaving the chaos there behind him, and over the past two years, now has caused yet another disaster in this new role.
My question is, and sort of the point of this post (which I understand is entirely philosophical): how do these people consistently survive their self-inflicted disasters? While her entire team (30+) has been let go now as a result of it all, the person who ran this into the ground continues onwards in the business?
They even made her fire and break the news to most of her team in weekly chunks recently, before telling her she's of course, also done.
I used to work for a big five too, but have been self-employed for over a decade now, so I may be too far out to understand, but it just puzzles me how certain people always manage to stay untainted admist the chaos they've caused, at the cost of everyone else. This used to be a thing I've heard over and over again from friends at Google, but it really seems to be very consistent thing elsewhere, too.
They're clearly not smart people, is it ruthlessness? Are they aware of the disaster they're navigating into and plan well ahead? It's a mystery to me, and it just can't be healthy for these companies to allow such folks to continue onwards.","title":"Ask HN: Why do corporate failures always seem to punish the wrong people?","updated_at":"2026-08-28T19:40:44Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"jdw64"},"story_text":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["crul","show"],"value":"Nowadays, almost everyone codes with AI, right? I am a freelancer, and the entire market is now setting deadlines based on AI-assisted coding speeds. If you don't use AI, you simply cannot meet the deadlines. Because of this, I completely transitioned to AI-driven coding at the beginning of this year (starting around GPT-5.3).
It's not that AI coding isn't fun. It is. But in the beginning, there were still many things AI couldn't do well. The community focus was on "Harness Engineering," attaching MCPs, or deep Prompt Engineering. I actually built and used quite a few custom AI tools during that phase.
In the past, you had to provide not just your intent and direction, but also detailed, step-by-step instructions to reach that goal. Lately, though, even if you skip the detailed instructions, both Claude and GPT-5.6 handle it perfectly. You don't even have to agonize over which model to use. "Graph engineering" is currently trending, but it seems the models have already learned those patterns natively\u2014if you establish a high-level plan with them, they automatically embed the execution policies into it. Whenever I try to learn and apply a new AI engineering technique, the next model update already does it out of the box, making the effort feel pointless.
The real problem is that AI is now too good. It applies ADT patterns, immutability, Option/Maybe, Result patterns, functional programming patterns, monadic chaining, and architectural structures like MVC, MVVM, MVI, and Hexagonal architectures far better than I can. It's heavily demotivating.
All those years I spent memorizing GoF patterns, slowly applying them to my work, and tweaking them into my own style\u2014it all feels meaningless now.
Honestly, the vast majority of freelance work revolves around simple CRUD apps. Sure, there is domain knowledge to memorize, but when it comes to structuring CRUD, using functional techniques for error and value handling, or writing boundary tests, AI is just far more meticulous and capable than I am.
It even feels like AI is better at domain modeling. I try to hypnotize myself into believing I'm still necessary, but realistically, I don't work in Big Tech. I'm a programmer who mostly assembles business logic on top of existing frameworks, and the truth is, AI does my job better. And faster. I don't think I can deny it anymore.
Top-tier developers might still outperform AI, but for the vast majority of us, including myself, that is no longer the reality. To write the kind of code AI generates, I used to have to keep documentation open on a second monitor, think deeply, and constantly Google better approaches. AI just generates it instantly, making my past efforts feel obsolete.
When AI writes better code than I do, how am I supposed to maintain my motivation and identity? I would especially love to hear from other freelance developers. With market deadlines getting tighter and tighter because of AI, how are you handling this transition?
Again, it's not that AI "vibe coding" isn't fun. It is. But entirely separate from that, the stark realization that I am losing my competitive edge is making me increasingly depressed."},"title":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["show","hn"],"value":"Ask HN: AI writes better code than me. How to keep my identity?"}},"_tags":["story","author_jdw64","story_49481969","ask_hn"],"author":"jdw64","children":[49482120,49482145,49482521,49482930,49483272,49483303,49483306,49483386],"created_at":"2026-08-28T17:40:39Z","created_at_i":1787938839,"num_comments":9,"objectID":"49481969","points":8,"story_id":49481969,"story_text":"Nowadays, almost everyone codes with AI, right? I am a freelancer, and the entire market is now setting deadlines based on AI-assisted coding speeds. If you don't use AI, you simply cannot meet the deadlines. Because of this, I completely transitioned to AI-driven coding at the beginning of this year (starting around GPT-5.3).
It's not that AI coding isn't fun. It is. But in the beginning, there were still many things AI couldn't do well. The community focus was on "Harness Engineering," attaching MCPs, or deep Prompt Engineering. I actually built and used quite a few custom AI tools during that phase.
In the past, you had to provide not just your intent and direction, but also detailed, step-by-step instructions to reach that goal. Lately, though, even if you skip the detailed instructions, both Claude and GPT-5.6 handle it perfectly. You don't even have to agonize over which model to use. "Graph engineering" is currently trending, but it seems the models have already learned those patterns natively\u2014if you establish a high-level plan with them, they automatically embed the execution policies into it. Whenever I try to learn and apply a new AI engineering technique, the next model update already does it out of the box, making the effort feel pointless.
The real problem is that AI is now too good. It applies ADT patterns, immutability, Option/Maybe, Result patterns, functional programming patterns, monadic chaining, and architectural structures like MVC, MVVM, MVI, and Hexagonal architectures far better than I can. It's heavily demotivating.
All those years I spent memorizing GoF patterns, slowly applying them to my work, and tweaking them into my own style\u2014it all feels meaningless now.
Honestly, the vast majority of freelance work revolves around simple CRUD apps. Sure, there is domain knowledge to memorize, but when it comes to structuring CRUD, using functional techniques for error and value handling, or writing boundary tests, AI is just far more meticulous and capable than I am.
It even feels like AI is better at domain modeling. I try to hypnotize myself into believing I'm still necessary, but realistically, I don't work in Big Tech. I'm a programmer who mostly assembles business logic on top of existing frameworks, and the truth is, AI does my job better. And faster. I don't think I can deny it anymore.
Top-tier developers might still outperform AI, but for the vast majority of us, including myself, that is no longer the reality. To write the kind of code AI generates, I used to have to keep documentation open on a second monitor, think deeply, and constantly Google better approaches. AI just generates it instantly, making my past efforts feel obsolete.
When AI writes better code than I do, how am I supposed to maintain my motivation and identity? I would especially love to hear from other freelance developers. With market deadlines getting tighter and tighter because of AI, how are you handling this transition?
Again, it's not that AI "vibe coding" isn't fun. It is. But entirely separate from that, the stark realization that I am losing my competitive edge is making me increasingly depressed.","title":"Ask HN: AI writes better code than me. How to keep my identity?","updated_at":"2026-08-28T20:01:28Z"}],"hitsPerPage":20,"nbHits":7,"nbPages":1,"page":0,"params":"query=crul+show+hn&advancedSyntax=true&analyticsTags=backend","processingTimeMS":7,"processingTimingsMS":{"_request":{"roundTrip":21},"fetch":{"query":4,"scanning":1,"total":6},"total":7},"query":"crul show hn","serverTimeMS":9}