Probably the most surprising/interesting one I've seen is automating job applications. Essentially spinning up multiple concurrent agents to mass-apply across various job sites, automating things like clicks and form-fills
I thought it was interesting, but would love to hear if you've thought of other quirky or unexpected use cases! :)","title":null,"type":"comment","url":null}],"created_at":"2025-04-08T15:00:26.000Z","created_at_i":1744124426,"id":43622572,"options":[],"parent_id":43621841,"points":null,"story_id":43621841,"text":"what's the most interesting use case you've seen so far?","title":null,"type":"comment","url":null},{"author":"ohnoah","children":[{"author":"theonlyt3","children":[],"created_at":"2025-04-08T16:14:14.000Z","created_at_i":1744128854,"id":43623412,"options":[],"parent_id":43623344,"points":null,"story_id":43621841,"text":"Great question!
We actually included a basic form-filling example (data_entry_example.py) in our GitHub repo\u2014definitely give it a spin and see how it goes.
One tip: filling out forms is currently a bit slow since each step runs sequentially. We're actively looking into concurrency improvements (for example, calculating multiple field interactions at once) to speed things up.
Excited to hear how it works for you\u2014feel free to share any issues or feedback you run into!","title":null,"type":"comment","url":null}],"created_at":"2025-04-08T16:07:20.000Z","created_at_i":1744128440,"id":43623344,"options":[],"parent_id":43621841,"points":null,"story_id":43621841,"text":"Very cool. I was thinking of writing a script for automating some immigration-related forms, might give this a go. Any thing to bear in mind for form-filling?","title":null,"type":"comment","url":null}],"created_at":"2025-04-08T13:56:08.000Z","created_at_i":1744120568,"id":43621841,"options":[],"parent_id":null,"points":9,"story_id":43621841,"text":"Hi HF, I'm Terrell, and we built an open-source app that lets developers create their own Operator with a Next.js/React front-end and a flask back-end. The purpose is to simplify spinning up virtual desktops (Xfce, VNC) and automate desktop-based interactions using computer use models like OpenAI\u2019s
There are already various cool tools out there that allow you to build your own operator-like experience but they usually only automate web browser actions, or aren\u2019t open sourced/cost a lot to get started. Spongecake allows you to automate desktop-based interactions, and is fully open sourced which will help:
- Developers who want to build their own computer use / operator experience\n- Developers who want to automate workflows in desktop applications with poor / no APIs (super common in industries like supply chain and healthcare)\n- Developers who want to automate workflows for enterprises with on-prem environments with constraints like VPNs, firewalls, etc (common in healthcare, finance)
Technical details: This is technically a web browser pointed at a backend server that 1) manages starting and running pre-configured docker containers, and 2) manages all communication with the computer use agent. [1] is handled by spinning up docker containers with appropriate ports to open up a VNC viewer (so you can view the desktop), an API server (to execute agent commands on the container), a marionette port (to help with scraping web pages), and socat (to help with port forwarding). [2] is handled by sending screenshots from the VM to the computer use agent, and then sending the appropriate actions (e.g., scroll, click) from the agent to the VM using the API server.
Some interesting technical challenges we ran into:
- Concurrency - We wanted it to be possible to spin up N agents at once to complete tasks in parallel (especially given how slow computer use agents are today). This introduced a ton of complexity with managing ports since the likelihood went up significantly that a port would be taken.\n- Scrolling issues - The model is really bad at knowing when to scroll, and will scroll a ton on very long pages. To address this, we spun up a Marionette server, and exposed a tool to the agent which will extract a website\u2019s DOM. This way, instead of scrolling all the way to a bottom of a page - the agent can extract the website\u2019s DOM and use that information to find the correct answer
What\u2019s next? We're working on adding support in the UI to run this locally on your own machine, and to spin up other desktop environments like Windows and MacOS. We\u2019ve also started working on integrating Anthropic\u2019s computer use model as well. There\u2019s a ton of other features we can build but wanted to put this out there first and see what others would want
Would really appreciate your thoughts, and feedback. It's been a blast working on this so far and hope others think it\u2019s as neat as I do :)
Here\u2019s the link to clone: https://github.com/aditya-nadkarni/spongecake","title":"Show HN: An open-source Operator that can use computers","type":"story","url":"https://github.com/aditya-nadkarni/spongecake"}