At least from a maintainability perspective.
I would like to see if this implementation is less destructive or at least more suitable for a red-green-refactor workflow.","title":null,"type":"comment","url":null},{"author":"NegativeLatency","children":[],"created_at":"2024-04-02T22:00:41.000Z","created_at_i":1712095241,"id":39911313,"options":[],"parent_id":39909420,"points":null,"story_id":39907468,"text":"Unless you weren't actually that successful but need to publish a "successful" result","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T18:45:30.000Z","created_at_i":1712083530,"id":39909420,"options":[],"parent_id":39907468,"points":null,"story_id":39907468,"text":"Friendly suggestion to the authors: success rates aren't meaningful to all but a handful of researchers. They should add a few examples of tests SWE-agent passed and did not pass to the README.","title":null,"type":"comment","url":null},{"author":"mdaniel","children":[{"author":"SrslyJosh","children":[],"created_at":"2024-04-03T09:31:44.000Z","created_at_i":1712136704,"id":39915176,"options":[],"parent_id":39909935,"points":null,"story_id":39907468,"text":"Probably created with an LLM.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T19:32:56.000Z","created_at_i":1712086376,"id":39909935,"options":[],"parent_id":39907468,"points":null,"story_id":39907468,"text":"I think that "Demo" link is just an extremely annoying version of an HTML presentation, so they could save me a shitload of clicking if they just dumped their presentation out to a PDF or whatever so I could read faster than watching it type out text as if it was live. It also whines a lot on the console about its inability to connect to a websocket server on 3000 but I don't know what it would do with a websocket connection if had it","title":null,"type":"comment","url":null},{"author":"matthewaveryusa","children":[{"author":"toddmorey","children":[{"author":"clement_b","children":[],"created_at":"2024-04-02T21:14:52.000Z","created_at_i":1712092492,"id":39910919,"options":[],"parent_id":39910462,"points":null,"story_id":39907468,"text":"I have a git feeling this comment was written on mobile.","title":null,"type":"comment","url":null},{"author":"hazn","children":[{"author":"toddmorey","children":[],"created_at":"2024-04-02T23:23:03.000Z","created_at_i":1712100183,"id":39911937,"options":[],"parent_id":39911841,"points":null,"story_id":39907468,"text":"Excellent! Thanks for sharing this!","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T23:08:52.000Z","created_at_i":1712099332,"id":39911841,"options":[],"parent_id":39910462,"points":null,"story_id":39907468,"text":"DSPy is the best tool for optimizing prompts [0]: https://github.com/stanfordnlp/dspy
Think of it as a meta-prompt optimizer, it uses a LLM to optimize your prompts, to optimize your LLM.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T20:22:45.000Z","created_at_i":1712089365,"id":39910462,"options":[],"parent_id":39910090,"points":null,"story_id":39907468,"text":"I\u2019m always fascinated to read the system prompts & I always wonder what sort of gains can be made optimizing them further.
Once I\u2019m back on desktop I want to look at the gut history of this file.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T19:46:34.000Z","created_at_i":1712087194,"id":39910090,"options":[],"parent_id":39907468,"points":null,"story_id":39907468,"text":"Very neat. Uses the langchain method, here are some of the prompts:
https://github.com/princeton-nlp/SWE-agent/blob/main/config/...","title":null,"type":"comment","url":null},{"author":"iLoveOncall","children":[{"author":"1letterunixname","children":[],"created_at":"2024-04-02T19:59:13.000Z","created_at_i":1712087953,"id":39910223,"options":[],"parent_id":39910195,"points":null,"story_id":39907468,"text":"Do spaces and spelling fixes count?
Copilot, so far, is only good for predicting the next bit of similar patterns of code","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T19:56:07.000Z","created_at_i":1712087767,"id":39910195,"options":[],"parent_id":39907468,"points":null,"story_id":39907468,"text":"And creates how many new ones?
This and Devin generate garbage code that will make any codebase worse.
It's a joke that 12.5% is even associated with the word "success".","title":null,"type":"comment","url":null},{"author":"noncoml","children":[{"author":"iwontberude","children":[{"author":"jeremyjh","children":[{"author":"iwontberude","children":[],"created_at":"2024-04-02T23:48:50.000Z","created_at_i":1712101730,"id":39912090,"options":[],"parent_id":39910654,"points":null,"story_id":39907468,"text":"Oh well you should hear the hype from CNBC and other places, they are strongly intimating that gen AI will replace SWEs on product development teams. I totally agree it\u2019s not likely, but it\u2019s starting to get baked into asset prices and I want to profit from that misunderstanding.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T20:42:23.000Z","created_at_i":1712090543,"id":39910654,"options":[],"parent_id":39910314,"points":null,"story_id":39907468,"text":"The most relevant companies focused on this aren't publicly traded. The ones that are publicly traded like MSFT have way too many other factors affecting their value - not to mention the fact that they'll make money on generative AI that has nothing to do with coding regardless of if an SWE-agent ever works.","title":null,"type":"comment","url":null},{"author":"noncoml","children":[{"author":"vertis","children":[],"created_at":"2024-04-03T09:55:55.000Z","created_at_i":1712138155,"id":39915371,"options":[],"parent_id":39910942,"points":null,"story_id":39907468,"text":"Conservatively, I think LLMs will replace SWE roles within the next 5-10 years. In that period SWE roles will change drastically, to be more hearding of AI agents.
We can't hope to compete with something that can edit all the files in less than 5 minutes.
If you're not at least trying to plan what your life will look like under these circumstances then you're doing yourself a disservice.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T21:17:21.000Z","created_at_i":1712092641,"id":39910942,"options":[],"parent_id":39910314,"points":null,"story_id":39907468,"text":"Ugh, I am not claiming that LLMs are not great innovation. Just that they are not going to replace SWE jobs in our(maybe my) lifetime.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T20:07:44.000Z","created_at_i":1712088464,"id":39910314,"options":[],"parent_id":39910293,"points":null,"story_id":39907468,"text":"Hypothetically, which ticker symbols would you buy put contracts on, at what strike prices, and at what expiration dates? As far as I can tell, a lot of people are betting a lot of money that you are wrong, but actually I think you are right.","title":null,"type":"comment","url":null},{"author":"vineyardmike","children":[],"created_at":"2024-04-02T20:13:26.000Z","created_at_i":1712088806,"id":39910371,"options":[],"parent_id":39910293,"points":null,"story_id":39907468,"text":"Yea the problem with that is the control group - grab any SWE and ask them the same thing. I don\u2019t think most would pass. Unless you want to give an SWE time to learn\u2026 then it\u2019s hardly fair. And I vaguely trust the LLM to be able to learn it too.
Also I just asked Claude and Gemini and they both provided an implementation that matches the \u201cbytes to UTF-8\u201d rust docs. Assuming those are right,LLMs can do this (but I haven\u2019t tested the code).
https://doc.rust-lang.org/std/string/struct.String.html","title":null,"type":"comment","url":null},{"author":"raggi","children":[{"author":"woodruffw","children":[{"author":"raggi","children":[{"author":"comex","children":[{"author":"raggi","children":[],"created_at":"2024-04-02T20:31:58.000Z","created_at_i":1712089918,"id":39910557,"options":[],"parent_id":39910538,"points":null,"story_id":39907468,"text":"all true, as I said in another fork of the thread, this comes down to part of what humans will still be valuable for in this loop: distilling poor requirements into better requirements.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T20:30:03.000Z","created_at_i":1712089803,"id":39910538,"options":[],"parent_id":39910478,"points":null,"story_id":39907468,"text":"Unlike the original version, this version compiles and seems to basically work correctly. However, the design is misleading: `buffer` is declared as an array of 4 bytes but only the first byte is ever used. The code also has suboptimal performance and error handling, though that's not the end of the world.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T20:23:39.000Z","created_at_i":1712089419,"id":39910478,"options":[],"parent_id":39910431,"points":null,"story_id":39907468,"text":"here's what it does if i feed your nit pick into it verbatim:
use std::fs::File;\n use std::io::{self, Read};\n use std::str::from_utf8;\n\n fn read_file_character_by_character(path: &str) -> io::Result<()> {\n // Open the file in read-only mode.\n let mut file = File::open(path)?;\n\n let mut buffer = [0; 4]; // UTF-8 character max byte length\n let mut temp_buffer = vec![];\n\n while let Ok(bytes_read) = file.read(&mut buffer[..1]) { // Read byte by byte\n if bytes_read == 0 { // End of file\n break;\n }\n\n temp_buffer.push(buffer[0]);\n\n if let Ok(s) = from_utf8(&temp_buffer) {\n if s.chars().count() == 1 { // Complete character read\n print!("{}", s);\n temp_buffer.clear(); // Clear temp buffer for next character\n }\n }\n }\n\n Ok(())\n }\n\n fn main() {\n let path = "path/to/your/file.txt";\n\n if let Err(e) = read_file_character_by_character(path) {\n eprintln!("Error reading file: {}", e);\n }\n } let mut file = File::open(path)?;\n let mut contents = String::new();\n file.read_to_string(&mut contents)?;You're going to have to buffer at least a little, to figure out where the USV / grapheme boundary is, depending on our definition of "character". To me, a BufReader is appropriate here; it avoids lots of tiny reads to the kernel, which is probably the right behavior in a real case.
To me, "read character by character" vaguely implies something that's going to yield a stream of characters. (Again, for some definition there.)","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T20:19:04.000Z","created_at_i":1712089144,"id":39910431,"options":[],"parent_id":39910386,"points":null,"story_id":39907468,"text":"If we're being sticklers, this isn't reading character-by-character: it's performing a buffered read, which then gets iterated over.","title":null,"type":"comment","url":null},{"author":"raggi","children":[],"created_at":"2024-04-02T20:21:59.000Z","created_at_i":1712089319,"id":39910454,"options":[],"parent_id":39910386,"points":null,"story_id":39907468,"text":"fwiw, the benchmark that matters really has nothing to do with authoring code.
the typing of code is the easy part even though it's a part a lot of folks are somewhat addicted to.
the things which have far more value are applying value judgements to requirements, correlating and incorporating sparse and inaccurate diagnostic information into a coherent debugging strategy, and so on. there will come a time when it can assist with these too, probably first on requirements distillation, but for more complex debugging tasks that's a novel problem solving area that we've yet to see substantial movement on.
so if you want to stave off the robots coming for you, get good at debugging hard problems, and learn to make really great use of tools that accelerate the typing out of solutions to baseline product requirements.","title":null,"type":"comment","url":null},{"author":"comex","children":[{"author":"raggi","children":[{"author":"phaer","children":[{"author":"scottlamb","children":[{"author":"1letterunixname","children":[{"author":"scottlamb","children":[{"author":"1letterunixname","children":[{"author":"scottlamb","children":[],"created_at":"2024-04-03T15:22:32.000Z","created_at_i":1712157752,"id":39918656,"options":[],"parent_id":39914096,"points":null,"story_id":39907468,"text":"> Duh. It doesn't really matter what Rust has have went it comes to enabling the use of specific edge-case performance improvements for specific purposes. Inefficient AI-generated code without a clue of other approaches doesn't move the needle. Religious purity doesn't matter, only results matter.
No idea what this incoherent, ungrammatical paragraph is supposed to be saying. But if you're under the impression Rust doesn't have its own buffered IO facilities or that using Rust-native libraries offers only "religious purity" benefits over extern "C" stuff, you're mistaken.
This has diverged from what I'm interested in discussing anyway; see my question upthread about if there are any LLM tools that gather requirements from incomplete specs in the way I expect human engineers to. In this case, I'd expect it to ask questions such as "how large are input files expected to be?" Better, ask what the greater purpose is, as "character by character" is rarely useful.","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T06:00:29.000Z","created_at_i":1712124029,"id":39914096,"options":[],"parent_id":39912757,"points":null,"story_id":39907468,"text":"Yes, the userland side presented such as with POSIX like ssize_t read(int fd, void* buf, size_t count). Calling that with count = 1 each time would be wasteful, but certainly libc's have been buffering this since at least the 1980's. I remember this was the case with Borland C/C++.
> Our comments are part of a thread discussing this prompt [2] that specifically requests Rust and this snippet in response [3]. Not portable C code. You can use those C stdio functions from Rust, but you really shouldn't without a very good reason. Rust has its own IO library that is safe and well integrated with other Rust things like `#![derive(Debug)]`.
Duh. It doesn't really matter what Rust has have went it comes to enabling the use of specific edge-case performance improvements for specific purposes. Inefficient AI-generated code without a clue of other approaches doesn't move the needle. Religious purity doesn't matter, only results matter.","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T01:39:51.000Z","created_at_i":1712108391,"id":39912757,"options":[],"parent_id":39912265,"points":null,"story_id":39907468,"text":"> There most certainly is getwchar() and fgetwc()/getwc() on anything that's POSIX C95, so that's more or less everything that's not a vintage antique.
Apologies for the imprecision: by OS API, I meant syscall, at least on POSIX systems. The functions you refer to are C stdio things. Note also they implement on top of read(2) one of the two options I mentioned: "loop over getting the next N bytes and getting all complete characters so far (with some extra complexity around characters that cross chunk boundaries)".
btw, if we're being precise, getwchar gets a code point, and character might mean grapheme instead. Same is true for the `str::chars` call in the LLM's Rust snippet. The docstring for that method mentions this [1] because it was written in this century after people thought about this stuff a bit.
> portable C code to do so would be able to run on anything made after 1982.
Our comments are part of a thread discussing this prompt [2] that specifically requests Rust and this snippet in response [3]. Not portable C code. You can use those C stdio functions from Rust, but you really shouldn't without a very good reason. Rust has its own IO library that is safe and well integrated with other Rust things like `#![derive(Debug)]`.
[1] https://doc.rust-lang.org/std/primitive.str.html#method.char...
[2] https://news.ycombinator.com/item?id=39910542
[3] https://news.ycombinator.com/item?id=39910542","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T00:15:18.000Z","created_at_i":1712103318,"id":39912265,"options":[],"parent_id":39911487,"points":null,"story_id":39907468,"text":"There most certainly is getwchar() and fgetwc()/getwc() on anything that's POSIX C95, so that's more or less everything that's not a vintage antique.
Reading individual UTF-8 codepoints is a trivial exercise if byte width getchar() were available, and portable C code to do so would be able to run on anything made after 1982. IIRC, they don't teach how to write portable C code in Comp Sci programs anymore and it's a shame.
Never read a file completely into memory at once unless there is zero chance of it being a huge file because this is an obvious DoS vector and waste of resources.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T22:22:18.000Z","created_at_i":1712096538,"id":39911487,"options":[],"parent_id":39910866,"points":null,"story_id":39907468,"text":"What would you expect? There's no OS API for "read one character", except in say ASCII where 1 byte = 1 code point = 1 character. And it'd be hideously inefficient anyway. So you either loop over getting the next N bytes and getting all complete characters so far (with some extra complexity around characters that cross chunk boundaries) or you read the whole thing into a single buffer and iterate the characters. This code does the latter. If this tool doesn't have the ability to respond by asking requirements questions, I'd consider either choice valid.
Of course, in real life, I do expect to get requirements questions back from an engineer when I assign a task. Seems more practical than anticipating everything up-front into the perfect specification/prompt. Why shouldn't I expect the same from an LLM-based tool? Are any of them set up to do that?","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T21:09:19.000Z","created_at_i":1712092159,"id":39910866,"options":[],"parent_id":39910542,"points":null,"story_id":39907468,"text":"But this doesn't read the file char-by-char, but uses buffering to read it into a string","title":null,"type":"comment","url":null},{"author":"deathanatos","children":[],"created_at":"2024-04-02T21:39:40.000Z","created_at_i":1712093980,"id":39911153,"options":[],"parent_id":39910542,"points":null,"story_id":39907468,"text":"On errors, it exits with success.","title":null,"type":"comment","url":null},{"author":"1letterunixname","children":[],"created_at":"2024-04-02T23:50:08.000Z","created_at_i":1712101808,"id":39912100,"options":[],"parent_id":39910542,"points":null,"story_id":39907468,"text":"A few notes:
- It should be generating `path: impl AsRef<Path>` to be properly generic.
- It's not setting a nonzero exit code on error.
- Edge case handling is a vital property for production-usable tools at scale. I'm wondering if it can yet special case situations such as creating a conditional compilation version for Linux that uses the splice syscall when the arguments are 2 file handles.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T20:30:21.000Z","created_at_i":1712089821,"id":39910542,"options":[],"parent_id":39910491,"points":null,"story_id":39907468,"text":"good catch. feeding it the error output of rustc it then produces:
use std::fs::File;\n use std::io::{self, Read};\n\n fn read_file_character_by_character(path: &str) -> io::Result<()> {\n let mut file = File::open(path)?;\n let mut contents = String::new();\n\n file.read_to_string(&mut contents)?;\n\n for c in contents.chars() {\n println!("{}", c);\n }\n\n Ok(())\n }\n\n fn main() {\n let path = "path/to/your/file.txt";\n\n if let Err(e) = read_file_character_by_character(path) {\n eprintln!("Error reading file: {}", e);\n }\n }(Also, the comment about the iterator element type is inconsistent with the code following it. Based on the comment, `c` would be of type `(char, usize)`, but then trying to print it with {} would fail because tuples don't implement Display.)","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T20:14:25.000Z","created_at_i":1712088865,"id":39910386,"options":[],"parent_id":39910293,"points":null,"story_id":39907468,"text":"it does an ok job with this task:
use std::fs::File;\n use std::io::{self, BufReader, Read};\n\n fn read_file_character_by_character(path: &str) -> io::Result<()> {\n // Open the file in read-only mode.\n let file = File::open(path)?;\n\n // Create a buffered reader to read the file more efficiently.\n let reader = BufReader::new(file);\n\n // `chars` method returns an iterator over the characters of the input.\n // Note that it returns a Result<(char, usize), io::Error>, where usize is the byte length of the char.\n for char_result in reader.chars() {\n match char_result {\n Ok(c) => print!("{}", c),\n Err(e) => return Err(e),\n }\n }\n\n Ok(())\n }\n\n fn main() {\n let path = "path/to/your/file.txt";\n\n if let Err(e) = read_file_character_by_character(path) {\n eprintln!("Error reading file: {}", e);\n }\n }This is such an extremely bullish case, I'm not sure why you'd think this is even remotely possible. A Google search is usually more valuable than ChatGPT. For example, the rust utf-8 example is already verbatim solved on reddit: https://www.reddit.com/r/rust/comments/l5m1rw/how_can_i_effi...","title":null,"type":"comment","url":null},{"author":"int_19h","children":[],"created_at":"2024-04-03T01:40:11.000Z","created_at_i":1712108411,"id":39912761,"options":[],"parent_id":39910642,"points":null,"story_id":39907468,"text":"The implicit assumption here is that a human "senior SWE" can engineer a model of the same quality that is capable of simulating him. Which is definitely not true with the best models that we have today - and they certainly can't simulate a senior SWE, so the actual bar is higher.
I'm not saying that the whole "robots building better robots" thing is a pipedream, but given where things are today, this is not something that's going to happen soon.","title":null,"type":"comment","url":null},{"author":"vertis","children":[],"created_at":"2024-04-03T09:57:15.000Z","created_at_i":1712138235,"id":39915382,"options":[],"parent_id":39910642,"points":null,"story_id":39907468,"text":"People (SWEs) don't want to hear this. I think it's an inevitability that something of this nature will happen.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T20:41:24.000Z","created_at_i":1712090484,"id":39910642,"options":[],"parent_id":39910293,"points":null,"story_id":39907468,"text":"The way I see it, its undetermined if Generative AI will be able to fully do a SWE job.
But, for most of the debates I've seen, I don't think it the answer matters all too much.
Once we have models that can act as full senior SWEs.. the models can engineer the models. And then we've hit the recursive case.
Once models can engineer models better and faster than humans, all bets are off. Its the foggy future. Its the singularity.","title":null,"type":"comment","url":null},{"author":"userbinator","children":[{"author":"int_19h","children":[{"author":"userbinator","children":[{"author":"int_19h","children":[],"created_at":"2024-04-05T00:22:33.000Z","created_at_i":1712276553,"id":39937438,"options":[],"parent_id":39925520,"points":null,"story_id":39907468,"text":"There is pushback, it's just not broad enough yet. Because, as broken as things are (which is especially visible to those of us making the sausage or watching it made), they still kinda sorta work most of the time, to the point where users grumble but learn to live with it.
But I think that AI coding will upset this equilibrium by reducing the quality even more, and significantly enough that users will very much notice - and for many of them it will push things into "what I need doesn't work most of the time" category. And then there will be payback.
Then again, I am an optimist.","title":null,"type":"comment","url":null}],"created_at":"2024-04-04T01:24:12.000Z","created_at_i":1712193852,"id":39925520,"options":[],"parent_id":39912735,"points":null,"story_id":39907468,"text":"Unfortunately, "lots of crap that's so badly broken most of the time" already describes a lot of software these days, and yet there hasn't been much pushback. Everyone seems to be mostly in a state of learned helplessness.","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T01:37:13.000Z","created_at_i":1712108233,"id":39912735,"options":[],"parent_id":39912138,"points":null,"story_id":39907468,"text":"I'm kinda hoping that LLMs will get pushed into production use writing code before they have acceptable quality (because greed), and the result will be lots of crap that's so badly broken most of the time that there will be a massive pushback against said culture from the users. Maybe from the governments as well, after a few well-publicized infrastructure failures.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T23:56:34.000Z","created_at_i":1712102194,"id":39912138,"options":[],"parent_id":39910293,"points":null,"story_id":39907468,"text":"I'm not afraid of LLMs replacing me because of their output quality. The problem is the proliferation of quantity-over-quality "churn out barely-working crap as fast as possible" culture that gives LLMs the advantage over real humans.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T20:05:40.000Z","created_at_i":1712088340,"id":39910293,"options":[],"parent_id":39907468,"points":null,"story_id":39907468,"text":"If you are afraid that LLMs will replace you at your job, ask an LLM to write Rust code for reading a utf8 file character by character
Edit: Yes, it does write some code that is "close" enough, but in some cases it is wrong, in others it doesn't not do exactly what asked. I.e. needs supervision from someone who understands both the requirements, the code and the problems that may arise from the naive line that the LLM is taking. Mind you, the most popular the issue, the better the line LLM is taking. So in other words, IMHO is a glorified Stack Overflow. Just as there are engineers that copy-paste from SO without having any idea what the code does, there will be engineers that will just copy paste from LLM. Their work will be much better than if they used SO, but I think it's still nowhere to the mark of a Senior SWE and above.","title":null,"type":"comment","url":null},{"author":"lispisok","children":[{"author":"lewhoo","children":[],"created_at":"2024-04-02T21:52:48.000Z","created_at_i":1712094768,"id":39911258,"options":[],"parent_id":39910369,"points":null,"story_id":39907468,"text":"Maybe that's the point of this research. Hey look, we reproduced the way to game the stats a bit. I really can't tell anymore.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T20:13:15.000Z","created_at_i":1712088795,"id":39910369,"options":[],"parent_id":39907468,"points":null,"story_id":39907468,"text":"Their demo is so similar to the Devin one I had to go look up the Devin one to check I wasnt watching the same demo. I feel like there might be a reason they both picked Sympy. Also I rarely put weight into demos. They are usually cherry-picked at best and outright fabricated at worst. I want to hear what 3rd parties have to say after trying these things.","title":null,"type":"comment","url":null},{"author":"Frummy","children":[{"author":"aussieguy1234","children":[],"created_at":"2024-04-02T21:44:32.000Z","created_at_i":1712094272,"id":39911193,"options":[],"parent_id":39910382,"points":null,"story_id":39907468,"text":"Similar to how early computers didn't have enough ram to display the whole text file, so old programmers had to work with parts of the file at a time. It's not a bad way to get around the context window problem, which is kind of similar.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T20:13:57.000Z","created_at_i":1712088837,"id":39910382,"options":[],"parent_id":39907468,"points":null,"story_id":39907468,"text":"Interesting idea to provide the Agent-Computer Interface for it to scroll and such, interact easier from its perspective","title":null,"type":"comment","url":null},{"author":"unit_circle","children":[{"author":"golergka","children":[],"created_at":"2024-04-02T20:45:31.000Z","created_at_i":1712090731,"id":39910686,"options":[],"parent_id":39910424,"points":null,"story_id":39907468,"text":"It's still abysmal from POV of actually using it in production, but it's a very impressive rate of improvement. Given what happened with LLMs and image generation in the last few years, we can probably assume that these systems will be able to fix most trivial bugs pretty soon.","title":null,"type":"comment","url":null},{"author":"stefan_","children":[],"created_at":"2024-04-02T21:07:55.000Z","created_at_i":1712092075,"id":39910857,"options":[],"parent_id":39910424,"points":null,"story_id":39907468,"text":"These \u201ebenchmark\u201c are tuned around reporting some exciting result, once you look inside, all the \u201efixes\u201c are trash.","title":null,"type":"comment","url":null},{"author":"blharr","children":[{"author":"WanderPanda","children":[],"created_at":"2024-04-02T21:39:21.000Z","created_at_i":1712093961,"id":39911151,"options":[],"parent_id":39911021,"points":null,"story_id":39907468,"text":"It is a bit worrisome but we manage to deal with subpar human code as well. Often the boilerplate generated by ChatGPT is already better than what an unexperienced coder would string together. I\u2018m sure it will not be a free lunch but the the benefits will probably outweigh the downsides.
Interesting scalability questions will arise wrt to security when scaling the already unmanagably large code bases by another magnitude (or two), though.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T21:25:41.000Z","created_at_i":1712093141,"id":39911021,"options":[],"parent_id":39910424,"points":null,"story_id":39907468,"text":"I still don't know. I feel like there are many ways where GPT will write some code or fix a bug in a way that makes it significantly harder to debug. Even for relatively simple tasks, it's kind of like machine-generated code that I would not want to touch.","title":null,"type":"comment","url":null},{"author":"SrslyJosh","children":[],"created_at":"2024-04-03T09:24:54.000Z","created_at_i":1712136294,"id":39915127,"options":[],"parent_id":39910424,"points":null,"story_id":39907468,"text":"If someone submitted 8 PRs and 7 of them were bullshit, I would close anything else they submitted in the future without even bothering to review.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T20:17:46.000Z","created_at_i":1712089066,"id":39910424,"options":[],"parent_id":39907468,"points":null,"story_id":39907468,"text":"A 1/8 chance of fixing a bug at the cost of a careful review and some corrections is not bad.
0% -> 12% improvement is not bad for two years either (I'm somewhat arbitrary picking the release date of ChatGPT). If this can be kept up for a few years we will have some extremely useful tooling. The cost can be relatively high as well, since engineering time is currently orders of magnitude more expensive than these tools.","title":null,"type":"comment","url":null},{"author":"anotherpaulg","children":[{"author":"a_wild_dandan","children":[{"author":"ein0p","children":[],"created_at":"2024-04-02T21:46:35.000Z","created_at_i":1712094395,"id":39911209,"options":[],"parent_id":39910628,"points":null,"story_id":39907468,"text":"I\u2019ve tried this with another similar system. FOSS LLMs including Mixtral are currently too weak to handle something like this. For me they run out of steam after only a few turns and start going in circles unproductively","title":null,"type":"comment","url":null},{"author":"swatcoder","children":[{"author":"nickpsecurity","children":[],"created_at":"2024-04-02T23:20:59.000Z","created_at_i":1712100059,"id":39911923,"options":[],"parent_id":39911624,"points":null,"story_id":39907468,"text":"I totally agree. My solution to this was limiting my AI use to (a) whatever didn't impair creativity and (b) just in general to keep the brain sharp. If using AI regularly, one could just manually solve a percentage of the problems.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T22:40:54.000Z","created_at_i":1712097654,"id":39911624,"options":[],"parent_id":39910628,"points":null,"story_id":39907468,"text":"> My debugging time costs many orders of magnitude more than SWE-agent
Unless your job is primarily to clean up somebody else's mess, your debugging time is a key part of a career-long feedback loop that improves your craft. Be careful not to shrug it off as something less. Many many people are spending a lot of money to let you forget it, and once you do, you'll be right there in the ranks of the cheaply replaceble.
(And on the odd chance that cleaning up other people's mess is your job, you should probably be the one doing it; and for largely the same reasons)","title":null,"type":"comment","url":null},{"author":"Aperocky","children":[],"created_at":"2024-04-02T22:53:49.000Z","created_at_i":1712098429,"id":39911739,"options":[],"parent_id":39910628,"points":null,"story_id":39907468,"text":"That's assuming that the other 95% stays the same with this new agent (vs creating more work for you to now also have to parse what the model is saying).","title":null,"type":"comment","url":null},{"author":"int_19h","children":[],"created_at":"2024-04-03T01:29:55.000Z","created_at_i":1712107795,"id":39912694,"options":[],"parent_id":39910628,"points":null,"story_id":39907468,"text":"Given that they got 12% with GPT-4, which is vastly better than any open model, I doubt this would be particularly productive. And powering compute at full load is going to add up.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T20:39:51.000Z","created_at_i":1712090391,"id":39910628,"options":[],"parent_id":39910452,"points":null,"story_id":39907468,"text":"Personally, I'd just use one of my local MacBook models (e.g. Mixtral 8x7b) and forget about any wasted branches & cents. My debugging time costs many orders of magnitude more than SWE-agent, so even a 5% backlog savings would be spectacular!","title":null,"type":"comment","url":null},{"author":"senko","children":[],"created_at":"2024-04-03T06:29:20.000Z","created_at_i":1712125760,"id":39914253,"options":[],"parent_id":39910452,"points":null,"story_id":39907468,"text":"If you don't mind me asking, which agentic tools/frameworks have you tried for code fixing/generation, with which LLMs?","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T20:21:50.000Z","created_at_i":1712089310,"id":39910452,"options":[],"parent_id":39907468,"points":null,"story_id":39907468,"text":"Very cool project!
I've experimented in this direction previously, but found agentic behavior is often chaotic and leads to long expensive sessions that go down a wrong rabbit hole and ultimately fail.
It's great that you succeed on 12% of swe-bench, but what happens the other 88% of the time? Is it useless wasted work and token costs? Or does it make useful progress that can be salvaged?
Also, I think swe-bench is from your group, right? Have you made any attempt to characterize a "skilled human upper bound" score?
I randomly sampled a dozen swe-bench tasks myself, and found that many were basically impossible for a skilled human to \u201csolve\u201d. Mainly because the tasks were under specified wrt to the hidden test cases that determine passing. The tests were checking implementation specific details from the repo\u2019s PR that weren't actually stated requirements of the task.","title":null,"type":"comment","url":null},{"author":"trebligdivad","children":[],"created_at":"2024-04-02T21:09:15.000Z","created_at_i":1712092155,"id":39910864,"options":[],"parent_id":39907468,"points":null,"story_id":39907468,"text":"So this issues arbitrary shell commands based on trying to understand the untrusted bug text ?\nShould be fun waiting until someone finds an escape.","title":null,"type":"comment","url":null},{"author":"sumeruchat","children":[{"author":"recursive","children":[{"author":"sumeruchat","children":[{"author":"int_19h","children":[{"author":"sumeruchat","children":[{"author":"int_19h","children":[],"created_at":"2024-04-04T00:09:44.000Z","created_at_i":1712189384,"id":39924982,"options":[],"parent_id":39915754,"points":null,"story_id":39907468,"text":"AI will quite readily write bad quality code with security vulnerabilities even for bog standard stuff (like say SQL injections).
And sure, a good developer can fix it if they will see it. But they won't when running on that kind of schedule.","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T10:51:00.000Z","created_at_i":1712141460,"id":39915754,"options":[],"parent_id":39912718,"points":null,"story_id":39907468,"text":"None because 1) this is pretty standard stuff with stripe 2) the good developer can go through the code and fix them in a few hours if there were any","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T01:33:39.000Z","created_at_i":1712108019,"id":39912718,"options":[],"parent_id":39912682,"points":null,"story_id":39907468,"text":"Depends on how many security vulnerabilities are in that payments system.
Or, I suppose, depending on whose value. The consultants that'll have to be hired by the poor shmuck who paid for that will make a fortune auditing and cleaning up the code.","title":null,"type":"comment","url":null},{"author":"recursive","children":[],"created_at":"2024-04-03T03:00:35.000Z","created_at_i":1712113235,"id":39913191,"options":[],"parent_id":39912682,"points":null,"story_id":39907468,"text":"Not without more information.","title":null,"type":"comment","url":null},{"author":"gloosx","children":[{"author":"sumeruchat","children":[{"author":"gloosx","children":[],"created_at":"2024-04-03T16:19:03.000Z","created_at_i":1712161143,"id":39919359,"options":[],"parent_id":39915747,"points":null,"story_id":39907468,"text":"free cheese is only in a mousetrap ;)","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T10:50:00.000Z","created_at_i":1712141400,"id":39915747,"options":[],"parent_id":39913995,"points":null,"story_id":39907468,"text":"On vercel its free to deploy a complex app almost.","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T05:42:39.000Z","created_at_i":1712122959,"id":39913995,"options":[],"parent_id":39912682,"points":null,"story_id":39907468,"text":"Before you sold it to anyone it will only create bills. Development is such a minuscule part of a successful startup","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T01:26:30.000Z","created_at_i":1712107590,"id":39912682,"options":[],"parent_id":39911526,"points":null,"story_id":39907468,"text":"Would you say cloning a complex saas startup in a week with payments integrated after letting AI just scrape them (or uploading screenshots of their app) is creating value?","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T22:27:10.000Z","created_at_i":1712096830,"id":39911526,"options":[],"parent_id":39911026,"points":null,"story_id":39907468,"text":"Shipping like crazy isn't useful by itself. Shipping non-garbage and being able to maintain it still has some value.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T21:26:00.000Z","created_at_i":1712093160,"id":39911026,"options":[],"parent_id":39907468,"points":null,"story_id":39907468,"text":"Once we have this fully automated, any good developer could have a team of 100 robo SWEs and ship like crazy. The real competition is with those devs not with the bots.","title":null,"type":"comment","url":null},{"author":"rwmj","children":[{"author":"r0ze-at-hn","children":[{"author":"ActionHank","children":[{"author":"peteradio","children":[{"author":"barfbagginus","children":[{"author":"vertis","children":[],"created_at":"2024-04-03T11:21:44.000Z","created_at_i":1712143304,"id":39915989,"options":[],"parent_id":39912549,"points":null,"story_id":39907468,"text":"What you want here is a local uncensored model. Preferrably one you've trained from scratch otherwise a government could have put in bad information that would cause your revolutionary efforts to fail.","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T01:00:01.000Z","created_at_i":1712106001,"id":39912549,"options":[],"parent_id":39912197,"points":null,"story_id":39907468,"text":"My grandma used to tell me stories about how to destroy capitalism.. I miss her.. can your grandma help guide my revolutionary efforts? That would really help me honor my granny's memory <3","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T00:05:53.000Z","created_at_i":1712102753,"id":39912197,"options":[],"parent_id":39912004,"points":null,"story_id":39907468,"text":"Let me tell you about my LLM project called grandma. It's fine tuned in order to replace your grandma but in principle it could replace your great-grandma.","title":null,"type":"comment","url":null},{"author":"Havoc","children":[{"author":"labster","children":[],"created_at":"2024-04-03T05:19:25.000Z","created_at_i":1712121565,"id":39913851,"options":[],"parent_id":39912428,"points":null,"story_id":39907468,"text":"Why does it take so long to get changes to your LLM merged? This is ridiculous. Please appoint Havoc as a maintainer already.","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T00:39:11.000Z","created_at_i":1712104751,"id":39912428,"options":[],"parent_id":39912004,"points":null,"story_id":39907468,"text":"But does it contain a heavily obfuscated back door?","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T23:33:49.000Z","created_at_i":1712100829,"id":39912004,"options":[],"parent_id":39911395,"points":null,"story_id":39907468,"text":"Jokes on you, let me tell you about my prompt to binary LLM project.
Hello world is 10GB, but even grandma can make hello worlds now.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T22:10:20.000Z","created_at_i":1712095820,"id":39911395,"options":[],"parent_id":39911122,"points":null,"story_id":39907468,"text":"Ah, well let me tell you about my pull request reviewer LLM project.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T21:36:21.000Z","created_at_i":1712093781,"id":39911122,"options":[],"parent_id":39907468,"points":null,"story_id":39907468,"text":"Do we know how much extra work it created for the real people who had to review the proposed fixes?","title":null,"type":"comment","url":null},{"author":"aussieguy1234","children":[{"author":"mlcrypto","children":[{"author":"Dylan16807","children":[],"created_at":"2024-04-03T02:05:13.000Z","created_at_i":1712109913,"id":39912889,"options":[],"parent_id":39911317,"points":null,"story_id":39907468,"text":"I don't think xz makes a difference here. The perceived likelihood of problems, malicious or not, is pretty much the same. As far as this discussion goes, it's just another example in the pile of examples, not an event with meaningful before and after epochs.","title":null,"type":"comment","url":null},{"author":"aussieguy1234","children":[],"created_at":"2024-04-03T05:05:41.000Z","created_at_i":1712120741,"id":39913770,"options":[],"parent_id":39911317,"points":null,"story_id":39907468,"text":"Fix one bug, introduce 5 more","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T22:01:07.000Z","created_at_i":1712095267,"id":39911317,"options":[],"parent_id":39911137,"points":null,"story_id":39907468,"text":"Yep. After xz we don't need a bot mindlessly fixing all suggestions from malicious actors","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T21:38:03.000Z","created_at_i":1712093883,"id":39911137,"options":[],"parent_id":39907468,"points":null,"story_id":39907468,"text":"12% fix rate = 88% bug rate","title":null,"type":"comment","url":null},{"author":"paradite","children":[{"author":"barfbagginus","children":[],"created_at":"2024-04-03T00:56:30.000Z","created_at_i":1712105790,"id":39912529,"options":[],"parent_id":39911165,"points":null,"story_id":39907468,"text":"Assuming the data set is proprietary, else please share the repo","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T21:41:15.000Z","created_at_i":1712094075,"id":39911165,"options":[],"parent_id":39907468,"points":null,"story_id":39907468,"text":"For anyone who didn't bother looking deeper, the SWEbench benchmark contains only Python code projects, so it is not representative of all the programing languages and frameworks.
I'm working on a more general SWE task eval framework in JS for arbitrary language and framework now (for starter JS/TS, SQL and Python), for my own prompt engineering product.
Hit me up if you are interested.","title":null,"type":"comment","url":null},{"author":"danenania","children":[{"author":"etheridev","children":[{"author":"danenania","children":[],"created_at":"2024-04-02T22:16:47.000Z","created_at_i":1712096207,"id":39911458,"options":[],"parent_id":39911350,"points":null,"story_id":39907468,"text":"I think we'll get there at some point, but one thing I've learned from this project is how difficult it is to stack AI interactions. Each little bit of AI-based logic that gets added tends to fail terribly at first. Only after a long period of intense testing and iteration does it become remotely usable. The more you are combining different kinds of tasks, the more difficult it gets.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T22:04:56.000Z","created_at_i":1712095496,"id":39911350,"options":[],"parent_id":39911324,"points":null,"story_id":39907468,"text":"You need to make yourself a business analyst agent to provide the feedback! To make it real, perhaps a team of them with conflicting personalities.","title":null,"type":"comment","url":null},{"author":"panqueca","children":[{"author":"danenania","children":[],"created_at":"2024-04-03T01:35:15.000Z","created_at_i":1712108115,"id":39912726,"options":[],"parent_id":39912604,"points":null,"story_id":39907468,"text":"Yes, at least up to the point of the context limit of the underlying model. If you needed to go beyond that, you would break the work up into separate "plans" (a plan is a set of tasks with an attached context and conversation).
The general workflow is to load some relevant context (could be a few files, an entire directory, a glob pattern, a URL, or piped in data), then send a prompt. Quick example:
plandex new\n plandex load components/some-component.ts lib/api.ts package.json https://react.dev/reference/react/hooks\n plan tell "Update the component in components/some- \n components.ts to load data from the 'fetchFooBars' \n function in 'lib/api.ts' and then display it in a \n datagrid. Use a suitable datagrid library."\nOne thing I like about it for large codebases compared to IDE-based tools I've tried is that it gives me precise control over context. A lot of tools try to index the whole codebase and it's pretty opaque--you never really know what the model is working with.","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T01:09:31.000Z","created_at_i":1712106571,"id":39912604,"options":[],"parent_id":39911324,"points":null,"story_id":39907468,"text":"Does it work with a large existing codebase?","title":null,"type":"comment","url":null},{"author":"barfbagginus","children":[{"author":"danenania","children":[{"author":"barfbagginus","children":[],"created_at":"2024-04-03T21:34:53.000Z","created_at_i":1712180093,"id":39923495,"options":[],"parent_id":39913301,"points":null,"story_id":39907468,"text":"Did I miss the plandex-cloud repo? It seems like it's proprietary at this time. I couldn't find the AWS design, billing system, user dashboards, and admin dashboards.
Can you point me to the missing code?","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T03:22:08.000Z","created_at_i":1712114528,"id":39913301,"options":[],"parent_id":39912811,"points":null,"story_id":39907468,"text":"It\u2019s AGPL licensed already :)","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T01:49:15.000Z","created_at_i":1712108955,"id":39912811,"options":[],"parent_id":39911324,"points":null,"story_id":39907468,"text":"How open are you to moving plandex cloud over to AGPL? I know, tough ask right out the gate! Think about that one for a bit.
How is your market testing going?
Do you have contracts with clients amenable to let you write case studies? Do you need help selling, designing, or fulfilling these kinds of pilot contacts?
What are your plans for docs a PR?
As a researcher, it's currently hard to situate plandex against existing research, or anticipate where a technical contribution is needed.
As a business owner, it's currently hard to visualize plandex's impact on a business workflow.
Are you open to producing a technical report? Detail plandex methodology, benchmark efficiency, ablation tests for key contributions, customer case studies, relevant research papers, and next steps/help needed.
What do you think?
If plandex is interested in being a fully open org, then I'd be interested in seeing it find its market footing and grow its technical capabilities. We need open source orgs like this!","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T22:01:40.000Z","created_at_i":1712095300,"id":39911324,"options":[],"parent_id":39907468,"points":null,"story_id":39907468,"text":"I'm working on a somewhat similar project: https://github.com/plandex-ai/plandex
While the overall goal is to build arbitrarily large, complex features and projects that are too much for ChatGPT or IDE-based tools, another aspect that I've put a lot of focus on is how to handle mistakes and corrections when the model starts going off the rails. Changes are accumulated in a protected sandbox separate from your project files, a diff review TUI is included that allows for bad changes to be rejected, all actions are version-controlled so you can easily go backwards and try a different approach, and branches are also included for trying out multiple approaches.
I think nailing this developer-AI feedback loop is the key to getting authentic productivity gains. We shouldn't just ask how well a coding tool can pass benchmarks, but what the failure case looks like when things go wrong.","title":null,"type":"comment","url":null},{"author":"dimal","children":[{"author":"drcode","children":[{"author":"dimal","children":[{"author":"hvis","children":[{"author":"citrin_ru","children":[],"created_at":"2024-04-03T07:08:29.000Z","created_at_i":1712128109,"id":39914443,"options":[],"parent_id":39912171,"points":null,"story_id":39907468,"text":"I sometimes write a detailed bug reports but not a PR when there are different ways to address the problem (and all look bad to me) or the fix can introduce new problems. But I would expect LLM to ignore tradeoffs and choose an option which not necessarily the best for the same reason I hesitate - luck of understanding of this specific project.","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T00:02:04.000Z","created_at_i":1712102524,"id":39912171,"options":[],"parent_id":39912059,"points":null,"story_id":39907468,"text":"To be fair, sometimes meticulous users investigate the bugs and write down logical chains explaining the causes and even offer a solution at the end (which they can't apply for the lack of commit access, for instance).
The proposed solution isn't always right, of course, but it would be incorrect to say that no bug reports come with a diagnosed cause. But that's exactly where a conscious reviewer is most needed, I believe.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T23:44:36.000Z","created_at_i":1712101476,"id":39912059,"options":[],"parent_id":39911606,"points":null,"story_id":39907468,"text":"I suppose I should nail down my point. No one would ever write a big report like this. A bug generally has an unknown cause. Once you found the cause of the bug, you\u2019d fix it. Nowadays, you could just cut and paste the problem into ChatGPT and get the answer right then. So why would anyone ever log this bug? All this demo proves that they automated a process that didn\u2019t need automation.","title":null,"type":"comment","url":null},{"author":"skywhopper","children":[],"created_at":"2024-04-03T01:59:54.000Z","created_at_i":1712109594,"id":39912868,"options":[],"parent_id":39911606,"points":null,"story_id":39907468,"text":"It fixes 12% of their benchmark suite, not 12% of bug reports.","title":null,"type":"comment","url":null},{"author":"medellin","children":[],"created_at":"2024-04-03T02:42:20.000Z","created_at_i":1712112140,"id":39913097,"options":[],"parent_id":39911606,"points":null,"story_id":39907468,"text":"In my 15 years i would say less than 1% of bug reports are like this. If you know the bug to this level most people just would fix it themselves","title":null,"type":"comment","url":null},{"author":"aiauthoritydev2","children":[{"author":"littlestymaar","children":[],"created_at":"2024-04-03T07:30:31.000Z","created_at_i":1712129431,"id":39914553,"options":[],"parent_id":39913506,"points":null,"story_id":39907468,"text":"Except this is automated, so you could get multiples orders of magnitude more bug filled, so you need to have a very low false positive ratio to avoid being overwhelmed by automatically generated crap (which is basically spam).","title":null,"type":"comment","url":null},{"author":"sitkack","children":[{"author":"killingtime74","children":[{"author":"sitkack","children":[],"created_at":"2024-04-03T19:21:18.000Z","created_at_i":1712172078,"id":39921822,"options":[],"parent_id":39921527,"points":null,"story_id":39907468,"text":"It is and it will!","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T18:59:17.000Z","created_at_i":1712170757,"id":39921527,"options":[],"parent_id":39914769,"points":null,"story_id":39907468,"text":"Why not have LLM write AGI while you're at it","title":null,"type":"comment","url":null},{"author":"fasa99","children":[],"created_at":"2024-04-04T14:42:46.000Z","created_at_i":1712241766,"id":39931050,"options":[],"parent_id":39914769,"points":null,"story_id":39907468,"text":"You'd want three LLMs, one to create the bugs, one to report it, one to fix it. I joke of course but on the other hand this is potentially a worthwhile architecture from a self-training perspective - a bug-creating LLM means your training set size is as big as you want it +/- GAN features.","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T08:12:11.000Z","created_at_i":1712131931,"id":39914769,"options":[],"parent_id":39913506,"points":null,"story_id":39907468,"text":"Have the LLM rewrite the bug reports.","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T04:05:22.000Z","created_at_i":1712117122,"id":39913506,"options":[],"parent_id":39911606,"points":null,"story_id":39907468,"text":"12% is a very very large number for that kind of problem. I doubt even 0.1% of bug reports in the wild are that well written.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T22:38:54.000Z","created_at_i":1712097534,"id":39911606,"options":[],"parent_id":39911375,"points":null,"story_id":39907468,"text":"> Most bug reports you get in the wild are more along the lines of
Since this fixes 12% of the bugs, the authors of the paper probably agree with you that 100-12= 88%, and hence "most bugs" don't have nicely written bug reports.","title":null,"type":"comment","url":null},{"author":"megablast","children":[{"author":"skywhopper","children":[],"created_at":"2024-04-03T02:05:38.000Z","created_at_i":1712109938,"id":39912891,"options":[],"parent_id":39912432,"points":null,"story_id":39907468,"text":"On the contrary, it\u2019s worse than useless. If it could fix 12% of bugs (it can\u2019t \u2014 it only fixes 12% of their benchmark suite), you\u2019d still have to figure out which 12% of the responses it gave were good. So, 88% of the time you\u2019d have wasted time confirming a \u201cfix\u201d that doesn\u2019t work. But it\u2019s worse than that. Because even on the fixes it got right, you\u2019d still have to fully vet it, because this tool doesn\u2019t know when it can\u2019t solve something, or ask for clarification. It just gives a wrong answer.
So you didn\u2019t save 12% of your effort, you wasted probably more than double your effort checking the work of a tool that is wrong eight out of nine times.","title":null,"type":"comment","url":null},{"author":"dimal","children":[],"created_at":"2024-04-03T19:32:44.000Z","created_at_i":1712172764,"id":39921975,"options":[],"parent_id":39912432,"points":null,"story_id":39907468,"text":"That\u2019s not what I said and you know it. I\u2019m not saying LLMs are useless. I\u2019m not even saying this tool is useless. I\u2019m saying I\u2019m not impressed with this tool, at least as represented in the demo.","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T00:39:40.000Z","created_at_i":1712104780,"id":39912432,"options":[],"parent_id":39911375,"points":null,"story_id":39907468,"text":"Exactly. This is not perfect and doesn't fix every report so it is useless.","title":null,"type":"comment","url":null},{"author":"codeonline","children":[],"created_at":"2024-04-03T02:36:32.000Z","created_at_i":1712111792,"id":39913067,"options":[],"parent_id":39911375,"points":null,"story_id":39907468,"text":"I agree that bugs aren't as well specified as the example. But a specification for a new feature certainly can be.
I'm going to give it a try on my side project and see if it can at least provide a hint or some guidance on the development of small new features in an existing well structured project.","title":null,"type":"comment","url":null},{"author":"bee_rider","children":[{"author":"bfdm","children":[{"author":"throwup238","children":[{"author":"ffsm8","children":[{"author":"shabble","children":[{"author":"bee_rider","children":[],"created_at":"2024-04-03T14:29:40.000Z","created_at_i":1712154580,"id":39917990,"options":[],"parent_id":39916388,"points":null,"story_id":39907468,"text":"This whole \u201cworrying about KPI\u2019s for my free, open source, community project\u201d thing seems weird to me. (Not to say I don\u2019t believe you, but I don\u2019t understand why people want to inject this annoying mini-game into their hobby).
I\u2019m not sure what to think about the auto-close bots. Which do you think would be more annoying as the person who made the report: having a report that just sits there forever and you just have to hope somebody decided to pick it up, or having the issue auto-closed? (I\u2019m truly and honestly not sure). At least in the case of the former you have a clear marker for when you should try again. But getting rejected by a bot can definitely be annoying.","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T12:10:03.000Z","created_at_i":1712146203,"id":39916388,"options":[],"parent_id":39914731,"points":null,"story_id":39907468,"text":""ratio of open/total issues" can definitely be gamed by autoclosing anything that isn't an easy fix.
"average time to resolution" is also susceptible.
Both of these are pretty common all over the place, including OSS\ne.g. https://isitmaintained.com/#metrics
I suspect this sort of thing is one of the major motivations for the (as a user/reporter) infuriating rise in automated "this bug hasn't been touched in NN days, autoclosing for staleness" bots on various issue trackers.","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T08:04:36.000Z","created_at_i":1712131476,"id":39914731,"options":[],"parent_id":39914021,"points":null,"story_id":39907468,"text":"KPI stands for key performance indicator. It is a tool to grade people or teams by applying numbers to their work.
The only relationship you can have between these is that a ticket with a "resolved" status can be used as a KPI, but you're trying to invert the relationship here, which doesn't work. After all, it's an indicator and not a causal relationship","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T05:46:57.000Z","created_at_i":1712123217,"id":39914021,"options":[],"parent_id":39913268,"points":null,"story_id":39907468,"text":"I think that depends on the exact KPI.","title":null,"type":"comment","url":null},{"author":"bee_rider","children":[],"created_at":"2024-04-03T14:44:02.000Z","created_at_i":1712155442,"id":39918168,"options":[],"parent_id":39913268,"points":null,"story_id":39907468,"text":"I was thinking in the context of an open source project, where the users are hopefully converting to productive community members. If it is, like, a job, with a customer service relationship, where they are paying to be able to just throw problems at you and you have to deal with fixing them, I\u2019m sure this wouldn\u2019t fly, so I agree there. (I think my brain short-circuited to open source because it is on GitHub, haha, but of course there\u2019s no reason this couldn\u2019t be used in a proprietary setting).
I\u2019m not sure how it would work out in the case of a free, community driven project, though. The goal isn\u2019t to serve users, it is to convert users into helpful community members. If the bot converts people who wouldn\u2019t otherwise be converted, it seems like a win. If it chases away users who could have been converted with human intervention, that\u2019s a lose. But the human community members can always jump into the thread as well\u2026 if the bot is filtering out lots of people and nobody from the community is intervening, I guess that tells us something about the priorities of the community, haha.","title":null,"type":"comment","url":null},{"author":"mdaniel","children":[],"created_at":"2024-04-03T16:06:49.000Z","created_at_i":1712160409,"id":39919218,"options":[],"parent_id":39913268,"points":null,"story_id":39907468,"text":"Oh, so you've experienced those "stale" bots on GitHub. Good times.","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T03:15:31.000Z","created_at_i":1712114131,"id":39913268,"options":[],"parent_id":39913241,"points":null,"story_id":39907468,"text":"While it might tickle metrics the right way, frustrating a user into giving up because your bot was not satisfied is not solving their problem.","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T03:09:18.000Z","created_at_i":1712113758,"id":39913241,"options":[],"parent_id":39911375,"points":null,"story_id":39907468,"text":"Maybe it just needs another, independent tool. One that detects poorly written bug reports and rejects them.
A cool thing about LLM is they have infinite patience. They can go back and forth with the user until they either sort out how to make a useable bug report, or give up.","title":null,"type":"comment","url":null},{"author":"stingraycharles","children":[],"created_at":"2024-04-03T05:32:26.000Z","created_at_i":1712122346,"id":39913923,"options":[],"parent_id":39911375,"points":null,"story_id":39907468,"text":"It appears that they\u2019re using the PRs from the top5000 most popular PyPi packages for their bench: https://github.com/princeton-nlp/SWE-bench/tree/main/swebenc...","title":null,"type":"comment","url":null},{"author":"forty","children":[],"created_at":"2024-04-03T07:29:13.000Z","created_at_i":1712129353,"id":39914546,"options":[],"parent_id":39911375,"points":null,"story_id":39907468,"text":"The trick is that people would use LLM to write very long and detailed bug reports :p","title":null,"type":"comment","url":null},{"author":"jcarrano","children":[],"created_at":"2024-04-03T09:28:21.000Z","created_at_i":1712136501,"id":39915153,"options":[],"parent_id":39911375,"points":null,"story_id":39907468,"text":"Maybe it would be better if the agent would help people submit better reports instead of trying to fix it. E.g. it could ask them to add missing information, test different combinations of inputs, etc. I could also learn which maintainer to ping according to the type of issue.","title":null,"type":"comment","url":null},{"author":"gorjusborg","children":[],"created_at":"2024-04-03T11:37:35.000Z","created_at_i":1712144255,"id":39916113,"options":[],"parent_id":39911375,"points":null,"story_id":39907468,"text":"If the bug report needs to be of a certain quality to work, they've just invented issue-oriented programming.","title":null,"type":"comment","url":null},{"author":"chinchilla2020","children":[],"created_at":"2024-04-03T15:22:29.000Z","created_at_i":1712157749,"id":39918655,"options":[],"parent_id":39911375,"points":null,"story_id":39907468,"text":"Agreed. I have never encountered a simple math bug in the wild.
To a non-programmer, putting in tests for myfunc(x) {return x + 2;} sounds useful but in reality computers do not tend to have any issues performing basic algebra.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T22:07:56.000Z","created_at_i":1712095676,"id":39911375,"options":[],"parent_id":39907468,"points":null,"story_id":39907468,"text":"The demo shows a very clearly written bug report about a matrix operation that\u2019s producing an unexpected output. Umm\u2026 no. Most bug reports you get in the wild are more along the lines of \u201cI clicked on on X and Y happened\u201d then if you\u2019re lucky they\u2019ll say \u201cand I expected Z\u201d. Usually the Z expectation is left for the reader to fill in because as human users we understand the expectations.
The difficulty in fixing a bug is in figuring out what\u2019s causing the bug. If you know it\u2019s caused by an incorrect operation, and we know that LLMs can fix simple defects like this, what does this prove?
Has anyone dug through the paper yet to see what the rest of the issues look like? And what the diffs look like? I suppose I\u2019ll try when I have a sec.","title":null,"type":"comment","url":null},{"author":"readthenotes1","children":[],"created_at":"2024-04-02T22:10:11.000Z","created_at_i":1712095811,"id":39911394,"options":[],"parent_id":39907468,"points":null,"story_id":39907468,"text":"I made a lot of money as I was paid hourly while working with a cadre of people I called "the defect generators".
I'm kind of sad that future generations will not have that experience...","title":null,"type":"comment","url":null},{"author":"tibbetts","children":[{"author":"worthless-trash","children":[],"created_at":"2024-04-03T02:58:48.000Z","created_at_i":1712113128,"id":39913185,"options":[],"parent_id":39912841,"points":null,"story_id":39907468,"text":"Copilot already does this.","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T01:53:36.000Z","created_at_i":1712109216,"id":39912841,"options":[],"parent_id":39907468,"points":null,"story_id":39907468,"text":"But can their AI quietly introduce a security exploit into a GitHub project?","title":null,"type":"comment","url":null},{"author":"barfbagginus","children":[{"author":"SrslyJosh","children":[],"created_at":"2024-04-03T02:17:57.000Z","created_at_i":1712110677,"id":39912966,"options":[],"parent_id":39912882,"points":null,"story_id":39907468,"text":"...or you could just use the GitHub API to find projects that match certain criteria (e.g., no pyproject.toml). I'm not sure what the stochastic parrot adds here, besides making noob mistakes that you'll have to find and fix before you can submit PRs. You'd learn a lot more by trying to actually automate the process yourself.","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T02:02:59.000Z","created_at_i":1712109779,"id":39912882,"options":[],"parent_id":39907468,"points":null,"story_id":39907468,"text":"I would like something like this that helps me, as a green developer, find open source projects to contribute to.
For instance, I recently learned about how to replace setup.py with pyproject.toml for a large number of projects. I also learned how to publish packages to pypi. These changes significantly improve project ease and accessibility, and are very easy to do.
The main thing that holds people back is that python packaging documentation is notoriously cryptic - well I've already paid that cost, and now it's easy!
So I'm thinking of finding projects that are healthy, but haven't focused on modernizing their packaging or distributing their project through pypi.
I'd build human + agent based tooling to help me find candidates, propose the improvement to existing maintainers, then implement and deliver.
I could maybe upgrade 100 projects, then write up the adventure.
Anyone have inspiration/similar ideas, and wanna brainstorm?","title":null,"type":"comment","url":null},{"author":"JonChesterfield","children":[{"author":"itsgrimetime","children":[],"created_at":"2024-04-03T06:06:30.000Z","created_at_i":1712124390,"id":39914129,"options":[],"parent_id":39913833,"points":null,"story_id":39907468,"text":"It\u2019ll likely keep getting better, if it gets to 30-40% I\u2019d say that\u2019s a decent trade off. Also could you boost your chances by having the AI do a 2nd pass and double check the work? I\u2019d be curious what the success rate of an LLM \u201cdetermining whether a bug fix is valid\u201d is","title":null,"type":"comment","url":null},{"author":"CGamesPlay","children":[{"author":"JonChesterfield","children":[],"created_at":"2024-04-03T06:51:30.000Z","created_at_i":1712127090,"id":39914361,"options":[],"parent_id":39914209,"points":null,"story_id":39907468,"text":"And thus issues will no longer be opened","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T06:22:02.000Z","created_at_i":1712125322,"id":39914209,"options":[],"parent_id":39913833,"points":null,"story_id":39907468,"text":"Not a chance. If AI-generated pull requests become popular, GitHub will automatically offer them in response to opened issues. Case in point: they already are popular for dependency upgrades.","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T05:16:45.000Z","created_at_i":1712121405,"id":39913833,"options":[],"parent_id":39907468,"points":null,"story_id":39907468,"text":"If AI generated pull requests become a popular thing we'll see the end of public bug trackers.
(not because bugs will be gone - because the cost of reviewing the PR vs the benefit gained to the project will be a substantial net loss)","title":null,"type":"comment","url":null},{"author":"pjmlp","children":[{"author":"iLoveOncall","children":[{"author":"pjmlp","children":[],"created_at":"2024-04-03T07:53:47.000Z","created_at_i":1712130827,"id":39914678,"options":[],"parent_id":39914574,"points":null,"story_id":39907468,"text":"Might be, still the point I was trying to make remains.","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T07:34:39.000Z","created_at_i":1712129679,"id":39914574,"options":[],"parent_id":39914479,"points":null,"story_id":39907468,"text":"Flipping burgers will be automated long before AI fixes any relevant number of bug reports.","title":null,"type":"comment","url":null},{"author":"fennecfoxy","children":[],"created_at":"2024-04-03T12:58:48.000Z","created_at_i":1712149128,"id":39916848,"options":[],"parent_id":39914479,"points":null,"story_id":39907468,"text":"I still think this is a long way off, but it definitely ties into UBI etc and improvement of the general human condition, taxing the rich, restricting investment on protected things like housing and public industries like water, electricity, healthcare and internet.
What's funny is that people on here & tech people in general seem to be the most averse to improving equity between all humans/stopping the obscenely rich from abusing and twisting the system. Do many HN peeps believe they're all somehow gonna become billionaires one day?","title":null,"type":"comment","url":null},{"author":"littlestymaar","children":[],"created_at":"2024-04-03T20:07:59.000Z","created_at_i":1712174879,"id":39922420,"options":[],"parent_id":39914479,"points":null,"story_id":39907468,"text":"Why would a human ever flip a burger at that point? It's not a particularly difficult task for a robot.
Unclogging sewers on the other hand\u2026","title":null,"type":"comment","url":null}],"created_at":"2024-04-03T07:15:40.000Z","created_at_i":1712128540,"id":39914479,"options":[],"parent_id":39907468,"points":null,"story_id":39907468,"text":"Eventually it will be 90% fix rate and everyone cheering for the 12% will be flipping burgers instead.","title":null,"type":"comment","url":null},{"author":"Madmallard","children":[],"created_at":"2024-04-03T07:22:57.000Z","created_at_i":1712128977,"id":39914512,"options":[],"parent_id":39907468,"points":null,"story_id":39907468,"text":"What veterans in the field know that AI hasn\u2019t tackled is that the majority of difficulty in development is dealing with complexity and ambiguity and a lot of it has to do with communication between people in natural language as well as reasoning in natural language about your system. These things are not solved by AI as it is now. If you can fully specify what you want with all of the detail and corner cases and situation handling then at some point AI might be able to make all of that for you. Great! Unfortunately, that\u2019s the actual hard part! Not the implementation generally.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T16:16:07.000Z","created_at_i":1712074567,"id":39907468,"options":[],"parent_id":null,"points":307,"story_id":39907468,"text":null,"title":"Princeton group open sources \"SWE-agent\", with 12% fix rate for GitHub issues","type":"story","url":"https://github.com/princeton-nlp/SWE-agent"}