The Ars Technica AI coding agent test: Minesweeper edition
- Thread starter JournalBot
- Start date
I don't see it. The systems don't have any judgment. Only people have that. And no one is actually working on new tech that will have judgment.
Sure, we'll need less accountants per unit of accounting work, but we won't dispense with the expertise. And in middle management, the same story.
Upvote
0
(0
/
0)
Details of what the systems can and can't theoretically do aside, the work is all somewhat fungible.
Less per unit, means less people employed and/or inventing new work for them to do.
Upvote
3
(3
/
0)
currently, the best way to use these products is to not one shot but to take advantage of the ability to design and think through a problem, then work with it to generate a list of test cases and then to methodically work through that list. you should also have a specific context management strategy. these tools are not smart enough to manage context on their own. working in this way, im able to get really good results very quickly.
To me, this experiment basically was: lets use a tool in a non-opitmal way and then judge it.
To me, this experiment basically was: lets use a tool in a non-opitmal way and then judge it.
Upvote
0
(1
/
-1)
Which IMHO is still a valid use case. Not all code is rigorously implemented with CI/CD or even separate test/prod environments. In the workplace, one big use case for a lot of these tools will be from business people who make a tool to automate some process they do daily or even things like office automation.
These kinds of tools are so big in my company they call them end user tools. LLMs will expand the kinds of tools that get built like this and none of them will start out with a well thought out plan or any kind of test-driven development.
And that's not even considering home use cases or small office tools. A ton of use will be prompts like in this article. Something like "make me an app that lets me track X" and then they'll iterate through features that they want to add.
Upvote
3
(4
/
-1)
Not testing Grok seems like a miss. Claude, Codex, and Grok are by far the best frontier closed source coding models.
Also for open source: Deepseek or Kimi coding models would be a far better comparable, if peered with a good coding system prompt like Cline.
Also for open source: Deepseek or Kimi coding models would be a far better comparable, if peered with a good coding system prompt like Cline.
Upvote
-5
(0
/
-5)
I pasted the prompt from the article verbatim into Gemini Pro (3) and it gave me the code for a single html file which contained the HTML, CSS, and JavaScript. The game functioned perfectly well and handled flagging using long presses on mobile and right-clicks for desktop. The added bonus feature it added was that one random safe space contained a shield. When it's clicked, you now have the shield active (it shows in the header at the top). With the shield active, if you make one mistake by clicking a bomb, it automatically protects you from that bomb and just flags the square instead. So, it's like a one-time-use "get out of jail" feature that you first have to find in a safe square. It was a nice touch. The sounds also worked well, though they were the typical primitive beeps and blips.
Upvote
1
(1
/
0)
Seems odd to put this article out without trying it on Gemini 3, which is making pretty big waves in the space. It's a big step forward compared to 2.5
Upvote
-2
(2
/
-4)
I would have liked you to have tried this with either
All produce much more maintainable code, and probably would have scored a perfect 10.
o3, o4, or best, Cursor's Composer 1 model. All produce much more maintainable code, and probably would have scored a perfect 10.
Upvote
-3
(0
/
-3)
Yes and no. If the economy is a crystal frozen in time that never expands or contracts, then yes. If things change (eg more jobs, or less jobs, due to events, ancillary inventions, tastes) then no.
For example, 200 years ago the number of jobs in software was zero. Tomorrow, who knows? Things change.
Upvote
3
(3
/
0)
Bear in mind, the Journalists asked an AI to do something that is akin to a learning exercise with no productive value. The minimalism of the prompts actually indicates this more than it builds a strong case for the AI. "Clone Minesweeper" is not a realistic set of requirements, and it offers the AI an opportunity to evade in the actual problem solving involved in software development.
What this test leaves totally absent is requirements refinement. The AI was given a task that involves no need to interface with a human being whatsoever, no need to break down a problem written in English, no need to deal with a customer who isn't aware of what is and isn't reasonable in terms of software development.
This test really only shows the "best-case-scenario" for whether an AI can write functional code, but is way too detached from what makes programming a valuable skill. I'm a software engineer myself, and writing code that I already know how to write is a menial part of my job. Most of my work involves figuring out what my software actually needs to accomplish in the real world and understanding an incredibly vast project so I can modify it to make it accomplish those objectives.
With no need to refine requirements, minimal need to figure out how to accomplish a real world effect, I cannot take this as an indication of AI's ability to work on anything more advanced than the reinvention of common software tasks. Minesweeper is just too well known and easily referenced for its recreation to be evidence of the AI completing valuable legwork. The surprise features range from utterly trivial to ill-conceived: The sorts of features that would be tacked on because they're easy. All of this is, at best, a project for a sophomore software student meant to teach more than to demonstrate ability.
A demonstration of AI being able to replace a software developer will at least require it to refine requirements with a layman to create a project that hasn't been publicly dissected a million times already.
Upvote
2
(4
/
-2)
Used to play a lot of minesweeper. Does anyone remember that there was a way to cheat?
It involved typing XYZZY and some specific actions with the mouse. Once the cheat was activated, a pixel in the top corner of the screen (not the game window) would light up when the
mouse was hovering over a square that concealed a mine.
It involved typing XYZZY and some specific actions with the mouse. Once the cheat was activated, a pixel in the top corner of the screen (not the game window) would light up when the
mouse was hovering over a square that concealed a mine.
Upvote
0
(0
/
0)
I share a lot your same feelings. But, there is a different way of looking at this.
You participated in an industry that documented so well and accomplished so much that you enabled the code to make the code.
This is not an accomplishment that any cabinet maker can claim.
At this point why would we not tax AI coding agents to give a fixed income salary to old coders?
Last edited:
Upvote
0
(0
/
0)
What a random tip. I have family who are volunteer firefighters in CT. I should ask what kind of software they need.
Upvote
1
(1
/
0)
Unrelated, kind of, I asked ChatGPT if it could reverse engineer the C64 game “Drop Zone”. It offered to look at the 6502, interrupt (VBL) code, how to export the graphics and create a player to emulate the sfx. To convert to C++ and use Cocos as a framework.
I suspect, with guidance and a month, it could get it working.
I suspect, with guidance and a month, it could get it working.
Upvote
-1
(0
/
-1)
I explained to my colleagues that I’d be on board with vibe coding for internal applications once they invent vibe product management. As it stands, vibe coded stuff seems headed for exactly the same pit in hell where all those write-once-read-never VBA apps ended up, except the pit is much deeper this time.
Upvote
7
(7
/
0)
Colleges and Universities failed Businesses… no fan of higher education and no Booster of Business, I went through the ComSci program early 70’s - mainframe era. Took what was offered at local college; 13th dimensional Calculus researcher on Sabbatical teaching programming languages.
That was pre unix*. In 1991, went back to Colombia to refresh on the unix bits.
WHAM!!
Immediately, AI hits the proverbial wall – "12 lines code/ programmer/ day"
This article highlights in what way the technology best operates - at low granularity level i.e. Minecraft .vs. functional description.
Modularity, AI existentially favors as its preferred eco-system. That may foster an organizing efficiency multiplier i.e. reuse for future integration with developer environs.
As a test case building block that’s my useful takeaway. How language, name space dependency is projecting a new "patent" system for softwares is AI' limiting orthogonal factor.
Upvote
0
(0
/
0)
The cost to the environment was higher than the dollar cost.
Upvote
2
(3
/
-1)
You kind of miss the point. I expected it to steal one that it found elsewhere in Github. But the main point is that it claimed to do so, appeared to run, and wasn't useful.
Upvote
1
(1
/
0)
You're right, I wasn't aware that those were your points.
Are there a lot of RL pipeline examples to steal from on Github? I guess I just googled it and found one for Connect 4 specifically but it's unclear how well (or if) it works.
What you're asking for is pretty complicated stuff. If anything is sub-optimal about the neural network architecture, or the hyperparameters, you're not going to get a good result.
I just asked ChatGPT what neural network architecture it would use for Connect 4 and it's assuring me that three convolutional layers is plenty, which strikes me as extremely suspect, since that wouldn't even allow information to propagate from one side of the 7x6 board to the other side...
Upvote
2
(2
/
0)
I'd like to see a similar experiment with more iteration. LLMs seem okayish at spitting out somewhat reasonable code on the first go around but basically every LLM I've tried (and similar results in discussions with other people and online content) points to them slowly destroying the programs when asked to debug themselves. Usually after one or two fixes they just start degrading existing features until the program eventually stops working entirely and the AI enters a death spiral.
Upvote
1
(2
/
-1)
Bolding mine.
That shit pisses me off more than any computer game ever has. Spend an hour getting a pic aaaaalmoooost right and suddenly there's a random icecream truck and the dwarf king's face has gone from beard to Wookie vagina.
I'm just an occasional/light hobbyist, but yeah, I do more work in Gimp2 than in my prompt engineering. And actually have been trying out different ways to build mosaic-type/layered pics as you mentioned (thanks for the validation on that).
Back on topic: My coding skills are exactly equal to "use Notepad to edit computer game files that open in Notepad / use Weidu to make my own NPCs for Baldur's Gate 1&2 /figure out what works through trial and error."
But even I look at someone just copy/pasting generated code (or frikken legal documents you're handing to a judge, FFS) without checking it first as "god I hope they don't breed".
Upvote
0
(1
/
-1)
I get the stress, more than most since I've been homeless before. Many places have free legal help, and you would be surprised how long you can tie it up in court just to keep a roof over your head a little bit longer, so if you haven't already I strongly advise looking into that.
But when it comes down to it: Can you think of a single person on their deathbed that ever uttered "I wish I spent more time working" as they shuffled off this mortal coil?
Upvote
2
(2
/
0)
How many H-1B visa coders are these going to displace? How many U.S. citizen coders? Is "all of them" the asymptotic answer?
Upvote
-1
(0
/
-1)
The problem is not time spent working, which I'd like to reduce myself, but time one can get paid for working. AI (and robotics, and conventional programs) are going to continue to displace human labor. Will they generate new job opportunities as they do so? Will these be as many as the ones lost? Will humans be better than machines at the new jobs? Or will they just go immediately to machines?
My answers are Some, No, Mostly not, and Mostly yes.
Things will be worst across the board for new entrants to fields like coding, art, writing, teaching, long-haul trucking, radiology, and so on. Senior experts / seasoned pros will be needed for some time to come in most of those fields, certainly long enough for senior incumbents to reach retirement, but AI is going to hoover up the entry-level work at a rate that may actually eliminate the entry level in multiple present-day professions. There is a coming crisis for new graduates, and nothing resembling a plan to address it in any nation I'm aware of.
Upvote
1
(1
/
0)
Personally, I'm hoping they displace enough human labor that we finally get the point and just move past this outdated "everybody has to have a job or starve in the streets" crap. I always kinda figured the whole point of technology was to do our work for us, and with fiat currency literally able to be produced at will in any denominations or amounts needed (thus the "fiat" part), the scarcity of $ is artificial and arbitrary.
I seriously doubt it will happen, because deep down us Homo sapiens are not Good or Evil (we're just mammals) and are probably going to go extinct soon because we're also a pretty stupid mammals that generally only think about ourselves and the content of our own lives.
But I'm also one of those weirdos that considers short-term future to be "next 100 million years". And I'm autistic, so beyond the immediate "gotta eat to live" concept, ya'lls weird obsession with jobs is honestly just insane to me.
And thus, if your job is the most important thing in your life, you should probably try to get a better life.
And finally, to all those dipshit libertarians out there that are extremely offended by these beliefs:
Upvote
1
(1
/
0)
Why? just because some AI built a game that has been implemented hundreds of times over on publicly available codebases as open source? Come on, now...
Upvote
-1
(0
/
-1)
Upvote
-4
(0
/
-4)
I'm with you, though I'm much more concerned with humanity surviving over the next several centuries. Beyond that, technological and external events are so uncertain that planning is impossible.Personally, I'm hoping they displace enough human labor that we finally get the point and just move past this outdated "everybody has to have a job or starve in the streets" crap. I always kinda figured the whole point of technology was to do our work for us, and with fiat currency literally able to be produced at will in any denominations or amounts needed (thus the "fiat" part), the scarcity of $ is artificial and arbitrary.
I seriously doubt it will happen, because deep down us Homo sapiens are not Good or Evil (we're just mammals) and are probably going to go extinct soon because we're also a pretty stupid mammals that generally only think about ourselves and the content of our own lives.
But I'm also one of those weirdos that considers short-term future to be "next 100 million years". And I'm autistic, so beyond the immediate "gotta eat to live" concept, ya'lls weird obsession with jobs is honestly just insane to me.
And thus, if your job is the most important thing in your life, you should probably try to get a better life.
And finally, to all those dipshit libertarians out there that are extremely offended by these beliefs:
IME, humanity is inherently greedy and lacking in empathy. As a race, we have known what is good" for millennia, and yet our society would collapse into violent anarchy overnight but for systems of police and courts and prisons.
It's pretty simple, IMO. On an individual level, treat everyone as you would wish to be treated yourself (The Golden Rule). On a societal level, design systems of government and laws as if you had no idea if you were going to be poor / weak or rich / powerful (Veil of Ignorance). But humanity doesn't even come close to this, and we're only able to approach it in a few places where there is plentiful wealth on a per-capita basis.
Upvote
0
(1
/
-1)
There was also Progress Quest, the granddaddy of modern idle games.
Upvote
0
(0
/
0)