Wriring the code is not the challenge, but it's what was taking up most of the time. Not the typing itself, but also because I had to think of how to implement it.
Now I can just say "add 2FA" and in 5 minutes, while I test something else, it is done.
It also made iterations a lot faster, you can try something out, see how it feels, if it doesn't work, you can just trash all the code and start again.
Haven't typed a line of code or read any code for over 6 months now.
And I used to love coding and be a competitive programmer, but this is how "coding" goes nowdays.
I have a mental model of what it would do, and how it would work, and I ask questions to confirm things and tell it to watch for specific gotchas. Then simply test the feature myself a bit.
Security-wise, I think the latest cyber models are better than me anyway at finding vulnerabilitates and pentesting such features.
Plus 2FA is a very common pattern, so it likely has in the training dataset many really good implementations.
> Security-wise, I think the latest cyber models are better than me anyway at finding vulnerabilitates and pentesting such features.
I don't doubt that, but they are equally good in making mistakes, over-engineering, or adding things you never asked for. They have all sorts of patterns in their training data from excellent to inadequate and I find them challenging to guide them consistently in one direction. Also with questions and tests, they can add something extra you didnt need and you dont know about, so your scrutinizing questions and test cases could miss that.
At least for myself, I didnt find them reliable enough yet to do what you describe and just not look at the code at all.
As someone who has an ide open with claude/codex running, I can never imagine leaving security up to models.
Models have found vulnerabilities that i wasnt aware of sure, but their fixes to the bugs they found often included "overengineering". In this case by "overengineering" i mean optimizing for passing test cases related to said vulnerability they found. eventually i have to step in to make things coherent and make sure that future agent can look at this part of my code and copy it to not introduce that particular class of vulnerability. Otherwise if i dont do that similar vulnerability and codesmell keep appearing throughtout the codebase.
I have increasingly automated encrypting and rotating secrets and setting permissions on them including better network level practices. Thanks to AI which helped me quickly implement those. So security wise i am better because of AI? But I also attribute it to my know how rather than the AI because I have never seen AI suggest robust but simple security postures.
This app[0] is entirely "vibe-coded", started on bolt.new which was like the OG way to vibe code and slowly improved over 1-2 years using various models.
I have no idea how the code looks like, and barely even tested the app entirely, because it is still not released yet, but I do think a lot about new features, tweaks, improvements, etc. Years of programming and game development did help, but I don't think anymore that code is relevant, as long as it looks ok and feels good.
Sort of, it's more of a heurisric of bugs found so far. There are very rarely any bugs, most of the changes I request are for tweaks/logic changes, not bug fixes. Usually things just work.
Also, I barely tested and kept changing things simply because of this: whatever I ask for, seems to work as expected.
No, it was about testing only. I will only do a full test of the app when it's ready to be released, otherwise it makes no sense, as the app/product/flow still changes a lot. It's like prototyping a game, you try it a few times, see what's fun, what's not, and keep iterating. Only once you think it's good enough, you can start fully testing and polishing it.
Even after release, I don't see reasons to check the code if everything works and people are happy with the app.
Right, I missed that context that this was about your own app still in getting the idea right/prototype phase. I was talking more in general and would be quite concerned if with today's LLMs it was my bank/government backend system, handling my financials and personal data, written without no one ever looking at the code.
Firefox had an unsupported missing WebAudio. It should be fixed now, and I've updated my workflow to also test on Firefox. Let me know if the crash was in a different browser (not Firefox).
It’s equally good at catching mistakes as it is at making them. So you put it in a loop, write code, make mistakes, catch mistakes, write code, make mistakes… etc. Eventually it converges and stabilizes. That process can take weeks. So the question is: when is it worth it versus not?
You are abdicating your responsibility, which is fine for toy apps for yourself, but less fine when people expect that 2fa implementation to protect their accounts.
I think it's the opposite, with the right guideance, testing, frameworks, and using the top models today, implementing something with AI is most of the times better than what most developers would do.
I basically only manually test e2e myself, other tests are automated, code review is automated. I can tell the model to test for me too specific things or to add tests for specific potential issues, performance benchmarks, compared different implementations, etc.
The focus is a lot more around the code than on the code.
1. If what you say is working, you have a working software factory that should be capable of matching the output of dozens of engineers.
What very impressive externally verifiable results have you had with this?
2. If you are working in software with plenty of customers, my strong suspicion is that there are people on your team who are looking at the code who furiously trying to reign in your output.
I make my own software, so no team to look over the code or be bothered by it. I don't even know how "vibe-coding" works in a team environment, because for me now it feels like it's "ideas to app" directly, so dumping by brain/ideas directly into a functional product.
I think this one is really cool[0], will be a free piano learning app. I do have other projects, but they are all at around 80% too, because some systems are shared amongst the projects and have to be finalized too (i.e. now I'm implementing my own transactional/marketing email service on top of Amazon SES, I need it before releasing ultimidi so people can register and receive email confirmations).
That is a cool app. But if you’d said you’d made it by hand in a few months I would have believed you.
If I had a working software factory like you describe, I’d expect you to have hundreds of apps of that level of complexity in a year.
> I don't even know how "vibe-coding" works in a team environment, because for me now it feels like it's "ideas to app" directly, so dumping by brain/ideas directly into a functional product.
But you don’t have any users much less paying users, so you have no idea if this system works when you do.
> I’d expect you to have hundreds of apps of that level of complexity in a year.
I am working at around 7-8 projects at the same time. The limit becomes me having to remember what I was doing for each one. AI can implement things nicely, but it really sucks at deciding which features and having its own ideas about novel game mechanics or UI/UX patterns.
I don't consider it a "software factory", just a more robust way to implement features, and it's still a WIP. One system I implemented locally is called "TaskHub", which receives some implementation or testing details from a SoTA mosel like Astra and implements it locally using Qwen 3.8 27b running on a RTX 3090.
I delegate things like app testing, navigating the app and taking screenshots, analyzing screenshots, etc.
And yes, this specific app doesn't have any users yet, I will launch it next week after I finalize the user authentication, payment flows and mobile apps. I will release it mostly as is, see user response, and then change it accordingly.
> I don't even know how "vibe-coding" works in a team environment, because for me now it feels like it's "ideas to app" directly, so dumping by brain/ideas directly into a functional product.
If you have a system that does “ideas to app directly”, that certainly sounds like what people are talking about when they say software factory. In fact my very large company would probably pay you millions if you could come in and make this work for our applications.
Hmm, my phrasing was ambigous, I didn't mean it's one idea fo one fully functional app.
But more of, one feature idea, that the AI can add to an app.
So it's not prompting "a gamefied piano learning app", but adding a single new feature like "add a minigame, where the user can control a character on the musical staff, [... 20 paraphs later ... ], make a good implementation plan, implement and test the app until everything is production-ready and bug free /goal"
That is a single "idea" that also requires a lot of fine-tuning after, but rarely in terms of code, most of the times it's only in terms of functionality and design choices.
> What very impressive externally verifiable results have you had with this?
I notice this weird hostility whenever the topic of AI coding comes up and it's never made much sense to me. If someone told me about their new method for practicing guitar I'd feel like a real tool if I demanded they prove it for me then and there.
People don't owe you their "very impressive externally verifiable results" - /u/XCSme already posted their app in another comment, it looked fine to me.
You've made your ideological position very clear here, you don't need to keep heaping it on.
More like, if someone told you their workout program that takes 5 minutes a day lets them lift the same weights as an average Olympic weightlifter, you'd ask to see the results.
I don’t care what somebody believes about the pile of code they’ve got sitting on their own computer. The appropriate level of engagement is sort of: well, they don’t owe us any evidence we don’t owe them any credulity, and if we’re all happy to ignore each other that’s fine.
I can see why some folks here want to quibble, though. In other comments in this thread, they’ve compared to code quality favorably to the output of an average developer. That’s slightly insulting to the field in general (although, I guess most programmers have a low opinion of average code quality).
If someone tells us about their new method for practicing guitar, it is very natural and not at all a toolish behaviour to hand them a guitar and say “go on, play us something”.
In fact it’s actually a very socially agreeable action, as it gives them the opportunity to show off their new skills without them looking like they’re bragging.
Now, if you happen to know for a fact that the guy cannot actually play guitar, then you’re just setting him up to embarrass himself, which is maybe an extreme punishment for the relatively minor crime of spouting some bullshit. I could buy that that is hostile, sure.
You don’t, that’s why in agentic world your codebase is only as good as what you can prove. For this reason, you’re going to see more languages evolving feature like capability permission, effect/coeffect types, refinement types, formal verifiers, strict type checkers, static analysis and so forth. Tests are only a small part of the verification. These ideas are old and have sat outside of the mainstream coding world, but the value proposition in the ai age has changed enough their relevance is renewed.
Not arguing for not reading test code. A lot can be achieved by instructing agents to balance out the testing pyramid with the right amount of fast end to end tests, property based tests for the right things, parametrized example based tests. Ensuring the local and CI has the right mix of tests running at right time. On projects where I have less time to review AI generated code I channel my anxiety into setting up guardrails and processes for the agents and then force them to bump up against them. In the end I view it as creating frameworks which allow me to outsource some of my attention to the agents, so that I can claw back some of that time to go set up more guardrails for more agents who are working on something else.
I trust a modern model implementing a standard feature like this much more than 99% of the people I've worked with.
People bash LLMs for overengineering but for this it's what you want. Taking extreme edge-cases into account that a human would never bother with and obsessing over security.
I mostly vibe coded a queuing system to replace something we’re using at work (last week. Spent about $1500). Then I meticulously went through the code.
It was much harder to review because it was ultra defensive and included guards for tons of edge cases that weren’t possible.
Unnecessary abstractions for possible extension later. Useless indirection. Probably 3x as much code as there would have been if I’d written it by hand.
I didn’t one shot this. I kept a pretty tight leash on the AI. I had probably a dozen markdown files with of plans that I created over hours of back and forth with the AI and reviewed before each implementation round. I had automated reviews and quality gates etc…
What I found in review was that it was full of very subtle bugs that would have bitten hard in prod. Committing offsets asynchronously that would lead to dropped messages. Clock drift bugs that would lead to dropped messages or write amplification storms. Lack of back pressure in some stages of the pipeline that would cause notes to get silently OOM killed. Weird over-insistence on never crashing in most places that would mask systemic errors.
If I’d just shipped it without review, it would have mostly worked. But at the scale it’s going to be used (tens of thousands of messages per second) it would have caused production issues for months while we tracked down each of these issues.
> included guards for tons of edge cases that weren’t possible
It's not possible until it is. This is the justification lazy developers like we all are have been using leading to bugs down the road. This glorification of hand-made code is strange, like we weren't writing dirty code full of shortcuts and hacks all the time.
The issue with LLM guarding isn't that it's "excessive" in outputting edge case handling, it's that the result often ends up just suppressing an error that actually indicates there is a bug or that should be handled elsewhere in a different way.
While I've definitely experienced it I don't think this is as much of a problem anymore. It's very easy to add an instruction to projects where you want every error to lead to a top-level throw rather than be handled. I also find that when it does try to mitigate it does so gracefully with a path you would actually make if you had infinite time, but your instinct tells you it's overkill.
> Haven't typed a line of code or read any code for over 6 months now
> I ask questions to confirm things
Oh my.
As someone who reads the code, I can tell you, asking questions to confirm things is inadequate. The models lie to me, daily.
Every day I have two experiences:
1. I’m blown away by what it can do
2. I say, ”wait, you said this, but the code shows that, so you were just going to leave that endpoint without requiring any authentication??” and I get the “you’re absolutely right, that was my mistake, and you’re right to call it out” song and dance. Daily.
It also adds all kinds of bloat to code, tests, and “documentation”. I’d say I spend ~30% of my dev time picking lines of code or documentation and asking, “why does this exist?” and “what would break if we deleted this line?” and then arguing with it and removing things.
Yeah, I agree, they are far from perfect and make a lot of mistakes.
Truth is, modern software was already quite shit and full of bugs. All major apps had bugs, issues, going down, etc, so users did get used to things not working. I honestly beleive AI coding nowadays, for better or worse, does things better than the average developer.
Yes, it is overly defensive and verbose, but the end result is in general ok and fully functional. Yes, it adds 30 tests and "release gates", and they are not even that useful, most of the times they just act as an extra safety mechanism to make parts of the code immutable, so release fails if the model accidentally changed things.
Another issue with looking at code, is that it's very hard to manually change things anyway. I can't just change a variable from 10 to 20, because I don't know where it is used. I have to ask the model to set that value to 20. It is quite stupid and inefficient, but this is one cost of coding using AI. But, if you do this, things will likely work.
That being said, I've mostly used Astra xhigh since it was released and things just work.
> Another issue with looking at code, is that it's very hard to manually change things anyway. I can't just change a variable from 10 to 20, because I don't know where it is used.
This should be a giant flashing red light. If you can't figure this out, either 1) you're too junior to be effective using AI, or 2) the AI is doing a truly awful job organizing the codebase. In either case, it's a sign to slow things down and understand what's happening before proceeding.
Is that that variable might be used in tests, UI, docs, agenr markdown files, other related projects too.
The design spec could say "always use 10px margin", then in code we have a const with value 10 used everywhere. If we update 10 manually in code to 20, then the design spec is now outdated.
Hmm, my experience is different. I noticed Astra always asks me to add credentials in a secure way, to never pasted them in chat, to rotate any potentially exposed tokens, etc.
I could probably look over, and get some snippets, but currently only the client-side is live, so the code is sort of there, albeit I assume it's minified for production.
The game seems quite bug free though, including minigames. The UI could be better, but it's not done yet.
I do for example have an automated system that simulates progression, takes screenshots of the game to find potential hidden buttons or overlapping elements, to test for performance, etc.
If it looks like a duck, and quacks like a duck, I honestly don't see why I would review 100k's of lines of code.
EDIT: I might have replied in a wrong thread, but it was about this entirely "vibe-coded" app: https://game.ultimidi.com
> If the core or that app is more than a few thousand lines of code, something is seriously wrong.
That's the thing about vibe-coding: it is not the core, it is the entire app. We no longer make MVPs and release those, with AI we can make directly the app including all bells and whistles, entire progression, not just one level, all the systems around it.
Why? Because if something needs changing, it's just one prompt away. I do think code is fluid now, any choice of architecture can be instantly changed at basically no cost.
Maybe my mind is just finding ways to cope, thinking that I "wasted" thousands of solving coding challenges and fixing bugs, but I do think, for better or worse, that manually coding is gone. Same as we no longer code in assembly anymore. We no longer write C. We no longer write JavaScript. We no longer write TypeScript. Maybe not today, people don't like change, but manually writing or even viewing code will only be done in a few educational and high-performance/risk cases.
An LLM isn’t a compiler because there’s no formal method for translating prompts to code. It’s chaotic and small changes in prompts will result in vastly different code.
Come back to me when your app has users and adding new features subtly (or not so subtly) breaks every work flow that you haven’t explicitly tested.
You can’t commit the prompt and regenerate the source code each time because the whole reason that an LLM is useful is that it makes thousands of decisions for you. And those decisions are different each time you regenerate.
The only way to enforce that those decisions are the same each time you regenerate is to encode all of them in tests. But at even moderate complexity that leads to an overconstraint problem that will halt development.
We see this when using LLMs on large apps. Anthropic gave up on their C compiler. Even with an unlimited budget they stopped being able to make progress on it.
I see this in some games I made for my 4 year old. I had them one shot some “juice” when he gets an addition problem right. Combination of screen shake, sounds, flashing light, explosions etc…
It looks pretty cool, but when I tried to tweak the animations with prompts it was always worse. I eventually went in and edited the code myself and I could see why it was so hard for the LLM to change anything because it was a horrific mess of interwoven animations.
I was able to pull everything apart and manually adjust what I wanted.
What helps is to ask the model to make design docs and to note down features and how they work in md files. It sort of provides another layer of persistence separate from code.
But yeah, overall you have to be ok with the app being approximate too. Maybe after an update a button is a different size, or in a different place, or it suddenly has an animation to it. Those smaller things are a bit harder to control when making changes at scale, and if not clearly documented.
For me this is not necessarily a big drawback, for things like games, the core game loop, performance and game feel are a lot more important than any small UI tweaks.
Hopefully, the better the models get, those side-effects will only be improvements, not degradations.
> But yeah, overall you have to be ok with the app being approximate too. Maybe after an update a button is a different size, or in a different place, or it suddenly has an animation to it. Those smaller things are a bit harder to control when making changes at scale,
Now come back to me when you have paying users.
Better yet come back when you have paying users who depend on your app to do their job. And in addition to buttons changing location, you are constantly breaking their work flows because they are using the app in ways you didn’t anticipate.
I only started using AI for development for this product (13 years developed without AI) a few months ago, and customers are really happy with the changes.
I managed to implement feature requests that were pending for years. It took probably 1 month to implement what would have taken 1 year without AI.
I’m talking about the vibe coded app you showed off earlier. If you’ve got another 15 year old app that is well architected and modular, you could probably get away with purely vibe coded improvements for a while without everything devolving into a tightly couple mess where the AI can’t touch X without changing Y.
>It took probably 1 month to implement what would have taken 1 year without AI.
1. I’m not saying that AI can’t speed you up. I’m only saying that you can’t ignore the code for anything beyond a toy app (at least not sustainably).
2. How much is that is down to motivation though? You’ve been developing something for 15 then years then suddenly there’s a brand new development methodology that is fun to mess around with and addictive.
> I’m only saying that you can’t ignore the code for anything beyond a toy app
I thought the same, but I do disagree with this now. Future software development will be mostly just creating black boxes and describing what the box should do, without ever caring what is inside the box. For now, we still have to guide the AI and tell what architecture it should likely use, or which libraries should use (just for the sake of performance and ease of development), but in the future this probably won't matter either. I honestly believe code does not matter anymore. What matters is knowing what to test when building an app, how to define performance metrics and knowing what is good/possible when developing an app. I know I won't be able to create a shooting game that supports 1 million CCU on a single vCPU. But I know that the input latency should feel good, and game should run smoothly at start and over time, to tell it to implement tests to check for memory leaks and avoiding JavaScript GC pauses, use object pooling when possible, etc. The complexity moves from telling how/what to code, to knowing exactly to tell it how things should behave and what's a good outcome. If I tell it "make sure bullets are object pooled", it will likely implement it properly, as object pooling is a very common pattern that exists in its training data, and it usually either works or doesn't, when it doesn't work there are obvious issues like objects shown at the wrong positions or not spawning properly, so the issues with the code would be reflected in e2e testing anyway.
2. That was both motivating and demotivating to be honest. The app was my "baby", having spent a lot of time designing everything, optimizing, carefully choosing libraries and make cool implementation decisions. Now I feel like all the newly added features are not really mine, and it doesn't even feel like my product anymore that I can proudly say: "hey, I wrote all the code for this app". It is a really demoralizing feeling, but at the same time, I like how all my ideas can now be materialized. And it actually works. And it works well. I am still getting used to the fact that I won't have full control or understanding of how the code works, and it pains me that this is the case, but there is no way I could achieve better results by manually coding. I would rather have a feature having 90% of the ideal possible performance and UX, than not having that feature at all. Plus, that 10% is still doable, it just requires a bit of testing and asking the AI. The problem is that most of the times that effort is not really worth it, not for me, not for the clients. There are a lot of other low-hanging fruits that must be addressed, and that's how software development always worked. Now I am happy that I can actually do nice things that before I would have never spent the time on, like making sure a specific settings menu has better UX on mobile (before I would have probably just made an element smaller to fit mobile, for ok usability, but now I can tell it to design an entire new UI tailored to mobile for that specific feature).
It is definitely addictive, as it comes with instant gratification, as opposed to slowly coding and spending hours before you see any results.
I agree with your linked post, and that is sort-of a big issue (even though, most of the times, those guards are not there for the functionality itself, but for the AI, so that in case things break, it gets a more clear error of what went wrong and it knows how to fix it better). This is my entire system prompt, which I assume fixes some of that defensiveness issue:
# Engineering style
- Prefer simple, readable data flows and strong invariants over layers of defensive checks, fallback branches, assertions, and recovery mechanisms.
- Validate at real trust boundaries, then let well-typed internal code rely on those validated contracts. Fix the source of invalid state instead of spreading null checks and guards through consumers.
- Keep code concise and add useful comments that explain intent, ownership, or non-obvious constraints. Do not add speculative protection, tests, or abstractions without a demonstrated failure mode or requirement.
- Use ASD-STE100 Simplified Technical English when you talk to me or caveman-like concise speech, be really direct
That just doesn’t work with current AI beyond a very small scale. I know. I’m on the team that spends our time fixing the vibe coded messes caused by people who think code doesn’t matter anymore.
I have a practically unlimited AI budget and I use it all day long. We have entire teams of very smart people trying to make it possible for PMs to turn jira tickets into features.
We ain’t there yet.
My strong suspicion is that we won’t get there without something close to AGI. And if we get there, there won’t be any white collar jobs left much less software developer jobs.
If AI can take your raw idea and turn it into code with no input from you, it will be able to generate the ideas too.
> I agree with your linked post, and that is sort-of a big issue
This was last week with frontier models. Practically what it means is that I need to audit the code. Approximate changes don’t work when you move past very small scale projects.
Thanks, only tested in webkit browsers, didn't know people use Firefox on mobile either.
EDIT: I can reproduce the crashes, thanks, will tell the AI to test on Firefox too, testing on Firefox mobile specifically might be trickier to debug actually.
I don't know what it built for you in 5 minutes, probably something that "works".
I have spent two weeks using opus just to write a plan/design for 2FA and iron it out until review (about 7 of them) doesn't flag it with 20+ problems (with security holes of various sizes), for which I had to guide it through to not turn it into a mess and whac-a-mole.
Aside from the fact that the implementation must be secure, you want for example to:
- handle accounts that have lost their second factor in an way appropriate for your business
- decide what to do with accounts who don't configure it. If e.g. you want to send them authentication codes via email or SMS that's another can of worms.
Let alone the simple things such as making sure that your implementation works with the various TOTP apps
Well yeah, and it wasn't, most of the pain was around rate limiting, lockouts and preventing exploits and making it work with this specific codebase.
I agree that with a clean codebase it will be simpler, maybe couple of days, just running code review workflow takes 15-30 minutes and then decisions llm makes for each problem is often not good and lead it into overengineering rabbit hole, which means I have to think about each problem and prevent it from escalating.
But yeah if "look, it sends the code and I can enter it to login" is enough validation then it can be made in 5 minutes, sure.
Any of the latest models can one shot a modern 2FA workflow given the right context in a sane codebase.
The critical part is providing it a way to validate its work end to end. Without that, it's similar to asking a human to implement a feature with pen and paper.
> The critical part is providing it a way to validate its work end to end.
Yes! Part of implementing it "one-shot" is actually a loop of planning, implementing, testing e2e on various devices and various edge-cases. It's not just writing the code, and this is not how "vibe-coding" works nowadays. It's not just writing code anymore, this is why most providers worked on their computer-use support too, not only for doing tasks, but also for being able to check and test the work they do e2e.
If you're not doing anything that matters, 2FA is easy. If you're doing something people actually care about, you're going to have to answer real world questions like "how do we provide support for this", "do we force certain roles to set up 2fa", "what if someone loses their phone", etc
I just asked the model to implement 2FA and it did address all the points you've mentioned.
EDIT: Support was not the case, because the app is self-hosted, but the docs should indeed provide instructions for recovery if both authenticator and recovery codes are lost.
If the problem and implementation is so well defined, and "determnistic", this means that LLMs should also be able to just "write code" from specs, without thinking.
Mimo scores higher, even though it got more questions wrong, because the coding tests are scored higher than other tests, and Grok failed some coding tests, which is surprising tbh, as they aren't that hard.
I tried them, but could only test Pro none and Flash none and low, the other ones (medium/high) used way too many tokens and all requests timed out. Not sure if they have a problem with their API, or this model is really token inefficient/basically unusable.
The full-response APIs of many providers have been inadequate for a while now because their timeout intervals do not account for lots of thinking. You can use the streaming API to avoid timeouts.
Now I can just say "add 2FA" and in 5 minutes, while I test something else, it is done.
It also made iterations a lot faster, you can try something out, see how it feels, if it doesn't work, you can just trash all the code and start again.
reply