AI is taking control of our development, security, and data!
And it is taking programmers’ jobs.
AI has not taken anything from us. It cannot. It does not have that power. But even if it did, it would not have to. We handed over control of development, security, and data ourselves. Because it is convenient, fast, and often works surprisingly well.
The problem is not when the code does not work. We notice that immediately, and we can fix it. The problem is when the code somehow works, but behind the scenes it does something we would not like it to do. In security, the fact that something somehow works is not enough.
We should know how it works, and when something changes, what changed and where.
A new actor
We often talk about AI as a tool. Similar to a text editor, a search engine, or “autocomplete”.
But AI is not just an ordinary tool.
AI:
- reads the assignment
- interprets the intent
- suggests architecture
- writes code
- modifies configuration
- recommends libraries
- explains errors
- and often even runs commands
So in the security model, this is no longer a passive hammer. It is another actor taking part in decisions – the threat model has expanded.
When I hold a hammer in my hand as a tool, I decide how hard to hit, what to hit, and whether to hit anything at all.
AI is a hammer that makes all those decisions itself, based only on our instructions (prompts). And we have to remember that we are still the ones carrying the responsibility.
I can give the hammer a task: "Hammer in the nails!".
The AI hammer “sees” a nail. It evaluates: "Hm, a nail, I will hammer it in. And it is a big one, so I will use more force.".
And then you watch the AI hammer smashing into a hook in the wall. A hook in a plasterboard wall. A hook with your car keys hanging on it.
AI made a mistake. We have a problem.
We delegated not only development to AI, but also control. And quite often, only the illusion of control. In chat, AI tells us that everything is secure. But is it really?
When AI generates code that allows attackers to get hold of our users’ data, the reputational problem is ours, not the AI agent’s. It is hard to respond to a data leak by pointing at an AI agent and saying, “It was his fault!”
Our now very possibly former customer usually does not care how the application was created. You offer it, you may even charge money for it, so it is your responsibility to make sure user data stays where it belongs.
The journey is no longer the destination
Vibe coding has made it possible to create applications even for people who have never programmed before. The problem is not that someone does not know the exact syntax of a particular programming language. That is only a small part of programming.
Other important parts of development include algorithmic thinking, connecting individual components, handling memory, working with data, and more. Before vibe coding tools arrived, people often “programmed” by copying pieces of code from sites like StackOverflow. It was not ideal, but the author at least had some idea how, for example, application login worked, because they knew they had to create (or copy) code for running a database, a login form on the web, and also server-side code that compared the data from the form, or stored it in the database. So the author of the application may not have remembered the exact command for storing a password{:cs}Heslo je obecný prostředek k ověření totožnosti (... More in the database, but at least they knew what path the password{:cs}Heslo je obecný prostředek k ověření totožnosti (... More took into the database, and whether it was stored safely. Today? We enter: “Create login for my application.” Enter. And moments later, we have login.
New feature – new door
So we have login. But is the password{:cs}Heslo je obecný prostředek k ověření totožnosti (... More stored securely in the database, or is it readable in plain text? What happens if I put something unexpected into the login form, for example a piece of code? Will I be able to read all the data from your database? Or change some parameter and log in as another user without knowing their password{:cs}Heslo je obecný prostředek k ověření totožnosti (... More?
Maybe we pay extra attention to login, because we somehow sense that handling credentials is a critical part of the application.
But what about other features that look harmless at first glance?
“Add file upload.” Sounds like a normal feature. But suddenly you have to deal with:
- Who can see the file?
- Where is it stored?
- Can it be downloaded without logging in?
- Can someone upload something unexpected?
- Does the file remain there after the account is deleted?
Upload is not just “save file”. It is a new warehouse for other people’s data.
“Add link sharing.”
Sounds like a simple, convenient feature.
- Who can open the link?
- Can the link be guessed?
- Can it be revoked later?
- Does it show more data than it should?
- Can some crawler index it somewhere?
A sharing link is basically a second login. Just without a password{:cs}Heslo je obecný prostředek k ověření totožnosti (... More.
Another one?
“Add export to PDF.”
- Sounds harmless, because the user is only downloading their own data.
- What exactly ends up inside the PDF?
- Are there metadata?
- Is the PDF temporarily stored somewhere?
- Can someone download another person’s export?
- Do exports remain on the server?
Export often creates a second copy of the data. And that second copy may no longer have the same protection as the original.
You could probably come up with examples like this for most features in your application. And are you sure your AI is thinking about them?
Or will it simply build the new feature quickly and easily, just to make you happy?
AI does not invent new bugs. It produces old ones faster.
With vibe coding, we are usually not dealing with completely new types of bugs. These are things we have known for a long time: a user sees someone else’s data, a file is accessible without authentication, sensitive data ends up readable in the database, the application forgets to check who has access to what, or a library nobody understands gets added to the project.
What is new is mainly the speed. In the past, a person would write such a bug manually in one part of the application. Today, AI can add login, upload, export, sharing, and an API within minutes – and the same type of bug can appear in several places at once.
AI is not a new kind of bug. AI is an accelerator for old bugs.
An almost real-life fuck-up
I am not a programmer. I have programmed a few small things in my life, but I have never done it seriously over the long term. And it showed. My creations “somehow” worked, but mostly for my own personal use. I probably could not charge money for them.
With the arrival of AI tools, I decided to “rewrite” one of my older projects into something generally usable. And because that meant handling user data, I wanted it to be designed so that not even I, as the server operator, would be able to read users’ data.
So one of the basic requirements was not just “put some encryption in there somehow”. It was a specific requirement for end-to-end encryption. Because it is not only that I am not interested in users’ data. I do not want to have it in readable form even in case the server is breached, which is a risk carried by everyone who handles user data, whether it is a small one-person project or a large international corporation. It can happen. And it does happen.
Development went quite well. AI was coding like crazy, and everything more or less worked as expected.
Later, I added export to an internet calendar. A practical feature, a fairly normal requirement. AI implemented the feature. It worked.
After some time, I wanted to know whether everything in the database was really encrypted. The application data was encrypted. But next to that data, there was a table with data for the internet calendar export. In readable form. The same data that was “next to it” safely encrypted.
Calendar export cannot really be end-to-end encrypted. My requirement, because of my own lack of knowledge, went against one of the basic rules.
AI implemented the feature and completed the task. But it somehow forgot to tell me that by doing so, it had violated one of the basic rules.
It is like putting a high-quality lock on the door, and then, for convenience, building a second door next to it without a lock. The first lock still exists. It just no longer protects the entire path.
The encryption was not broken. It was bypassed. Not by a hacker. By the developer.
My advantage is that I deal with security. So not only do I know how to check it, but more importantly, I know what I should check. And that I should check it at all. So I was able to deal with it and disabled automatic calendar export.
But if I had not done that, I would be promising my customers something that was not true. And sooner or later, someone would probably notice. And I would really rather avoid that 🙂
Don’t vibe. Verify.
I do not want to say we should not vibe code. I do it too. AI is great for quick prototypes, refactoring, looking for solutions, generating parts of code, or explaining problems. The problem is not vibe coding. The problem is when only the vibe remains – when I accept the result because it looks good, but I do not verify what has changed.
In crypto, we know the phrase “Don’t trust, verify.” It does not mean “never trust anyone, never use anything.” It means: trust is not control.
And exactly the same applies to AI. Don’t vibe. Verify does not mean “do not use AI”. It means: do not accept the result just because it feels right. Verify the promises, the data, the permissions, and the new doors you have just created.
Everyone knows that debugging is twice as hard as writing a program in the first place. So if you’re as clever as you can be when you write it, how will you ever debug it?
* Brian Kernighan, The Elements of Programming Style, 2nd edition, chapter 2
