AI is taking control of our development, security, and data!
And it is taking programmers’ jobs.
AI has not taken anything from us. It cannot. It does not have that power. But even if it did, it would not have to. We handed over control of development, security, and data ourselves. Because it is convenient, fast, and often works surprisingly well.
The problem is not when the code does not work. We notice that immediately, and we can fix it. The problem is when the code somehow works, but behind the scenes it does something we would not like it to do. In security, the fact that something somehow works is not enough.
We should know how it works, and when something changes, what changed and where.
A new actor
We often talk about AI as a tool. Similar to a text editor, a search engine, or “autocomplete”.
But AI is not just an ordinary tool.
AI:
- reads the assignment
- interprets the intent
- suggests architecture
- writes code
- modifies configuration
- recommends libraries
- explains errors
- and often even runs commands
So in the security model, this is no longer a passive hammer. It is another actor taking part in decisions – the threat modelA simple plan that says what you protect, from whom, and how an attack could happen. More has expanded.
When I hold a hammer in my hand as a tool, I decide how hard to hit, what to hit, and whether to hit anything at all.
AI is a hammer that makes all those decisions itself, based only on our instructions (prompts). And we have to remember that we are still the ones carrying the responsibility.
I can give the hammer a task: "Hammer in the nails!".
The AI hammer “sees” a nail. It evaluates: "Hm, a nail, I will hammer it in. And it is a big one, so I will use more force.".
And then you watch the AI hammer smashing into a hook in the wall. A hook in a plasterboard wall. A hook with your car keys hanging on it.
AI made a mistake. We have a problem.
We delegated not only development to AI, but also control. And quite often, only the illusion of control. In chat, AI tells us that everything is secure. But is it really?
When AI generates code that allows attackers to get hold of our users’ data, the reputational problem is ours, not the AI agent’s. It is hard to respond to a data leak by pointing at an AI agentAn AI system that does not only answer, but can use tools and perform steps. More and saying, “It was his fault!”
Our now very possibly former customer usually does not care how the application was created. You offer it, you may even charge money for it, so it is your responsibility to make sure user data stays where it belongs.
The journey is no longer the destination
Vibe codingBuilding an application mainly with AI tools and quick prompts, often without deep review of the res... More has made it possible to create applications even for people who have never programmed before. The problem is not that someone does not know the exact syntax of a particular programming language. That is only a small part of programming.
Other important parts of development include algorithmic thinking, connecting individual components, handling memory, working with data, and more. Before vibe codingBuilding an application mainly with AI tools and quick prompts, often without deep review of the res... More tools arrived, people often “programmed” by copying pieces of code from sites like StackOverflow. It was not ideal, but the author at least had some idea how, for example, application loginThe information used to log in to a service, usually username, email, password, code, or security ke... More worked, because they knew they had to create (or copy) code for running a database, a loginThe information used to log in to a service, usually username, email, password, code, or security ke... More form on the web, and also server-side code that compared the data from the form, or stored it in the database. So the author of the application may not have remembered the exact command for storing a passwordIn general, a password is an arbitrary string of characters including letters, digits, or other symb... More in the database, but at least they knew what path the passwordIn general, a password is an arbitrary string of characters including letters, digits, or other symb... More took into the database, and whether it was stored safely. Today? We enter: “Create loginThe information used to log in to a service, usually username, email, password, code, or security ke... More for my application.” Enter. And moments later, we have loginThe information used to log in to a service, usually username, email, password, code, or security ke... More.
New feature – new door
So we have loginThe information used to log in to a service, usually username, email, password, code, or security ke... More. But is the passwordIn general, a password is an arbitrary string of characters including letters, digits, or other symb... More stored securely in the database, or is it readable in plain text? What happens if I put something unexpected into the loginThe information used to log in to a service, usually username, email, password, code, or security ke... More form, for example a piece of code? Will I be able to read all the data from your database? Or change some parameter and log in as another user without knowing their passwordIn general, a password is an arbitrary string of characters including letters, digits, or other symb... More?
Maybe we pay extra attention to loginThe information used to log in to a service, usually username, email, password, code, or security ke... More, because we somehow sense that handling credentialsThe information used to log in to a service, usually username, email, password, code, or security ke... More is a critical part of the application.
But what about other features that look harmless at first glance?
“Add file upload.” Sounds like a normal feature. But suddenly you have to deal with:
- Who can see the file?
- Where is it stored?
- Can it be downloaded without logging in?
- Can someone upload something unexpected?
- Does the file remain there after the account is deleted?
Upload is not just “save file”. It is a new warehouse for other people’s data.
“Add link sharing.”
Sounds like a simple, convenient feature.
- Who can open the link?
- Can the link be guessed?
- Can it be revoked later?
- Does it show more data than it should?
- Can some crawler index it somewhere?
A sharing link is basically a second loginThe information used to log in to a service, usually username, email, password, code, or security ke... More. Just without a passwordIn general, a password is an arbitrary string of characters including letters, digits, or other symb... More.
Another one?
“Add export to PDF.”
- Sounds harmless, because the user is only downloading their own data.
- What exactly ends up inside the PDF?
- Are there metadataData about communication or a file, not necessarily the content itself. For example time, sender, re... More?
- Is the PDF temporarily stored somewhere?
- Can someone download another person’s export?
- Do exports remain on the server?
Export often creates a second copy of the data. And that second copy may no longer have the same protection as the original.
You could probably come up with examples like this for most features in your application. And are you sure your AI is thinking about them?
Or will it simply build the new feature quickly and easily, just to make you happy?
AI does not invent new bugs. It produces old ones faster.
With vibe codingBuilding an application mainly with AI tools and quick prompts, often without deep review of the res... More, we are usually not dealing with completely new types of bugs. These are things we have known for a long time: a user sees someone else’s data, a file is accessible without authenticationIdentity verification. A service checks whether you really are who you claim to be. More, sensitive data ends up readable in the database, the application forgets to check who has access to what, or a library nobody understands gets added to the project.
What is new is mainly the speed. In the past, a person would write such a bug manually in one part of the application. Today, AI can add loginThe information used to log in to a service, usually username, email, password, code, or security ke... More, upload, export, sharing, and an API within minutes – and the same type of bug can appear in several places at once.
AI is not a new kind of bug. AI is an accelerator for old bugs.
An almost real-life fuck-up
I am not a programmer. I have programmed a few small things in my life, but I have never done it seriously over the long term. And it showed. My creations “somehow” worked, but mostly for my own personal use. I probably could not charge money for them.
With the arrival of AI toolsThe ability of an AI system to use external tools, services or functions. More, I decided to “rewrite” one of my older projects into something generally usable. And because that meant handling user data, I wanted it to be designed so that not even I, as the server operator, would be able to read users’ data.
So one of the basic requirements was not just “put some encryptionTurning data into a form that cannot be understood without the right key. More in there somehow”. It was a specific requirement for end-to-end encryptionEncryption where only the sender’s and recipient’s devices read the message, not the server in b... More. Because it is not only that I am not interested in users’ data. I do not want to have it in readable form even in case the server is breached, which is a risk carried by everyone who handles user data, whether it is a small one-person project or a large international corporation. It can happen. And it does happen.
Development went quite well. AI was coding like crazy, and everything more or less worked as expected.
Later, I added export to an internet calendar. A practical feature, a fairly normal requirement. AI implemented the feature. It worked.
After some time, I wanted to know whether everything in the database was really encrypted. The application data was encrypted. But next to that data, there was a table with data for the internet calendar export. In readable form. The same data that was “next to it” safely encrypted.
Calendar export cannot really be end-to-endEncryption where only the sender’s and recipient’s devices read the message, not the server in b... More encrypted. My requirement, because of my own lack of knowledge, went against one of the basic rules.
AI implemented the feature and completed the task. But it somehow forgot to tell me that by doing so, it had violated one of the basic rules.
It is like putting a high-quality lock on the door, and then, for convenience, building a second door next to it without a lock. The first lock still exists. It just no longer protects the entire path.
The encryptionTurning data into a form that cannot be understood without the right key. More was not broken. It was bypassed. Not by a hacker. By the developer.
My advantage is that I deal with security. So not only do I know how to check it, but more importantly, I know what I should check. And that I should check it at all. So I was able to deal with it and disabled automatic calendar export.
But if I had not done that, I would be promising my customers something that was not true. And sooner or later, someone would probably notice. And I would really rather avoid that 🙂
Don’t vibe. Verify.
I do not want to say we should not vibe code. I do it too. AI is great for quick prototypes, refactoring, looking for solutions, generating parts of code, or explaining problems. The problem is not vibe codingBuilding an application mainly with AI tools and quick prompts, often without deep review of the res... More. The problem is when only the vibe remains – when I accept the result because it looks good, but I do not verify what has changed.
In crypto, we know the phrase “Don’t trust, verify.” It does not mean “never trust anyone, never use anything.” It means: trust is not control.
And exactly the same applies to AI. Don’t vibe. Verify does not mean “do not use AI”. It means: do not accept the result just because it feels right. Verify the promises, the data, the permissions, and the new doors you have just created.
Everyone knows that debugging is twice as hard as writing a program in the first place. So if you’re as clever as you can be when you write it, how will you ever debug it?
* Brian Kernighan, The Elements of Programming Style, 2nd edition, chapter 2
