Engineering practices that keep AI-built systems from falling apart

I'm a technical partner for businesses that outgrew the internal tools they built with AI.

These are the practices I use on real projects to keep systems built with AI agents working. You don't need all of them, and you don't need them all at once. Which ones matter depends on two things: how many people and agents use your system, and how many people and agents build it. I learned most of these the hard way.

Where is your system? →

A robot agent at a desk reading a wall of notes that describe the system

The quality of a system comes from the practices around it, not from the talent of whoever writes the code. That holds for a team of people and for a team of agents.

There is one real difference. A person builds up experience with your system over time. An agent starts from zero every session, like the main character of the film Memento, who has to read his own notes to know what's going on. So what the system knows about itself has to live in the system: in files the agent reads before it touches anything.

Without that, an agent makes decisions that are sensible locally and make the whole thing more complicated. Every next change costs more and breaks more. The goal is the opposite: each change should leave the system a little simpler than before.

Practices also have a price. A test suite, a review process, a staging environment: each one is worth it at some point and a burden before that. So I pick them by stage, and I don't apply everything to everyone. The list below is organized that way.

Stage 1: Just me and one agent

You're building this yourself with an AI agent, and hardly anyone uses it yet. The main job is to keep the system from becoming a pile of one-off decisions.

  • Your tech stack, written down before any code

    Languages, frameworks, versions and what not to use go in a file the agent reads before every task.

    Without it: The agent picks different libraries in different corners. A month later nobody can say what the system runs on, and every change costs more than the last.

  • One instruction file for the agent

    A single file in the repository: rules, boundaries, where things live, what never to do.

    Without it: Every new chat, the agent starts blank and guesses the rules again. It guesses differently each time.

  • All code in version control

    Every change is recorded, and any earlier version can be brought back.

    Without it: One bad change from the agent and there's nothing to go back to. The whole system lives on one laptop.

  • Small steps, each one checked

    One task, one change, one check. Not “rework everything.”

    Without it: The agent rewrites half the system in one go, and nobody can tell what exactly broke.

  • Automated tests for the scenarios that matter

    A few checks on critical flows, growing with the system. They also tell the agent how things should work.

    Without it: The agent fixes one thing and quietly breaks another. You learn about it from your team or a client.

  • Linters and autoformatters

    A machine catches small mistakes and keeps the code in one consistent style.

    Without it: Small mistakes a tool would catch in a second reach your client. Code looks different in every file, and the agent copies the worst examples.

  • Quality gates after every piece of work

    Before calling a task done, the agent must run linters, formatters and tests, and they must pass.

    Without it: When the agent says “done,” it means “I finished writing,” not “it works.”

  • Design patterns that fit this system

    Recurring problems get one proven solution chosen for your system, not every pattern in the book.

    Without it: Every feature is built its own way. The system turns into a heap of one-off solutions, and each change costs more than the last.

  • A catalog of UI components

    Buttons, forms and tables are defined once, and the agent reuses them.

    Without it: The agent draws a new button and form on every screen. The interface drifts, and a style change becomes a manual pass through every page.

  • Documentation that updates with the code

    A description of the system lives next to the code, and the agent must update it after every change.

    Without it: What the system does lives in the author's head and old chat history. In three months neither you nor the agent know why. Outdated docs are worse than none.

  • A log of decisions and why

    Short notes on what was decided and the reason.

    Without it: The agent “improves” something you did on purpose and brings back a problem you already beat once.

  • Locked library versions, no code run on install

    Exact versions are pinned in the repository. Installing a package can't run the package's own code.

    Without it: One day a library updates itself and the system stops building. Or a package brings code that can reach your keys.

Stage 2: People rely on it

People use it, things break, and you should hear about it before they do.

  • Automated backups of your data

    Copies of your data are made on a schedule, with nobody having to remember.

    Without it: One mistake or a server failure and the company's data is gone, with nothing to restore.

  • Backups you've actually restored from

    Regularly I restore a backup into a separate environment and check the data is there.

    Without it: You find out the backups don't work on the day you need them.

  • Production and test kept apart

    New work gets tried on a copy of the system, not on your clients' live data.

    Without it: Changes get tested on live data, and one mistake corrupts real orders or sends wrong emails to clients.

  • Test data and a fixed set for tests

    One dataset for manual checks, another that never changes for automated tests.

    Without it: Tests pass one day and fail the next because the data moved. Checking turns into “let's look at it by eye.”

  • Settings and keys outside the code (12-factor)

    Configuration and secrets live outside the code, and the system starts the same way anywhere.

    Without it: The system only works on that one server. Moving or restoring it becomes a week-long quest, and keys sit in the code itself.

  • Monitoring and alerts tied to business impact

    The system itself tells you when it's down or when a key process stops working.

    Without it: You hear about a breakage from your team or a client, not from the system.

  • Logs and error tracking

    When something fails, a trail remains: what happened, where and why.

    Without it: “Something broke,” with no answer to what or why. The agent fixes by guesswork, and the problem comes back.

  • A quick way to undo a bad release

    Any update can be rolled back to the previous working state quickly.

    Without it: A bad update means hours of downtime while someone repairs it on the fly.

  • Runbooks for common incidents

    Step-by-step instructions: if X happens, do Y.

    Without it: It breaks on Friday evening, and how to restart it or where to restore from is known by one person, or by a chat that's gone.

  • A post-mortem after every incident

    A short review: what happened, why, and which practice changes as a result.

    Without it: The same breakage comes back every few months.

  • Automatic checks for vulnerable libraries

    Known security holes in the libraries you use are flagged automatically (Dependabot, for example).

    Without it: A library with a known hole sits in your system for years. Everyone knows about it except you.

  • A basic security check of your data

    Periodically I check database permissions, exposed endpoints and leaked keys.

    Without it: Client data is open to anyone who knows the address. You hear about it from a stranger.

  • Periodic check of test coverage

    Once per cycle I look at which critical parts of the system no test covers.

    Without it: Tests exist for show while new critical code sits uncovered. It feels safe, but isn't.

  • Periodic check of code complexity

    Once per cycle I measure cyclomatic complexity to find parts that have grown into a tangle.

    Without it: Some parts turn into a tangle that both people and agents are afraid to touch.

Stage 3: Several people and agents build at once

More than one person or agent is changing the system at the same time. The main job is keeping them out of each other's way.

If the system works and one person is what limits it, the move into this stage from stage 1 or 2 is Speedup. A system that is breaking starts at Rescue.

  • Separate working copies for parallel tasks

    Each parallel task runs in its own working copy (git worktree).

    Without it: Two agents, or a person and an agent, edit the same files in one folder and overwrite each other's work.

  • Branches and human review before merging

    Every change goes in its own branch and a person reads it before it reaches the main code.

    Without it: Nobody looked at the agent's change, and a client finds the mistake.

  • Automatic checks on every change (CI)

    Build and tests run by themselves on each change.

    Without it: “It works on my machine,” and the main code is broken for everyone by morning.

  • Periodic check of technical agreements

    Once per cycle I compare the system to what was agreed: stack, patterns, access rules, no extra dependencies.

    Without it: Everyone reads the rules their own way. The system slowly becomes something nobody designed, and extra dependencies become extra risk.

  • Database changes kept as code

    Every change to the data structure is written in the repository, applied automatically and reversible.

    Without it: Two people change the database by hand, versions drift apart, and there's no way back.

  • Outside services behind adapters

    Each external service is handled in exactly one place in the code.

    Without it: A service changes its API and fixes are needed all over the system. Switching vendors turns into a rewrite.

Stage 4: Groups with different interests

There are many users, and they fall into groups whose needs can conflict. The main job is making sure each group gets what it needs and nothing more.

  • Access rules written down for the agent

    Who can see and do what is described in one place and loaded into every task (an ACL).

    Without it: When adding a feature, the agent guesses who sees what, and sooner or later shows a client someone else's data.

  • Access tests for every user group

    For each group, at minimum anonymous and signed-in users, tests confirm what each part of the system allows.

    Without it: Access is checked by eye, and the mistake is found the day one client opens another client's data.

  • Key users and their jobs to be done

    Who they are, what job they use the system for, and what habits and needs they have.

    Without it: The system grows features for the loudest user and breaks the work of those who stay quiet.

  • A register of features by user group

    A list of features: who needs each one and why.

    Without it: Nobody knows who a feature is for or why it exists. Changing or removing it is scary because you can't tell whose work it breaks.

  • An audit log of who did what

    The system records who changed or viewed what, and when.

    Without it: A dispute over who deleted something can't be settled, and after a leak you can't tell how far it went.

What I don't do

Four things I decline to do, and why.

I don't lock you in to one vendor.
You should be able to walk away from any service, including me. If a provider raises prices or shuts down, you shouldn't have to change everything at once.
I don't optimize early.
Complexity gets added when numbers show a problem, not for a future that may never come.
I don't build heavy infrastructure.
Kubernetes for a blog is a bill for complexity your business never uses. A simple system is one you can understand and fix.
I don't build what you couldn't fix yourself.
Every decision passes one test: could your team, or any other engineer, repair this without me? If not, I choose something else.

What this approach won't do

It isn't instant.
Practices go in step by step, and a system gets steadier over weeks, not days.
It isn't free.
Every practice costs time now to save more later, and I'll tell you when one doesn't pay off.
It won't turn an internal tool into industrial software.
It doesn't replace decisions that belong to the business:
what to build, for whom, and what to stop doing.

Not sure which stage you're at?

The diagnostic finds your stage and the practices you actually need, in five business days.

Diagnostic: $1,900, 5 business days

You get a written report that stays yours. No obligation to continue.

Want to look around first? Try the free self-check.

Prefer email? Write to me directly: me@artkiselev.dev

Start the diagnostic