Every system we build spends a full phase trying to break itself. We call it hardening, and it's the difference between software that works in a demo and a system that stays working when real operations hit it.
By the time hardening starts, the build is functionally done. Screens work, data flows, integrations talk. We've been shipping working slices every week so there are no surprises about what the system does. Then we stop being polite. We throw the kinds of days at it that production throws at real systems — because the only alternative is doing that for the first time with your customers watching.
Why hardening is a separate phase
Most software projects treat testing as something that happens during build: unit tests, integration tests, QA sign-off. That's necessary but not sufficient. It answers the question "does it work?" — it doesn't answer "does it stay working when something unexpected happens?"
Hardening answers the second question. It's adversarial by design. We're not looking for bugs in normal operation; we're looking for the conditions under which the system fails, and we're finding them before your customers do. This is a core part of ourfour-phase engagement model: Design, Build, Harden, Operate. Nothing we build skips it.
Three kinds of breaking
Security
We review the system the way an attacker would. That means:
- Input boundary review. Every surface that accepts external data — API endpoints, file uploads, webhook receivers — is tested for injection, boundary overflow, and malformed payloads. We don't trust that the client sends what the spec says.
- Secrets audit. Leaked credentials are the most common root cause of breaches. We scan the codebase, the build artefacts, and the environment configuration for any secret that's ended up somewhere it shouldn't be.
- Privilege escalation paths. In systems with multiple user roles — which is most of the core platforms we build — we map every permission boundary and test whether it can be crossed.
- Dependency review. Third-party libraries are a known attack surface. We audit the dependency tree for known CVEs and pin versions before anything goes to production.
Load
We push traffic past the forecast and watch what degrades first. In practice, the failure mode is rarely "the server falls over". It's subtler:
- The slowest database query becomes visible at 3× normal volume, and suddenly a page that renders in 200ms takes 4 seconds.
- A background job that runs fine in isolation backs up when it shares resources with normal request handling, and the backlog grows faster than it drains.
- A message queue that handles 1,000 events/hour starts dropping events at 8,000 because the consumer wasn't sized for the burst.
For our payments reconciliation system, we load-test to 3× peak forecast before every major release — not because we expect triple volume, but because the system that handles 3× gracefully is the system with headroom for the surprises production always has.
Failure
We kill services mid-flight, cut the network, restart the database. The question isn't "will things break?" — they will. The question is: when they break, what happens?
- Does the system recover automatically, or does it need a human to restart something?
- Does it lose data during the outage, or does it queue and replay when the dependency comes back?
- Does it degrade gracefully — staying partially functional while one component is down — or does one failing service take the whole system with it?
- Does it alert the right people, or does it fail silently until a customer notices?
This failure testing is deeply connected to theevent bus architecture we use in most systems. An event-driven system with a dead-letter queue and replay capability fails gracefully by design. A system with direct point-to-point calls fails brittlely. The hardening phase is where we find out whether the design choices made during build actually hold up.
What hardening actually costs
It costs time and it costs the kind of honesty that's uncomfortable on a timeline. A hardening phase that finds nothing is genuinely possible — and a sign that the build phase did its job well. A hardening phase that finds twelve things is also a success, because you found twelve things before your customers did.
The alternative is doing all of this for the first time in production. Every incident you've ever had was a hardening test running late. We prefer to be the ones who find out — in a controlled environment, with time to fix it properly.
Every incident you've ever had was a hardening test running late.
When we say a system is hardened, it doesn't mean nothing will ever fail. It means we know how it fails, it fails small, and it gets itself back up. That's what we commit to when we say a system is ready for production — and it's what makes the operations phase, described inOperations is where systems are actually built, manageable rather than chaotic.