Software safety is product safety, applied to software.
It asks not only how to defend a system, but why the product was capable of failing — and what would prevent that class of failure for everyone who ships software.
In brief
Our society runs on software.
With few exceptions, that software is built from unsafe components, developed through unsafe practices, and delivered with unsafe defaults.
These conditions are the result of market incentives that reward speed and feature delivery, while shifting risk onto users and obscuring the consequences of failure. These market incentives are created and amplified by the fragmented and weak U.S. policy and law governing software products.
We have seen this before. Industries that once produced widespread, preventable harm became safer through measurement, transparency, and accountability. Software is now at that same inflection point.
Security is the property. Safety is the outcome.
Security
A property of the product: how well software resists attack and failure. Manufacturers build it in, or don’t, through their design and engineering choices.
Safety
The outcome for people: whether the people and communities who depend on software are protected from harm when it is attacked or fails.
Software safety is a discipline. It traces harm back through the system: what failed, what choices in design and engineering made that failure possible, and what would prevent it from happening again.
If software insecurity causes this much harm, why hasn’t the market fixed it?
In mature industries, harm creates a feedback signal. Products cause injuries or losses. Evidence accumulates. Costs travel back toward producers. Standards change. Products improve. Software frequently interrupts that process. Five properties weaken the signal between the decisions that create risk and the people who ultimately bear its consequences.
Evidence
Delayed
The harm may arrive years after the defect ships. By the time a vulnerability is exploited, the code, product, team, and business incentives may all have changed.
Diffuse
A single defect can impose costs across thousands of organizations and millions of people. The aggregate harm can be enormous while no individual loss is large enough to force systemic change.
Accountability
Displaced
The organizations that operate vulnerable software, their customers, insurers, governments, and the public absorb much of the cost. The manufacturer that introduced the defect often does not.
Deniable
It is difficult to connect a specific security incident to a specific engineering, product, or business decision made years earlier. Without attribution of harm, accountability is weak.
Correction
Deflected
When software is exploited, attention naturally focuses on the attacker, who is responsible for the attack. But that can obscure a separate question: why was the product so exploitable in the first place?
The harm is real. But the signal is weak. When consequences are delayed, diffuse, displaced, deniable, and deflected, they do not reliably travel back to the decisions that created them. Software safety has not failed because safer software is technically impossible. It has failed because we have not built strong enough institutions, incentives, evidence systems, and accountability mechanisms to force the system to learn from harm.
Mature industries build institutions that learn from failure.
Other safety-critical fields built systems for turning failure into learning: measurement, reporting, independent investigation, testing, standards, and mechanisms for correcting defects. Those feedback loops help prevent one failure from becoming the next.
Learn from failure
Independent investigation, incident reporting, and shared findings help turn accidents and near misses into changes in systems, standards, and practice.
Test and correct
Crash testing, defect reporting, safety standards, and recalls create mechanisms to identify hazards and require manufacturers to address safety-related defects.
Build evidence before and after release
Pre-market testing and review are paired with post-market surveillance, creating an ongoing evidence base for identifying and responding to safety risks.
Software needs stronger feedback loops of its own.
Software today runs the circuit only halfway: incidents are detected and patched, then the evidence stops. Nothing feeds back into the engineering decisions that produced them.
What it should be
- DesignPrevent predictable failure modes.
- BuildEliminate known classes of defect.
- MeasureGenerate meaningful safety evidence.
- LearnTurn incidents into shared knowledge.
- ImproveChange engineering, markets, standards, policy.
The highest-leverage fix is the earliest one.
Responding to incidents is necessary but expensive and late. Every step upstream — toward build, and toward design — removes whole classes of failure rather than individual instances.
A defect in the design is copied into every product. Fixed at the design stage, it’s fixed once. Fixed after release, it has to be fixed in every copy.
Why now?
There is a widespread assumption that the next generation of technology will solve the problems created by the last one. Powerful AI systems can now analyze software, identify weaknesses, and write exploits. Developers can use them to find defects that have sometimes remained dormant for years or even decades. That could help build a safer software ecosystem. But AI is accelerating two things at once, and neither, on its own, makes software safer.
First, AI speeds the discovery of security defects, but not their remediation. Locating the faulty code is hard, and fixing it without introducing new bugs or regressions takes time. That is a growing burden on developers, and on the operators who must deploy each update without disrupting their operations. Playing whack-a-mole at ever-higher speed is not a strategy for safer software or lower operator risk.
Second, AI is accelerating the creation of new code. These systems learned from a vast body of existing software, much of which is not secure. Without significant compensating controls, the code they generate is likely to inherit those weaknesses. Scaling unsafe code is not a strategy either.
The two reinforce each other: more unsafe code means more whack-a-mole.
Technology alone won’t fix this, because the root problem isn’t technical. It lies in incentive structures that put features ahead of customer safety.
That is why the time to act is now. Defects shipped today become the dormant flaws of the next decade. The tools, habits, and expectations that will govern AI-generated code are being set now, while they can still be changed. Left alone, the default is a growing base of unsafe code that outpaces our ability to fix it.
There is another path, one where software is secure by design. Getting there means treating this as an incentive problem, not a technical one, and making that case before today’s defaults harden into tomorrow’s norms. That is the work SSI was built to do. The future of software safety will not be determined by what AI can do. It will be determined by what we choose to require of the technologies and institutions we build around it.
Drawing: U.S. Pub. 2025/0033838