Machine learning, one of the cornerstones of artificial intelligence, is increasingly shaping technologies that interact closely with human lives — from autonomous vehicles and advanced robotics to medical diagnostic devices. As these innovations promise unprecedented levels of efficiency and automation, they also introduce complex safety challenges that demand serious attention.

Engineers envision a future where machine learning (ML) enhances decision-making and responsiveness in critical systems. Yet the same characteristics that make ML powerful — its capacity to learn from data and adapt — also make it unpredictable. For instance, a self-driving car could misinterpret visual data and strike a pedestrian, or an industrial robot might confuse a worker with a movable object. These errors, though statistically rare, can have catastrophic outcomes.

According to George Pappas, Associate Dean of Research at the University of Pennsylvania’s School of Engineering and Applied Science, machine learning currently falls far short of the reliability required for safety-critical systems. “There’s a fundamental gap between what machine learning can achieve and what’s expected in environments where human safety is at stake,” he explained. “While a 95% or 97% success rate may be considered excellent in AI applications, safety-critical systems demand error rates closer to one in a billion.”

Pappas chaired a committee for the National Academies that recently released a report focused on improving the safety of machine learning-based systems. The findings highlight the need to both enhance the performance of ML components and reinforce the broader systems in which they operate.

Building safer systems through better data and design

Machine learning models derive their intelligence from large datasets — identifying patterns and applying them to future decisions. But when those datasets fail to capture the full range of real-world conditions, systems can behave unpredictably. Thomas Dietterich, Distinguished Professor Emeritus at Oregon State University, emphasized that one major issue lies in how training data are collected.

“Many datasets are gathered incidentally, as by-products of routine business activities, rather than through careful design that reflects the system’s real operating environment,” Dietterich noted. To make systems safer, data collection must intentionally cover the diversity of conditions the technology will face — from weather variations to unusual lighting and movement scenarios.

Another persistent challenge is novelty. When an ML system encounters something it has never seen before — an unfamiliar animal on a road, for instance — it can fail catastrophically. “Machine learning systems tend to break down in the presence of novelty,” Dietterich warned. “We need mechanisms to recognize when that happens, gather new data, and retrain the models so they adapt safely.”

The National Academies report stresses that even the most sophisticated ML models carry inherent uncertainty. Safety-critical designs must account for this. In a self-driving car, for example, an ML component should be able to signal when its confidence level drops — prompting the vehicle to slow down, scan its environment, and collect additional information before proceeding.

Introducing redundancy and safeguards

Machine learning-based systems will inevitably make mistakes, such as misjudging distances or overlooking obstacles. Dietterich argued that robust system architecture — complete with redundancy and fallback mechanisms — is essential to minimize the consequences of those errors. “We need to rethink how we design safety-critical systems so that when a machine learning component fails, there are backup strategies ready to take over,” he said.

Pappas agreed, stressing that the deployment of ML in safety-critical settings must always include “safety filters and guardrails.” If a vehicle or robot misclassifies a human being, for instance, secondary control systems should immediately intervene to prevent an accident. “That’s our most urgent challenge,” he said.

Strengthening standards, transparency, and public trust

The report also calls for the development of updated safety standards, certification processes, and regulations to govern machine learning applications in sensitive domains. Jonathan How, the Ford Professor of Engineering at the Massachusetts Institute of Technology, emphasized that while progress has been made, existing measures are still insufficient to ensure the level of safety the public expects.

How underscored the need for transparency — both in technical evaluations and in public reporting of safety incidents, including near misses. “Transparency is crucial to maintaining trust,” he said. “We must treat each incident as an opportunity to learn and improve.”

Bridging disciplines for safer innovation

The path to safer machine learning systems lies in bridging the cultural and methodological divide between two communities: the safety-critical engineering field and the AI research community. Dietterich pointed out that these groups often prioritize different outcomes. “In machine learning, researchers focus on average-case accuracy, whereas safety-critical engineering is all about worst-case scenarios — the rare moments when failure is not an option,” he said.

Pappas added that the two fields also diverge in their design philosophies. Safety-critical engineers deliberately restrict system scope to maintain control and predictability, while machine learning scientists strive for broad generalization across multiple environments. Reconciling these perspectives, he suggested, will require a new interdisciplinary approach — “a scientific community that understands both sides of the equation and can design systems that are powerful yet inherently safe.”

Preparing the next generation of engineers

Education will play a key role in this transformation. How emphasized the need for specialized training programs to prepare engineers capable of designing and managing machine learning-driven safety systems. “We must educate a new generation of researchers and practitioners who understand both AI and safety engineering principles,” he said.

This commitment should not stop at universities. How urged industry leaders to invest in continuous professional development for their current workforce, ensuring that engineers stay informed about evolving safety regulations and learn how to embed risk awareness into their design processes. “The integration of machine learning into critical infrastructure isn’t just a technical challenge — it’s an educational one,” he concluded.

As machine learning becomes more deeply embedded in everyday life, the question is no longer whether it can outperform humans, but whether it can do so safely and reliably. Achieving that balance will require collaboration, transparency, and a shared dedication to protecting human lives while advancing the frontiers of technology.