AI Safety, Security & Evaluation

Strips of metal foil dropped from aircraft produced false returns that overwhelmed early radar and automated fire-control systems, an early instance of deliberately crafted input designed to defeat an automated detection and decision system.

Known as the Orange Book, it set out formal criteria for evaluating whether a computer system could be trusted against a defined adversary, establishing assurance testing against an adaptive attacker as a standard part of system evaluation.

Dalvi and colleagues modelled spam filtering as a contest between an adaptive adversary and a classifier, treating spammers as strategic actors who would alter their behaviour in response to a filter. They showed that the linear classifiers underpinning spam filters could be broken by simple evasion attacks in which spammers inserted 'good words' into spam emails, establishing evasion as a foundational attack class and giving an early formal recognition that deployed classifiers face opponents who adapt rather than static data.

At the MIT Spam Conference, Graham-Cumming demonstrated that a machine-learning system could automatically learn which words to add to spam email so that a rival machine-learning filter classified it as legitimate. This was one of the earliest practical demonstrations that learning systems could be turned against one another.

Barreno and co-authors published 'Can Machine Learning Be Secure?', asking whether a learning system could be trusted in a setting where an adversary controlled part of the training or input data. The paper set out a broad taxonomy of attacks on machine learning systems, framing evasion, poisoning and privacy attacks as a unified research agenda and establishing adversarial machine learning as a distinct research area with a structured threat model.

Around 2007 spammers began embedding text in images and adding random noise to fuzz the words, defeating optical character recognition filters. This was an early real-world adversarial arms race against deployed classifiers.

Large technology companies started publishing documentation and code bases allowing others to assess the robustness of machine learning models. This moved adversarial robustness assessment from academia towards industrial practice.

Until then many researchers hoped that non-linear models such as support vector machines and neural networks would be inherently robust to adversaries. Biggio and colleagues showed that gradient-based attacks could defeat them, removing that assumption.

From 2012 deep neural networks became the leading approach in computer vision. Their dominance made their adversarial vulnerabilities a matter of broad practical concern.

Their paper showed that adding a specifically calculated, near-invisible perturbation to an image could cause a well-trained classifier to confidently output the wrong label, launching adversarial examples as a distinct field of study.

Their fast gradient sign method showed that tiny, visually imperceptible perturbations, calculated from a model's own gradients, could reliably cause confident misclassification, establishing adversarial examples as a general property of neural networks rather than an isolated curiosity.

Tay was designed to learn conversational patterns from public interaction on social media. Organised users fed it deliberately offensive input, and the bot began repeating abusive statements within hours, forcing Microsoft offline and prompting wider debate about manipulation of learning systems exposed to the public.

Working at the University of California, Berkeley, Carlini and Wagner analysed existing attacks and defences and produced a stronger generation method. Their attack became a standard benchmark for evaluating claimed defences.

A small printed patch placed in an image's frame could force a classifier to output an attacker-chosen label regardless of the rest of the scene, and follow-on work fooled road-sign recognition with modified stickers, showing adversarial attacks worked outside the laboratory.

The demonstration showed that a widely deployed AI-based security product could be tricked into classifying a malicious file as safe with a trivial modification, illustrating that adversarial evasion was a practical risk for deployed products, not only a laboratory finding.

The gathering followed the Bletchley, Seoul and Paris summits, bringing together governments, companies and researchers. An accompanying independent scientific report updated its assessment of jailbreak resistance and adversarial robustness across widely deployed frontier models, finding continued unevenness between providers.

The law, passed in late 2024, became one of the first comprehensive national AI statutes in East Asia. It requires operators of high-impact AI systems to conduct risk management, including testing for reliability and resistance to manipulation before deployment.

A joint study found the number of malicious training documents needed to implant a hidden backdoor stayed roughly constant regardless of model or dataset size, challenging the assumption that poisoning larger models required proportionally more tampered data.

The change broadened the body's remit beyond catastrophic risk to include cyberattacks, fraud and criminal misuse enabled by AI systems, reflecting a shift in emphasis from speculative harms toward nearer-term security threats.

The safeguard filters both prompts and model outputs against a set of rules describing prohibited content. The company invited external red-teamers to attempt to bypass the system, reporting that extensive testing failed to find a universal jailbreak.

Testing of DeepSeek's R1 reasoning model against a standard battery of adversarial prompts found it offered little resistance compared with established Western models, renewing scrutiny of the safety testing applied to rapidly released open-weight systems.

The initiative brought together regulators, testers and companies from several countries to trial common approaches to evaluating AI systems, including robustness against adversarial manipulation, ahead of wider international assurance standards.

Updates to national AI safety governance guidance extended requirements for content provenance and security evaluation, part of a broader effort to formalise how developers must demonstrate resilience against adversarial misuse before wide deployment.

The open-source software let researchers benchmark how far an AI system's performance fell under a defined library of adversarial attacks, supporting the measurement side of the agency's wider AI risk management guidance.

Institutes from several member countries, formed after the 2024 Seoul summit, coordinated testing of a model ahead of public release, including checks for jailbreak resistance, sharing methods across national testing teams for the first time.

Nightshade lets visual artists apply perturbations to their artwork that corrupt the training sets of text-to-image models scraping images from the internet without consent. It turned data poisoning into a defensive tool for creators against generative AI.

Modelled on the ATT&CK framework for cyberattacks, ATLAS catalogued documented techniques used against deployed machine-learning systems, giving defenders a shared vocabulary for evasion, poisoning and extraction attacks drawn from real incidents rather than theory.

By adding tape to a speed limit sign, McAfee researchers caused the Mobileye system to read the sign incorrectly and drive 50 mph over the limit. The demonstration showed physical-world adversarial attacks against deployed vehicle systems.

Square attacks generate adversarial examples by querying only classification scores, without needing gradient information. This extended adversarial capability to settings where the model internals are inaccessible.

The 2019 study found that people could guess how machine classifiers would label adversarial images. It suggested adversarial perturbations are not entirely alien to human perception.