What the International AI Safety Report 2025 has to say about Privacy Risks from General Purpose AI

The International AI Safety Report 2025 is the most comprehensive scientific assessment of general-purpose AI risks published to date. Produced by a panel of 96 AI experts from across the globe and released ahead of the AI Action Summit in Paris in February 2025, the report builds on the Interim Report from May 2024 — which preceded the AI Seoul Summit — and reflects the latest advancements in AI capabilities, newly documented risks, and the current state of mitigation strategies.

The report does not advocate for specific policies. Its stated purpose is to provide scientific clarity to inform international decision-making. But for any organization that handles personal data — whether patient records, financial information, or customer communications — what this report documents about privacy risks carries immediate operational relevance.

Why This Report Matters for Organizations Handling Sensitive Data

Since the Bletchley Park AI Safety Summit in November 2023, AI capabilities have continued to advance rapidly. The report highlights breakthroughs in scientific reasoning, programming, and autonomous decision-making — as well as a significant shift toward AI agents, systems capable of planning, executing tasks, and delegating actions without continuous human direction.

A key theme running through the document is uncertainty. While some experts believe that risks such as AI-enabled cyberattacks and large-scale job displacement remain years away, others warn that these challenges may arrive sooner than the field expects. The report points specifically to OpenAI's early test results from its o3 model, which demonstrated reasoning and problem-solving performance that surpassed many human experts on select tasks — suggesting AI capabilities may be advancing faster than anticipated.

That uncertainty reinforces one practical conclusion: organizations that wait for regulatory mandates to address AI-related privacy risks are already behind. Proactive risk management is no longer optional — it is a competitive and legal imperative. Whether your organization operates in healthcare, financial services, pharmaceuticals and life sciences, or insurance, the risks the report identifies apply directly to how sensitive data flows through your AI systems.

If your organization is deploying or building AI systems that touch personal data, now is the time to act. Speak with Limina's team to understand your current exposure and what automated de-identification can do about it.

What Are the Three Categories of Privacy Risk Identified in the Report?

The report organizes AI-related privacy risks into three distinct categories: risks that arise during training, risks that emerge during active use, and risks created through intentional misuse. Each category presents a different kind of challenge, and each requires a different mitigation strategy.

Training Risks: Personal Data Embedded Inside AI Models

General-purpose AI models are trained on enormous datasets assembled from public web sources, proprietary databases, and user interaction logs. This data often includes personally identifiable information (PII) and other sensitive categories, frequently collected without the knowledge or explicit consent of the individuals involved.

The report identifies several specific concerns within this category. First, AI models can unintentionally memorize and reproduce sensitive details from their training data, including health records, financial information, and private communications. This is not a theoretical vulnerability — researchers have demonstrated that it is possible to extract training data from deployed language models through carefully constructed queries.

Second, many AI models are trained on publicly available data that was never intended for large-scale AI processing. This directly conflicts with a foundational principle of modern privacy law: that individuals should remain in control of how their data is used, with informed consent guiding that use.

Third, once sensitive data has been incorporated into a trained model, removing it is technically difficult. This creates a compliance gap for organizations subject to regulations that include the right to deletion — a right enshrined in the GDPR, Quebec's Law 25, and a growing number of other frameworks.

The report notes that in sensitive fields like healthcare and finance, training on real-world data improves model performance but also increases the risk of privacy leaks. Efforts to reduce these risks — such as Google's use of anonymized data in developing Gemini-Med — exist, but the report cautions that further research is needed to assess their effectiveness.

Use Risks: How AI Handles Sensitive Information in Real Time

The second category focuses on risks that arise while AI systems are actively in use, rather than during the training phase. The report pays particular attention to a technique called Retrieval-Augmented Generation (RAG), which allows AI models to access current and personalized data beyond what was present at training time. This is what enables digital assistants to reference your actual documents, calendar, or records rather than just their baseline knowledge.

While RAG enables more relevant and personalized AI responses, it also creates new privacy exposure. Sensitive information fed into a model to personalize its output can potentially be retained, reflected in subsequent responses, or accessed by parties who should not have visibility into it. These risks are especially pronounced when data moves beyond local devices into cloud-based processing environments.

The report acknowledges that cybersecurity measures exist to address some of these concerns, but concludes that balancing privacy, transparency, and utility remains a central challenge requiring both technical solutions and updated policy frameworks.

Intentional Harm Risks: AI as a Tool for Privacy Violations at Scale

The third category addresses the deliberate exploitation of general-purpose AI to violate privacy. Malicious actors can use AI to enhance cyberattacks, automate surveillance, and generate fraudulent content at a scale that would have been impossible with earlier tools.

Specific examples cited in the report include AI-enhanced cybercrime, where attackers use AI to scan breached datasets, identify high-value targets, and automate phishing campaigns with greater precision and volume. Deepfake and identity fraud capabilities allow synthetic media to be used for impersonation, misinformation, and targeted harassment. Advanced pattern recognition and image analysis tools make it possible to infer sensitive personal information from data points that appear innocuous in isolation, enabling mass privacy violations without directly accessing protected systems.

The report also notes an important asymmetry in how these harms develop: privacy violations from AI training and deployment can remain hidden for an extended period given the significant lag between when data is collected and when it is actually used or exposed. At the same time, the report observes that no high-profile leak of PII or harmful use of confidential commercial information has yet been publicly confirmed, despite mandatory breach reporting requirements in many jurisdictions. The absence of a documented incident does not mean the risk is contained — it may reflect the time delay before these harms surface.

What Technical Solutions Does the Report Recommend?

The report is careful to note that legal and policy frameworks alone cannot address AI privacy risks. Technical solutions are equally essential, and must be applied across the full AI lifecycle — from data collection and model training through deployment and ongoing use.

Data Minimization and Privacy-Preserving Training

The most direct way to prevent AI models from leaking personal information is to ensure that personal information is removed before it enters the training pipeline. The report identifies this as technically feasible but notes the absence of clear standards as a significant barrier. There is no universally agreed-upon benchmark for what constitutes sufficiently de-identified training data, which creates compliance ambiguity for organizations building or procuring AI systems.

Synthetic data generation — creating artificial datasets that mirror the statistical properties of real data without including actual personal records — is identified as a promising approach. However, the report finds that synthetic data either maintains privacy risks when optimized for utility, or requires strong differential privacy techniques that reduce utility, leading back to the same fundamental trade-off.

Differential privacy, which adds mathematically calibrated noise to datasets to prevent models from memorizing individual data points, is also discussed. The challenge is the same: stronger privacy protections tend to reduce model accuracy, and the trade-off is especially pronounced for large, general-purpose text models.

Privacy-Enhancing Deployment Strategies

Beyond training, the report identifies a range of technical approaches for protecting privacy during AI deployment. On-device processing — running AI models locally rather than in the cloud — reduces exposure of sensitive data to third parties, though it limits what models can be practically deployed. For larger models that cannot run locally, secure cloud deployment with end-to-end security controls is recommended.

Confidential computing, which uses hardware-based security mechanisms such as secure enclaves and encrypted computation, ensures that AI operations occur in protected environments where even the infrastructure provider cannot access raw data. Cryptographic approaches including homomorphic encryption and zero-knowledge proofs allow AI systems to process data without revealing its contents, though these remain computationally expensive at scale.

The report also highlights user-controlled data governance — dashboards and interfaces that allow individuals to manage permissions, track how their data is used, and remove consent — as a meaningful complement to technical controls, particularly when paired with data provenance systems.

Cybersecurity Against AI-Enabled Attacks

The report notes that general-purpose AI can also be applied defensively: AI-driven security tools can detect and neutralize threats including phishing campaigns, malware, and data breach attempts. The report also emphasizes liability as a mechanism for improving safety outcomes, suggesting that holding developers and distributors accountable for misuse could disincentivize unsafe deployment practices.

Despite these advances, the report is direct in its assessment: privacy-enhancing technologies and security measures are still maturing, and many AI privacy challenges remain unresolved. The gap between what is technically possible and what is operationally implemented at scale remains wide.

What Does This Mean for Organizations Using AI Today?

The report's practical implications are significant for any organization that is either building AI systems or deploying third-party AI tools that interact with sensitive data. The three risk categories the report identifies are not abstract future concerns — they describe conditions that exist in deployed systems today.

Organizations in regulated industries face the most immediate pressure. Healthcare providers and health data processors must meet HIPAA's de-identification standards before sensitive patient information can be safely used in AI workflows. Healthcare organizations handling clinical notes, discharge summaries, or any form of unstructured patient data are particularly exposed to training and use risks as described in the report. Similarly, pharmaceutical and life sciences organizations conducting AI-assisted research on patient-derived data carry significant obligations under both HIPAA and GDPR.

Financial institutions and insurers face analogous obligations under their respective regulatory frameworks. Contact centers that use AI to process call recordings, chat transcripts, or support tickets are handling PII in exactly the real-time use scenarios the report flags as high-risk.

The common thread across all of these contexts is unstructured data. Personal information does not only live in structured databases where fields are clearly labeled and access can be controlled. It appears in clinical notes, legal documents, support transcripts, research interviews, and internal communications. AI systems that process this content at scale, without a systematic mechanism for detecting and removing PII, create the kind of exposure the report describes.

How Limina Addresses the Privacy Risks Outlined in the Report

Limina's data de-identification platform is designed specifically to address the categories of risk the report identifies — across training, use, and deployment.

For training risks, Limina's technology detects and redacts PII before it enters AI training pipelines. This supports developers building privacy-compliant models and helps organizations meet strict de-identification standards, including HIPAA's Safe Harbor and Expert Determination methods. Limina's platform achieves over 99.5% accuracy across more than 50 entity types and 52 languages, providing the kind of precision that enterprise-grade compliance requires.

For use risks, Limina enables real-time PII detection and removal in AI interactions, ensuring that sensitive information is not inadvertently retained or reflected in model outputs. This applies to RAG pipelines, AI chat interfaces, and any workflow where unstructured data is processed by a language model.

For deployment more broadly, Limina's automated privacy filters work across structured and unstructured data alike, supporting compliance with GDPR, Quebec's health privacy law, PIPEDA, Japan's APPI, and other global frameworks. The platform is built by linguists, which means it understands context — it can distinguish between a reference to a medication dosage and a patient identifier, or between a company name and a personal name, in ways that pattern-matching tools cannot.

Ready to see how Limina addresses the specific risks the AI Safety Report identifies? Request a demo or contact the team to discuss your organization's needs.

‍