Safety Norms: Large Language Models

1Purpose, Scope, and Safety Principles

These norms set minimum expectations for safe work with large language models, which are neural networks trained on large collections of text and code to predict and generate language. In this context, safety means preventing harm to people, protecting confidential and personal data, keeping systems secure, and ensuring outputs are reliable enough for the decisions they support. The norms apply to research, model development, fine-tuning, evaluation, red teaming, product integration, internal assistants, and retirement of models.

Four principles guide every rule: least privilege, named human accountability, verify before trust, and controls proportionate to risk. A model output is a draft, not a verified fact or an approved action, unless a qualified reviewer has checked it against a reliable source.

2Roles, Competence, and Accountability

Named owners reduce confusion. The model owner approves intended use, risk tier, and release criteria. The data steward approves sources, licenses, and retention periods. The security lead reviews access, secrets, and integrations. The evaluation lead defines test sets and acceptance thresholds. The incident lead coordinates response. In small teams one person may hold several roles, but the separation of approval from execution must be documented.

Competence is required before access. Staff who handle personal data, production endpoints, or high-impact outputs must complete role-specific training and receive written authorization from the model owner. Authorization should be time-limited and reviewed whenever duties change.

• Mandatory: Every production model has one named owner recorded in the model register.
• Mandatory: No one may grant production or administrative access to themselves.
• Mandatory: Staff without written authorization may not connect to production inference endpoints, weight repositories, or customer data stores.
• Recommended: Contractors receive the same training and access reviews as employees, with contract terms that state data handling duties.

3Course-Specific Hazards and Risk Assessment

Realistic hazards fall into several groups. Privacy hazards include memorized personal data, sensitive information pasted into prompts, and logs that retain user content. Security hazards include prompt injection, where untrusted text in a document or web page instructs the model to ignore its rules, data exfiltration through tool calls, poisoned training or retrieval data, and leaked API keys. Reliability hazards include hallucinated facts, fabricated citations, inconsistent answers, and performance drops after an update. Harm hazards include biased or abusive outputs, unsafe advice in medical, legal, financial, or engineering contexts, and content that facilitates violence, fraud, or self-harm.

Each use case should be scored for likelihood and severity before launch. A simple three-level scale works: low where errors are easily caught and cause minor inconvenience, medium where errors cause financial loss or rework, and high where errors affect health, safety, legal rights, critical infrastructure, or access to essential services. High-risk uses require human review of outputs, documented testing, and sign-off by the model owner and a relevant specialist.

• Mandatory: Document a risk tier for every use case before development or deployment begins.
• Mandatory: Treat any use that influences medical, legal, employment, credit, or safety decisions as high risk.
• Mandatory: Record known limitations, such as unsupported languages or topics, in the risk assessment and in user-facing documentation.
• Recommended: Re-score risk when the user group, data source, or tool permissions change.
• Note: Obligations for automated decision making vary by jurisdiction and must be confirmed with qualified counsel.

4Preventive Controls and Safe Work Practices

Controls work best in layers. Start with design: limit what the model can see and do. Use retrieval only from approved sources, restrict tools to the minimum functions needed, and require confirmation before any action that sends messages, changes records, moves money, or deletes data. Then apply input and output filtering, but never rely on filters alone, because they miss novel attacks and produce false negatives. Filters should be tested with the same rigor as the model.

Testing must happen before release and after every material change. Build an evaluation set that includes routine tasks, edge cases, adversarial prompts, and sensitive topics. Record pass thresholds and rerun the set when the model, prompt template, retrieval index, or tool list changes. Testing against production data is prohibited unless the privacy and security reviews approve it in writing for a defined purpose.

• Mandatory: Do not use real personal, health, financial, or confidential data in training, evaluation, or prompts unless a data protection review has approved the specific purpose and retention period.
• Mandatory: Keep API keys, tokens, and credentials in an approved secrets manager. Never place them in prompts, source code, notebooks, or shared chat channels.
• Mandatory: Run adversarial testing for prompt injection and data leakage on every externally facing system before release.
• Mandatory: Tool calls that write, send, purchase, or delete require a human confirmation step or a documented automated control with tested rollback.
• Mandatory: Disclose AI-generated content where the applicable policy or law requires it, and confirm the labeling rule with the compliance lead.
• Recommended: Pin model versions in production and test a new version or snapshot before upgrading.
• Recommended: Apply rate limits, output length limits, and request timeouts to reduce abuse and runaway costs.
• Recommended: Maintain a list of known failure modes and show users plain warnings about them in the interface.
• Safe practice: Do not paste confidential source code, customer records, or unpublished research into public tools unless the organization has an approved agreement covering that data.

5Data, Access, Privacy, and Security Safeguards

Large language model systems involve many components: training corpora, vector stores, model weights, inference endpoints, logging pipelines, plugins, and user interfaces. Each component needs an owner, an access list, and a retention rule. Model weights and fine-tuning datasets are sensitive assets because they may encode proprietary information or personal data, so they should be stored in access-controlled repositories with audit logging. Development, test, and production environments should use separate accounts or tenants so that a test mistake cannot expose live data.

Privacy controls follow data minimization. Collect only what the task requires, mask identifiers in logs, and set retention periods that match the approved purpose. Users should be told what is stored and how to request deletion where applicable law provides that right. Those obligations vary by jurisdiction, so verify them with the privacy office. Accelerator clusters and on-premises servers require physical access control, and vendor safety and installation instructions must be followed by qualified facilities staff rather than model developers.

• Mandatory: Enforce multifactor authentication and role-based access for model consoles, data stores, and deployment pipelines.
• Mandatory: Review privileged access at an interval set by the security lead, and remove access that is no longer needed the same day the role changes.
• Mandatory: Log prompts, tool calls, and administrative changes, with personal data masked or minimized.
• Recommended: Scan retrieval sources and uploaded documents for hidden instructions before indexing them.
• Recommended: Run model-generated code only in a sandbox with no access to production networks or stored secrets.

6Incident Response, Escalation, and Emergencies

An incident is any event where a model produces harmful, unlawful, or confidential content, leaks data, is manipulated into unauthorized actions, or performs materially below its approved evaluation thresholds. Near misses, such as an output caught by human review before release, should also be recorded because they show where controls work and where they are weak.

Report suspected incidents immediately to the model owner and security contact. Do not delete logs, prompts, or outputs, because they are needed for investigation. Disable the affected feature or revert to a pinned version only when the incident lead authorizes it or the runbook allows it. If there is an immediate threat to health or physical safety, contact local emergency services first and then notify the internal lead. Do not attempt to fix hardware, power, or cooling faults in a data center yourself.

• Mandatory: Report suspected data exposure within the timeframe set in the organization's incident policy and notify the privacy and legal leads at once.
• Mandatory: Maintain a tested runbook for disabling endpoints, rotating keys, and removing indexed content.
• Mandatory: Notify affected users or regulators only as required by law and approved by counsel, since notification duties vary by jurisdiction.
• Recommended: Conduct a blameless post-incident review and track corrective actions until each one is closed and verified.

7Records, Training, Review, and Continuous Improvement

Records prove that controls were designed, approved, tested, and monitored. Keep a model register listing purpose, owner, risk tier, training data sources and licenses, evaluation results, known limitations, and version history. Store access reviews, incident reports, and change approvals with the register so auditors and reviewers can trace decisions.

Training should be role-specific and refreshed when the system changes. Developers need secure design and evaluation methods. Reviewers need guidance on recognizing hallucinations and unsafe outputs. End users need plain instructions on what the system is for, what it cannot be trusted to do, and how to report problems. Monitoring should track error rates, refusal rates, complaint volume, and drift against baseline metrics. The review cycle should be set by the risk tier, with high-risk systems reviewed more often than low-risk ones.

• Mandatory: Update the model register before every production change.
• Mandatory: Restrict or retire any model that fails acceptance criteria until it is corrected and retested.
• Mandatory: Record training completion for every person with production or administrative access.
• Recommended: Compare live performance against the evaluation baseline at a regular interval and document any drift.

8Responsible Use and Limits of Authority

Responsible use means matching the tool to suitable tasks. LLMs can draft, summarize, classify, translate, and explain, but they should not replace licensed professional judgment in medicine, law, engineering, finance, or safety-critical operations. Users must not present model output as verified expert advice. Disclose limitations, avoid claims that the system understands or feels, and respect any right a person has to opt out of automated processing under applicable law.

This document is educational guidance. It does not replace legal advice, formal training, professional authorization, approved procedures, or emergency services. Verify current laws, standards, and vendor terms with competent authorities or qualified professionals, and adapt these norms into locally approved procedures before use.

Implementation checklist

• Confirm the model owner, data steward, and security lead are named in the model register.
• Score the use case and assign a risk tier before development begins.
• Confirm that no personal or confidential data enters prompts, logs, or training without documented approval.
• Store all credentials in an approved secrets manager and rotate any exposed key immediately.
• Run prompt injection and data leakage tests before each release and after each material change.
• Require human confirmation for any tool action that writes, sends, buys, or deletes.
• Pin production model versions and retest before upgrading.
• Preserve logs and outputs during an incident and report within the policy timeframe.
• Verify disclosure, privacy, and sector rules with qualified professionals.
• Update the model register and review the risk tier after every material change.