Safety Norms: Generative AI Applications

1Purpose, scope, and safety principles

This document sets norms for teams that design, build, test, deploy, and operate generative AI applications. These applications produce text, images, code, summaries, or recommendations from a prompt, often combined with retrieved documents or connected tools. The purpose is to prevent harm to people, protect data and systems, and keep each application inside its approved purpose.

Scope covers the model, system instructions, prompts, retrieval sources, connected tools, user interface, logs, and human review steps. It applies to prototypes, pilots, and production services. A prompt is the input that instructs the model. Retrieval-augmented generation, or RAG, pulls documents into the prompt before generation. A hallucination is output that sounds plausible but is false or unsupported. Prompt injection is untrusted text that tries to override the application's instructions.

• Mandatory: no application reaches real users without a written purpose statement, a named owner, and a completed risk assessment.
• Mandatory: safety decisions rest on documented testing and records, not on demonstrations or vendor claims alone.
• Recommended: design for the least capable configuration that meets the need, because fewer capabilities mean fewer failure modes.

2Roles, competence, and accountability

The application owner is accountable for purpose, approved users, risk acceptance, and go-live approval. Engineers build and test the system. Data stewards govern sources and access. Security staff review credentials, network paths, and tool permissions. Privacy and legal staff confirm that data handling matches applicable laws, which vary by jurisdiction and sector. Domain specialists, such as clinicians, lawyers, auditors, or underwriters where relevant, review outputs in their field.

Competence means documented ability, not job title. People who handle personal or confidential data complete privacy and information security training. Reviewers of high-impact outputs must hold the qualifications their profession requires. Team members without that qualification must not approve or override a specialist judgment or attempt tasks reserved for licensed professionals.

• The owner signs a go-live record naming approved users, purpose, and accepted residual risks.
• The security lead approves every tool integration and every credential with write access.
• Any team member may pause a feature flag or disable a tool when output or behavior appears unsafe, and must report the pause the same working day.
• A named vendor contact tracks model changes and terms of service.

3Realistic hazards and risk levels

Hazards arise from the activities around generative AI, not only from the model. Typical activities include writing system prompts, indexing documents, connecting tools, fine tuning models, testing responses, and serving end users. People involved include developers, prompt writers, reviewers, end users, administrators, and affected third parties. Environments include cloud accounts, on-premises GPU servers, user devices, and workplaces where outputs are read and acted upon.

Score each hazard by likelihood and severity on a five-point scale. High means serious harm, legal exposure, or large data loss is plausible; controls and sign-off are required before release. Medium means limited harm is plausible; documented controls and monitoring are required. Low means harm is unlikely and minor; routine logging is sufficient. Re-score after each material change.

• High: harmful or false output used for medical, legal, financial, employment, or safety decisions without a qualified check.
• High: sensitive personal data, credentials, or confidential records entered into prompts, logs, or third-party services without approved terms.
• High: prompt injection through web pages, emails, or uploaded files that causes an agent to send messages, change records, or move money.
• High: retrieval that returns documents the user may not read, because permissions were checked only at indexing time.
• High in regulated decisions: biased or unexplained outcomes affecting hiring, lending, housing, insurance, or service access, with rules that are jurisdiction dependent.
• Medium: overreliance by users who accept fluent answers without verification.
• Medium: copyright, licensing, or attribution problems, to be checked against current legal guidance.
• Medium: cost spikes or service denial from unbounded requests or runaway agent loops.
• Low to medium: energy use and heat from high-volume inference, recorded for review.

4Preventive controls and safe work practices

The following norms are mandatory unless the owner records an approved exception with compensating controls. Each norm has a responsible role. Where a control cannot be implemented, the feature must be limited or withheld.

• Intended use: the owner publishes allowed and prohibited uses. The application refuses or redirects out-of-scope requests where technically feasible, and the interface states its limits.
• Data minimization: engineers must not place regulated personal data, passwords, keys, or confidential records in prompts, examples, or fine tuning sets unless privacy and security approve the pipeline, retention period, and vendor terms.
• Least privilege: each connected tool receives only the scopes it needs. Actions that send messages, change records, spend money, or delete data require explicit human confirmation at the moment of action.
• Untrusted input: user text, retrieved documents, web pages, and tool results are treated as data, not instructions. System instructions are kept separate from content, and injection tests run before release.
• Permissions at query time: retrieval enforces the requesting user's access rights on every query. Index-level access alone is not sufficient.
• Output handling: generated content is escaped before rendering as HTML or markdown. Generated code never runs in production without sandboxing and review.
• Disclosure: AI-generated content is labeled. Factual claims show sources, or state that no source was found.
• Human decision: in medical, legal, financial, employment, housing, insurance, and safety contexts, a qualified person makes and records the final decision. The system may support but not decide.
• Evaluation: a documented test set covers accuracy, harmful content, refusal behavior, privacy leakage, injection attempts, and performance across user groups. Acceptance thresholds are set before testing.
• Regression: the test set is repeated after any change to model version, prompt, retrieval source, tool, or safety filter.
• Change control: model versions are pinned where the vendor allows. A vendor model change is treated as a new release.
• Limits: rate limits, per-user quotas, and spending caps are set. Agent loops have a maximum step count and a timeout.
• Prohibited uses: the application must not generate malware, deceptive impersonation of real people, sexual content involving minors, or assistance with weapons or hazardous materials. Testers do not attempt these outside an approved red team plan.
• Recommended: run red team exercises before launch and after major changes, with results fed into the test set. Publish a system card describing limitations and evaluation results. Use smaller specialized models where they meet the need, to reduce cost, energy use, and attack surface.

5Equipment, environment, information, and protective safeguards

Infrastructure must be managed as carefully as the model. Cloud accounts and API keys are stored in an approved secrets manager, rotated on a schedule set by security, and never embedded in client code or prompts. Development, testing, and production environments are separated by network controls. Self-hosted GPU servers involve electrical, thermal, and heavy equipment hazards. Installation and maintenance belong only to qualified facilities or hardware staff following manufacturer instructions and local electrical codes, which vary by jurisdiction.

Information safeguards include redaction of personal and secret data in logs, defined retention limits, restricted log access, and encryption in transit and at rest. Confirm where vendor processing occurs and whether provider data is used for training, and document the answers. Protective safeguards include content safety classifiers, output filters, a tested kill switch that disables generation without redeployment, and a fallback message that directs users to a human or an official resource. Environmental safety includes right-sizing models, batching non-urgent jobs, and recording estimated energy use as a review metric.

• Secrets: keys are scoped, rotated, and revoked immediately when a team member leaves or a leak is suspected.
• Vendor review: data use, retention, subprocessors, processing location, and breach notification terms are checked before onboarding. Terms vary and must be confirmed with qualified advisers.
• Kill switch: a tested control disables the feature within a defined and rehearsed time.

6Incident reporting, emergency response, and escalation

An incident is any event where the application causes or nearly causes harm. Examples include harmful or false output acted on by a user, exposure of personal or confidential data, injected instructions that trigger unauthorized actions, a cost spike, an outage, or a complaint suggesting unfair treatment. A near miss is an event caught before harm occurred. It is reported and analyzed like an incident because it reveals weak controls.

Report incidents to the application owner and security lead immediately, and no later than the same working day. The owner classifies severity and activates the response. Privacy or legal staff confirm any notification duties, which depend on jurisdiction and contract. Containment comes first: disable the feature or tool, revoke affected credentials, preserve logs, and stop further output of the same type. Then investigate cause, communicate with affected users through an approved message, and record corrective actions. Do not delete logs, prompts, or outputs during investigation, even if they contain errors.

• Severity 1: physical harm, large data exposure, or unlawful action. Escalate immediately to leadership, security, and legal.
• Severity 2: a user acted on harmful or false output, or a limited data exposure occurred. Escalate the same working day.
• Severity 3: a contained error with no harm, or a near miss. Log it and review at the next scheduled meeting.
• Emergency: if a person may be in immediate danger because of advice the application gave, the user or operator must contact local emergency services. This application and this document do not replace emergency services or qualified medical, legal, or safety advice.
• Post-incident reviews are blameless, focus on system causes, and assign each action an owner and a deadline.

7Records, training, inspection, and review

Records prove that controls exist and were followed. Keep the purpose statement, risk assessment, approvals, model and prompt versions, data source inventory, test sets and results, change tickets, incident reports, and training records. Store them where authorized staff can retrieve them, and keep them for the period set by applicable law, contract, or internal policy.

Training must match the role. All contributors complete a basic course on prohibited uses, data handling, and incident reporting. Developers and prompt writers receive secure development and injection testing training. Reviewers use a written rubric and calibration sessions so judgments stay consistent. Refresher training follows any material change to policy or design.

• Inspection: the owner checks logs, access lists, and open actions at the interval set in the go-live record, and security audits tool permissions.
• Review triggers: a model or vendor change, a new data source, a new user group, a new jurisdiction, a serious incident, or a sustained rise in complaints.
• Metrics: track sampled error rate, refusal accuracy, injection test results, complaints, cost per task, and time to contain incidents.
• Improvement: each review produces numbered actions with owners, and the owner reports status to leadership.

Implementation checklist

• Confirm the written purpose statement, named owner, and risk assessment exist before any real user access.
• Keep sensitive personal data, credentials, and confidential records out of prompts unless approved.
• Enforce each user's permissions at retrieval time, not only at indexing.
• Require human confirmation before any agent action that sends, changes, spends, or deletes.
• Run the documented evaluation and injection tests, and repeat them after every change.
• Label AI-generated content and show sources for factual claims.
• Test the kill switch and rehearse it on the schedule set by the owner.
• Report every incident and near miss the same working day, and preserve logs.
• Obtain legal, privacy, and domain specialist sign-off for regulated uses, based on current local rules.