vaayaa Inclusive Education
Technical risk assessment

Generative AI Applications Risk Assessment: Output Quality, Data Handling and Oversight Controls

A worked example of how a product team building applications on generative AI finds, scores and controls risks to output accuracy, data, people, suppliers and public trust, with residual ratings and clear escalation.

Updated November 2026 8 hazards assessed Review every 3 months Highest risk: Critical to Medium
See the Generative AI Applications course

Executive summary

This assessment shows how a product team building on generative AI can find, rate and control the risks that matter most to accuracy, data, people, suppliers and public trust. Eight risks were chosen because they are plausible in everyday building and running of the application. Fabricated output is the most serious inherent risk, followed by data entry, hidden instructions and public failure. Each risk is scored before controls and again after existing and added controls. Most risks fall to Medium or Low once the added controls are in place, but public failure and data exposure stay at Medium because the impact remains serious. The first actions are source grounding for answers, input scanning, a tested rollback switch and a named owner for public response. The application is a teaching example, so every score illustrates method rather than a real organization.

Focus area
Generative AI applications and responsible deployment
Setting
Product and engineering teams that build internal or customer-facing applications on third-party generative AI services
Method
Five by five likelihood and impact risk register with inherent and residual scoring

Scope

This assessment covers a team that designs, builds and runs an application using generative AI to draft text, summarize documents and answer questions for staff and customers. It covers the path from requirements and data handling through testing, release, operation and retirement. It does not cover research into new underlying systems or any named vendor. Jurisdiction, user numbers and the sensitivity of data are not specified, so they are treated as assumptions and must be confirmed in any real assessment.

Context

The team has a product lead, two engineers, a data protection lead, a security lead and a small support desk. The application connects to a hosted generative service through an interface, reads approved internal documents and writes replies into a web chat and an internal tool. Usage is logged, complaints go to the support desk and the product lead reports to a steering group each quarter. Each risk is scored before controls to show its natural level, then again after existing controls and the additional controls the team intends to add.

Objectives, assets and stakeholders

The objective is to deliver useful, accurate and fair answers, protect personal and confidential data, keep people skilled and in charge of decisions, maintain a reliable supply of the service and keep the trust of customers and staff. The assets in scope are the application, its connected sources, usage logs, customer and staff data, the contracts with the hosted provider and the reputation of the product. Stakeholders include product, engineering, security and data protection staff, managers, customers, rights holders, the provider, the steering group and regulators where they apply. Every risk has one owner role, and the product lead keeps the register current and reports to the steering group each quarter.

Risk matrix: before and after controls

Each cell shows hazard numbers placed by likelihood and impact. Compare the two grids to see how controls move each hazard.

Before controls

Inherent risk

Risk before controls: rows are likelihood from almost certain to rare, columns are impact from insignificant to catastrophic. Cells list hazard numbers.
1
Insignificant		 2
Minor		 3
Moderate		 4
Major		 5
Catastrophic
5 Almost certain
4 Likely Likelihood 4, impact 3, score 12, high: H04 Likelihood 4, impact 5, score 20, critical: H01
3 Possible Likelihood 3, impact 3, score 9, medium: H06 Likelihood 3, impact 4, score 12, high: H03 H05 Likelihood 3, impact 5, score 15, high: H02 H08
2 Unlikely Likelihood 2, impact 4, score 8, medium: H07
1 Rare

After controls

Residual risk

Risk after controls: rows are likelihood from almost certain to rare, columns are impact from insignificant to catastrophic. Cells list hazard numbers.
1
Insignificant		 2
Minor		 3
Moderate		 4
Major		 5
Catastrophic
5 Almost certain
4 Likely
3 Possible Likelihood 3, impact 2, score 6, medium: H04
2 Unlikely Likelihood 2, impact 2, score 4, low: H06 H07 Likelihood 2, impact 3, score 6, medium: H01 H02 H03 H05 Likelihood 2, impact 4, score 8, medium: H08
1 Rare
  • Low 1 to 4
  • Medium 5 to 9
  • High 10 to 16
  • Critical 17 to 25

Hazard register

A summary first, then the detail for each hazard.

Summary of hazards with scores before and after controls
No.HazardBeforeAfterOwner
H01 Inaccurate or fabricated output relied on as fact 20 Critical 6 Medium Product lead
H02 Sensitive data entered into the application 15 High 6 Medium Data protection lead
H03 Hidden instructions in untrusted content redirect the application 12 High 6 Medium Security lead
H04 Over-reliance and loss of skill among staff 12 High 6 Medium Team manager
H05 Biased or unfair outcomes for some groups 12 High 6 Medium Ethics and fairness lead
H06 Dependence on one external provider 9 Medium 4 Low Supplier manager
H07 Output that copies protected or licensed material 8 Medium 4 Low Legal and compliance lead
H08 Public-facing failure harms reputation 15 High 8 Medium Product director
H01

Inaccurate or fabricated output relied on as fact

Operational Before 20 After 6

The application produces plausible but false statements, citations or figures, and staff or customers act on them without checking, leading to wrong decisions and harmful advice.

Who or what is affected: Staff, customers, decision makers and the product team

Causes

Generated text is probabilistic, user requests are vague and no source checks are built into the workflow.

Consequences

Wrong advice, rework, complaints, regulatory questions and loss of trust in the product.

Existing controls

  • Output carries a notice that it may be wrong.
  • Staff training covers the limits of generated answers.

Control gaps

There are no automatic checks against source material, and no record shows which answers were reviewed by a person.

Additional controls

  • Prevent: Ground answers in approved source documents and decline to answer when no source supports the reply.
  • Detect: Sample answers each week and score them for accuracy against the source material.
  • Respond: Flag disputed answers to the support desk and correct the source or the guidance.
Likelihood 4 x impact 5 before; 2 x 3 after Owner: Product lead Target: Within 45 days Early warning: Accuracy sample scores falling below the agreed threshold or a rise in user corrections.
H02

Sensitive data entered into the application

Technology and data Before 15 After 6

Users paste personal, financial or confidential information into the application, and that data is stored, logged or passed to a third party without the controls the organization expects.

Who or what is affected: Customers, employees, data subjects and the data protection lead

Causes

Users are busy, the interface looks like a private notepad and logs keep more than the team needs.

Consequences

Privacy breach, contract breach, investigation by a regulator and damage to customer confidence.

Existing controls

  • Users are told not to enter personal or confidential data.
  • Access to usage logs is limited to the support team.

Control gaps

Logs are kept for an unspecified period and no one checks what users actually enter.

Additional controls

  • Prevent: Remove personal identifiers from inputs before they leave the organization boundary.
  • Detect: Scan inputs for patterns of personal or confidential terms and alert the data protection lead.
  • Recover: Keep a documented process to delete stored inputs and inform affected parties where required.
Likelihood 3 x impact 5 before; 2 x 3 after Owner: Data protection lead Target: Within 30 days Early warning: Input scans that flag personal or confidential terms more than a few times in a week.
H03

Hidden instructions in untrusted content redirect the application

Technology and data Before 12 After 6

Text inside a retrieved document, web page or user upload carries instructions that override the application's rules, causing it to reveal data, change its tone or trigger an action it should not take.

Who or what is affected: Users, connected systems and anyone whose data the application can read

Causes

The application treats all text as potentially meaningful, and retrieved content is not checked before use.

Consequences

Data disclosure, corrupted records, misleading replies and an incident needing external review.

Existing controls

  • Actions that change records need a user to confirm them first.
  • Only approved sources are connected to the application.

Control gaps

Retrieved content is not tested for embedded instructions, and action logs are not reviewed.

Additional controls

  • Prevent: Separate trusted instructions from retrieved content and limit the actions the application may take.
  • Detect: Log unusual outputs and actions and alert when a reply departs from its expected format.
  • Respond: Disable the connected source, review the logs and notify the security lead.
Likelihood 3 x impact 4 before; 2 x 3 after Owner: Security lead Target: Within 30 days Early warning: Logged replies that refer to instructions or that break the expected answer format.
H04

Over-reliance and loss of skill among staff

People Before 12 After 6

Staff accept generated drafts without thought, lose the skills the work needs and stop checking the reasoning behind a reply, so errors pass into decisions and the team cannot cope when the tool fails.

Who or what is affected: Staff, their managers and teams that depend on their judgment

Causes

Time pressure, a tool that sounds confident and little practice in checking work.

Consequences

Lower quality work, weaker professional judgment and dependence that becomes hard to undo.

Existing controls

  • Training on the tool is given at launch.
  • Managers review a sample of outputs each month.

Control gaps

Managers do not record how often outputs are edited, so the trend is invisible.

Additional controls

  • Prevent: Define which tasks need a named person to check and sign off the output before use.
  • Respond: Rotate staff through manual tasks and refresh training whenever the application changes.
Likelihood 4 x impact 3 before; 3 x 2 after Owner: Team manager Target: Within 60 days Early warning: A rising share of outputs accepted with no edit or check recorded.
H05

Biased or unfair outcomes for some groups

Legal and compliance Before 12 After 6

Output quality differs between groups of users, or wording reflects stereotypes, so some people receive poorer or more hurtful answers and decisions that rely on the output treat them unfairly.

Who or what is affected: Applicants, customers and groups who may be treated unfairly

Causes

Training material is uneven, testing covers few groups and the wording of questions is not reviewed.

Consequences

Unfair treatment, complaints, regulatory questions and harm to affected people.

Existing controls

  • The team reviews outputs for offensive wording.
  • Complaints can be raised through the support desk.

Control gaps

Testing has not been repeated since launch and the complaint categories do not record the group concerned.

Additional controls

  • Prevent: Test outputs across representative user groups before release and after every change.
  • Detect: Track complaint themes and compare outcomes between groups each quarter.
  • Respond: Suspend the affected feature for the group concerned until the cause is fixed.
Likelihood 3 x impact 4 before; 2 x 3 after Owner: Ethics and fairness lead Target: Within 90 days Early warning: Complaints or test results showing uneven quality for a single group.
H06

Dependence on one external provider

Supplier Before 9 After 4

The application relies on a single hosted service, so a price rise, a change in terms, a service outage or the withdrawal of a feature stops the product or makes it much more expensive to run.

Who or what is affected: The product team, customers and the finance function

Causes

Single source selection, rapid change in service terms and little cost headroom.

Consequences

Service interruption, budget overrun, forced redesign and lost customer confidence.

Existing controls

  • The contract includes a notice period for material changes.
  • Usage and cost are reported each month.

Control gaps

The fallback plan has never been run and no second source has been assessed.

Additional controls

  • Prevent: Keep the application interface separate so a second provider can be connected.
  • Respond: Hold a tested fallback plan for an outage or a sudden price rise.
Likelihood 3 x impact 3 before; 2 x 2 after Owner: Supplier manager Target: Within 90 days Early warning: Provider notices of changed terms or cost increases above the budget line.
H07

Output that copies protected or licensed material

Legal and compliance Before 8 After 4

Generated text reproduces passages from protected sources or closely follows licensed material, and staff publish it without checking, exposing the organization to claims from rights holders.

Who or what is affected: Rights holders, customers who reuse the output and the organization

Causes

Generated text can echo training material and users rarely check sources.

Consequences

Takedown demands, claims, licence disputes and reputational harm.

Existing controls

  • Users are told to review output before publishing.
  • Terms of use state who owns the output.

Control gaps

Scanning is not applied to internal documents or to replies shared outside the team.

Additional controls

  • Detect: Scan published output for close matches to protected source text.
  • Respond: Remove flagged content quickly and record the cause and the takedown.
Likelihood 2 x impact 4 before; 2 x 2 after Owner: Legal and compliance lead Target: Within 60 days Early warning: Matches reported by a scan or complaints from rights holders about reused text.
H08

Public-facing failure harms reputation

Reputational Before 15 After 8

A visible error, offensive reply or leak in the customer-facing feature is shared widely, and the organization is seen as careless with customers and their information.

Who or what is affected: Customers, partners, staff and the wider public

Causes

Rapid release, limited pre-launch testing and no plan for public response.

Consequences

Loss of customers, media attention, pressure from regulators and staff morale damage.

Existing controls

  • A launch review checks the product against known failure cases.
  • A public contact route is listed on the product page.

Control gaps

No one owns the public response and the rollback switch has not been tested.

Additional controls

  • Prevent: Limit public release to a pilot group until failure rates are within agreed limits.
  • Recover: Publish a clear account of the fault and the fix within an agreed time.
Likelihood 3 x impact 5 before; 2 x 4 after Owner: Product director Target: Within 30 days Early warning: Support or social contacts describing the same fault repeatedly within a week.

Rating method and evaluation

Each risk is rated on two five-point scales. Likelihood runs from Rare to Almost certain, and impact runs from Insignificant to Severe. Multiplying the two gives a score from 1 to 25, which falls into one of four bands: Low from 1 to 4, Medium from 5 to 9, High from 10 to 16 and Critical from 17 to 25. The score is a guide for ordering work, not a measurement. Inherent risk is the score before any controls. Residual risk is the score after existing and added controls. Impact is judged on harm to people, data exposure, cost, service continuity and trust, and a risk is never rated down only because it has not yet happened. Anything Critical needs immediate action and steering group attention, anything High needs a dated plan with a named owner, anything Medium needs routine controls, and anything Low is accepted and watched. Owners should challenge each rating with evidence from testing and complaints, and every rating is reviewed whenever an indicator reaches its threshold.

Likelihood, impact and rating bands

Likelihood

  • 5 Almost certain: Expected to happen in most circumstances.
  • 4 Likely: Will probably happen in many circumstances.
  • 3 Possible: Could happen at some time.
  • 2 Unlikely: Not expected, but it could happen.
  • 1 Rare: Only in exceptional circumstances.

Impact

  • 1 Insignificant: No lasting effect. Handled within routine work.
  • 2 Minor: Small effect that is quickly corrected.
  • 3 Moderate: Noticeable effect that needs managed effort to correct.
  • 4 Major: Serious effect on people, delivery or reputation.
  • 5 Catastrophic: Severe or lasting effect that threatens the whole activity.

Bands (likelihood x impact)

  • Low (1 to 4): Accept and monitor.
  • Medium (5 to 9): Manage with routine controls.
  • High (10 to 16): Senior attention and a treatment plan.
  • Critical (17 to 25): Act first. Do not proceed until reduced.

Monitoring, escalation and review

Review this assessment every 3 months, and sooner when an indicator below reaches its threshold.

Early-warning indicators and escalation thresholds
Indicator to watchEscalate when
Accuracy sample scores and user corrections for each week of operation.Escalate to the product lead when the score falls below the agreed threshold, and to the steering group if it stays low for two review cycles.
Input scans flagging personal or confidential terms in user entries.Escalate to the data protection lead at once, and hold the affected feature if any exposure is confirmed.
Logged replies that show signs of hidden instructions or unexpected actions.Escalate to the security lead immediately, and disable the connected source until the logs are reviewed.
Complaints grouped by theme and by the user group concerned.Escalate to the ethics and fairness lead when one group shows repeated worse outcomes, and suspend the feature for that group.
Provider notices of changed terms and monthly cost against budget.Escalate to the supplier manager when a notice is received or cost passes the budget line, and to finance if the overrun continues for a month.

Contingency arrangements

If a serious event occurs, such as a confirmed data exposure, a harmful public reply or a provider outage, the product lead pauses the affected feature and the security lead preserves logs. The data protection lead assesses whether any person must be told, and the supplier manager activates the fallback plan. The communications lead prepares a plain account for customers and staff, and the steering group is briefed within one working day.

Prioritized action plan

  1. Ground answers in approved sources and decline unsupported replies, owner product lead, within 45 days.
  2. Scan inputs for personal and confidential data and set a log retention limit, owner data protection lead, within 30 days.
  3. Test retrieved content for hidden instructions and limit the actions the application can take, owner security lead, within 30 days.
  4. Pilot public release with a rollback switch tested before launch, owner product director, within 30 days.
  5. Assess a second provider and run the fallback plan once, owner supplier manager, within 90 days.

Conclusion

Trustworthy generative applications rest on steady habits: grounding answers in approved sources, scanning inputs, checking outputs across groups, keeping people in charge of decisions, planning for provider change and having a tested rollback. The highest inherent scores sit with fabricated output, data entry and public failure, and each has actions that reduce its residual score. Treat the register as a living record, review it on the stated cycle, and seek qualified legal, privacy and security advice for any real deployment.

Assumptions and limitations

Assumptions

  • The application uses a hosted generative service accessed through a documented interface rather than one built in-house.
  • Users are employees and a limited group of customers, and no automated decision is made about a person without a human check.
  • Jurisdiction is not specified, so legal duties are described in general terms only.

Limitations

  • The scores are examples for learning and do not describe any real product, vendor or organization.
  • Privacy, security, employment and consumer rules differ by place and change over time, so specialist review is needed.
  • This is educational content. It does not replace a formal site-specific assessment, security testing or licensed legal advice.

Related frameworks to study

  • Risk register with likelihood and impact scoring
  • Threat modelling for applications
  • Human oversight and review of automated output
  • Data minimization and classification practice
  • Secure development lifecycle practices

Technical Risk Assessment

China, India, United States, Indonesia, Pakistan, Nigeria, Brazil, Bangladesh, Russia, Japan, Mexico, Philippines, Vietnam, Egypt, Turkey, Germany, United Kingdom, France, Italy, Canada, South Korea, Australia, Spain.

For Learning Reference Only: This assessment demonstrates how risk assessment principles are applied in a practical setting, providing a useful reference for learning and understanding. Actual assessments may be adapted to suit specific site conditions and circumstances.

Keep learning

Study the subject behind this assessment.

Generative AI Applications course Read the best practices for this course

More Technical risk assessments

Hourly Active: 35,661 Learners