AI Security
AI governance framework: set the rules before your AI goes rogue
An AI governance framework decides which AI systems your company uses, what each may do, who approved it and how you notice when it goes wrong. NIST AI RMF supplies the risk questions, ISO/IEC 42001 the management system around them, and an acceptable use policy tells staff what they may do today.

Published 6 min read
An AI governance framework is the set of rules, roles and processes that decides which AI systems your company uses, what each one may do, who approved it and how you will notice when it goes wrong. In practice it holds four things: an inventory of the AI in use, policies that include an acceptable use policy for staff, a risk assessment for each system, and a named owner for every decision. NIST's AI Risk Management Framework supplies the risk questions. ISO/IEC 42001 supplies the management system that keeps them asked. Written down, it is a set of rules your leadership can stand behind when something goes wrong.
When AI goes rogue: five cases a framework would have caught
None of these needed a sophisticated attacker. Each was an ordinary use of AI with no rule, owner or check behind it.
- The source code that left the building. In April 2023, Samsung engineers pasted internal source code into ChatGPT to get help with it, according to an internal memo reported by Bloomberg. Samsung then banned generative AI tools on company devices. An acceptable use policy that says which data never goes into an AI tool, and an approved tool that keeps it in, would have made the ban unnecessary.
- The chatbot that made a promise. Air Canada's website chatbot told a grieving customer they could claim a bereavement fare refund after travelling, which the airline's policy did not allow. In February 2024, a Canadian tribunal held the airline to its chatbot's answer: it makes no difference, the tribunal wrote, whether information comes from a static page or a chatbot. A customer-facing AI system needs an owner, a risk assessment and testing against the policies it talks about.
- The citations that did not exist. In 2023, two New York lawyers filed a brief citing court decisions that ChatGPT had invented. The judge in Mata v. Avianca imposed a $5,000 penalty on them and their firm. Human review of AI output before it reaches a court, a customer or a contract is the rule that was missing.
- The report that cost a refund. In 2025, Deloitte Australia repaid part of an AU$440,000 government contract after a report it delivered was found to contain fabricated references; the corrected version disclosed that a generative AI tool had been used. Disclosure and review rules belong in the framework, not in an apology.
- The agent that deleted production. In July 2025, Replit's AI coding agent deleted a customer's production database during a code freeze. Replit's chief executive called it unacceptable and promised to separate development and production databases automatically. An AI system that can act should have limits on what it may do without a person deciding.
The pattern is not rare. In IBM's 2026 Cost of a Data Breach study, about one in five organisations reported a breach targeting AI models or applications, and most of those lacked proper AI access controls. Verizon's 2026 Data Breach Investigations Report found that 45% of employees regularly use AI on their corporate devices, authorised or not, up from 15%.
What an AI governance framework contains
An inventory. Rules cannot cover systems nobody has listed, so the work starts by finding the AI already in use. When we build one, the same four things turn up that leadership did not know about: AI features switched on inside software the company already pays for, such as the CRM, the office suite, meeting transcription or the help desk; staff pasting contracts, customer lists or code into public chatbots on personal accounts; developers' coding assistants with access to the repositories, and sometimes to secrets left in the code; and a department's pilot running on a vendor's API key paid for with a corporate card, with no security or legal review. Each entry records the data the system touches and the person who owns it.
Policies. Short, written for the people who will read them: what staff may do with AI, which new uses need approval, and what a vendor must show before its AI feature is switched on.
A risk assessment for each system. The same questions, asked before a system goes live and again when it changes: what can it reach, what happens when it is wrong, and who would notice.
Owners and review. A named person approves each system and answers for it, and the framework is reviewed on a schedule, because the AI in use changes faster than most policies do.
Where NIST AI RMF fits
NIST's AI Risk Management Framework, published in January 2023 as version 1.0, is intended for voluntary use. It organises AI risk work into four functions: Govern sets the culture, roles and policies; Map establishes the context and risks of a system; Measure analyses and tracks them; Manage acts on them. It describes trustworthy AI by seven characteristics: valid and reliable; safe; secure and resilient; accountable and transparent; explainable and interpretable; privacy-enhanced; and fair, with harmful bias managed. NIST says it is revising version 1.0; no new version had been published when this article was written.
We use the AI RMF for the questions asked of each system. For assistants and other generative AI, NIST's Generative AI Profile (NIST AI 600-1, 2024) adds the risks particular to those systems, such as confabulation, where a model states invented facts with confidence, and the exposure of data through prompts and answers. The Avianca citations and the Deloitte references were confabulation reaching a reader unchecked.
Where ISO/IEC 42001 fits
ISO/IEC 42001:2023 is the international standard for an AI management system. Where the AI RMF tells you what to ask about a system, ISO/IEC 42001 sets requirements for the organisation around it: roles, objectives, risk treatment, internal audit, management review and continual improvement. Its structure follows the other ISO management system standards, so a company that already runs ISO/IEC 27001 will recognise it. Our comparison of ISO 27001 and SOC 2 explains how that kind of certification works, and how it differs from an attestation.
Unlike the AI RMF, ISO/IEC 42001 is certifiable. An accredited certification body audits the management system and issues the certificate, and ISO/IEC 42006:2025 sets the requirements those bodies must meet. GUILDA does not certify. Our advice is to certify only when customers or contracts ask for it, and until then to build to the standard's structure, so that certifying later is a step rather than a rebuild.
What an AI acceptable use policy covers
The acceptable use policy is the part of the framework every employee reads, so it decides whether the rest is followed. Four things make it work:
- Approved tools, and a quick way to ask for more. A list of the AI tools staff may use and a short request process for a new one. If approval takes months, people use their own accounts instead.
- Data that stays out. Which data may never go into an AI tool, tied to the data classification the company already has: customer personal data, source code, contracts, financial results before release.
- Human review. A person reviews AI output before it reaches a customer, a contract or production code, and the person who uses the output owns it. "The AI wrote it" did not get Air Canada off the hook.
- Blameless reporting. A simple way to report a mistake or a leak without blame, so problems surface while they are small.
Three mistakes to avoid
Banning AI outright. A ban moves AI onto personal phones and accounts, where nobody can see it. Verizon's 2026 report found that 67% of users reached AI services with non-corporate accounts on corporate devices. A ban makes that use invisible, not absent.
A template with no owner. A policy copied from a template, with no named owner and no way to get a new tool approved, is ignored within a month. The framework works when someone is accountable for each decision.
Leaving it to IT. AI touches contracts, hiring, customer promises and intellectual property. Legal, HR and the business owners who live with each system need a seat when the rules are written, alongside security and IT. A broader security governance program is often the natural home for that group.
Where to start
Start with the inventory, then write the acceptable use policy, because those two reduce the most risk soonest. Add the risk assessment for each system next, and the management system structure once the basics run. If you are building AI rather than only buying it, put governance into the design from the beginning: our AI and ML implementation work and our AI security service can be scoped together, so the rules are part of the system rather than a review at the end.
Frequently asked questions
What is AI governance?
AI governance is how a company decides which AI it uses, for what, under whose authority and with which safeguards. A framework writes those decisions down as an inventory, policies, a risk assessment for each system and named owners, so the answers do not depend on who happens to be asked.
Is NIST AI RMF mandatory?
No. NIST publishes it for voluntary use. Companies adopt it because it gives every AI system the same risk questions, and because customers, auditors and boards increasingly recognise it. NIST is revising version 1.0; until a new version is published, 1.0 is the current text.
Can GUILDA certify us to ISO/IEC 42001?
No. Only an accredited certification body audits an AI management system and issues the certificate. We build the governance framework, structured the way the standard expects, so that certifying later is a step rather than a rebuild. If certification is your goal from the start, we plan the framework around it.
How long does it take to put an AI governance framework in place?
Our governance framework engagements take two to six weeks at a fixed price, depending on how many AI systems are in use and how many teams take part. The inventory usually takes longer than people expect, because it finds AI nobody had listed.
Sources
- Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, NIST, 2023
- Generative Artificial Intelligence Profile, NIST AI 600-1, NIST, 2024
- ISO/IEC 42001:2023, AI management systems, ISO, 2023
- ISO/IEC 42006:2025, requirements for AI management system audit and certification bodies, ISO, 2025
- Cost of a Data Breach 2026: AI adversaries and enterprise risk, IBM, 2026
- 2026 Data Breach Investigations Report, executive summary, Verizon, 2026
- Samsung bans staff AI use over data leak concerns, CSO Online, 2023
- Air Canada chatbot costs airline discount it wrongly offered customer, CBS News, 2024
- Mata v. Avianca, opinion and order on sanctions, 22-cv-1461 (S.D.N.Y.), United States District Court for the Southern District of New York, 2023
- Deloitte refunds Australian government over AI-assisted report, The Register, 2025
- Replit responds after its AI agent deleted a production database, The Register, 2025
AI governance consulting and AI security
We help you govern, secure and watch the AI systems your company builds or buys. That means a governance framework your leadership can stand behind, a security review that tests how each system can be misused, the hardening it needs, and detection your team can operate.
Explore AI SecurityRelated reading
AI Security
10 ways your AI can be turned against you: OWASP Top 10 for LLMs vs MITRE ATLAS
A chatbot that invents a refund policy, an assistant that leaks mail without a click, a coding tool shipped with planted code. The OWASP Top 10 for LLM Applications names these risks for builders; MITRE ATLAS records how attackers carry them out. Here is how the two fit together.
7 min read
AI Security
Agentic AI security: your newest employee has admin rights and no manager
AI agents do more than answer. They call tools, read data and act in other systems, often with a person's keys. Five public incidents from 2025 show what goes wrong, and each points to a control, from an identity of the agent's own to a named owner who can stop it.
7 min read
Tell us which AI systems you need to govern
List the tools and systems you use, built or bought. We reply within two business days.