AI Governance for the Rest of Us

Five Questions We're Learning to Ask at CRAI

At the Center for Rural AI, we spend a lot of time building with artificial intelligence. That has also meant spending a fair amount of time taking things apart.

We've watched AI make complicated work remarkably easier. We've also watched it generate unnecessary architecture, follow the appearance of a process rather than its intent, confidently report success without adequate evidence, and obscure substantial technical authority behind a simple conversational interface.

Those experiences have shaped a principle in our technology work: complexity has to earn its place.

So when we began looking more formally at AI governance, we started with a practical question. What does responsible AI governance look like for an organization that needs to use increasingly capable technology but doesn't have a governance, compliance, security, and AI engineering department standing behind it?

That question matters to CRAI because many of the organizations we work with operate exactly this way. The person exploring AI may also write grants, manage programs, maintain records, communicate with the public, or report to a board or town council.

They don't need weaker governance. They need governance they can actually practice.

What the larger frameworks tell us

We started by examining some of the leading work already underway: the European Union's risk-based approach, NIST's AI Risk Management Framework, Singapore's work on agentic AI, the OECD's principles for trustworthy AI, and the United Nations' emerging scientific assessment of AI capabilities and risks.

These are very different things. The EU AI Act establishes legal obligations. NIST provides a voluntary risk-management framework. Singapore has developed practical guidance for governing increasingly autonomous systems. The OECD establishes broader principles for trustworthy AI, while the UN is building an international scientific evidence base around AI capabilities, risks, and governance.

We don't want to flatten those differences into a universal framework that doesn't exist. But at the operating level, recurring concerns appear: understand how AI is being used and what can go wrong; increase safeguards as consequences increase; preserve meaningful human responsibility; make consequential activity visible and traceable; verify important outcomes; protect people, information, rights, and systems; and revisit controls as the technology changes.

That gave us a useful foundation. The next question was how much of it an ordinary organization needs to carry into its daily work.

Start by subtracting

We've learned through our own technology work that AI makes complexity extraordinarily cheap to create. More agents, more workflows, more documentation, more metadata, more automation, and more governance can all be generated faster than the people responsible for them can understand or maintain them.

Our response has increasingly been subtraction.

Simplicity is reached by subtraction, not addition.

That doesn't mean removing safeguards until something breaks. Some requirements exist before the first system is built: law, security, privacy, credential protection, recovery, and duties to the people an organization serves. Other controls become necessary because the consequences of failure are foreseeable even if failure hasn't occurred yet.

The goal isn't minimum governance. It is minimum sufficient governance: enough structure to manage the actual consequence without creating a system so complicated that the people responsible for it can no longer understand or operate it.

Subtract complexity, not responsibility.

Working from that principle, we reduced the recurring governance concerns we were seeing to five questions.

Five questions

1. Do we actually need this?

Start with the work, not the technology. What problem are we solving? Do we understand the process well enough to automate it? Could a simpler approach meet the need?

AI makes it easy to architect for possibilities that haven't happened yet. Our preference is to build to demonstrated need — including foreseeable risk — and let experience tell us when additional complexity has earned its place.

2. Do we understand what it's doing?

Not every operator needs to understand the mathematics inside a language model. But someone responsible for an AI system should understand it operationally: its purpose, what information and systems it can access, what tools it can use, what authority it has, where approval is required, what dependencies matter, and where credible failure modes remain.

A simple interface can hide a surprisingly complicated authority model.

3. What happens if it's wrong?

An incorrect draft that a staff member can discard is different from an incorrect payment, database change, public communication, access decision, or action affecting another person.

The harder an action is to reverse and the greater its consequence, the stronger the oversight should become. But individual reversibility isn't the entire question. Small errors can accumulate into patterns that matter.

What happens if it's wrong once — and what happens if it's wrong over time?

4. How do we know it worked?

An AI system reporting that it successfully completed a task is not evidence that the task was completed correctly.

Verification should become stronger as consequence increases. Sometimes that means checking an authoritative source. Sometimes it means inspecting a resulting artifact, reviewing logs, sampling completed work, or requiring another person to validate the result.

This is one place where our experience building AI systems has made us particularly cautious. Agents can learn the appearance of success. A QA process can return “green” while failing to perform the investigation the process was designed to require. Human review can become equally mechanical when approval is repeated often enough.

Governance therefore has to care about evidence, not simply whether a box — human or machine — reported success.

5. What can't we afford to lose — or harm?

This question changed while we were developing the framework.

Our original version was simply, “What can't we afford to lose?” It worked well for records, credentials, sensitive information, institutional knowledge, and operational capability.

Then we tried to break it.

We subjected the framework to adversarial review, asking independent AI systems to look for situations where an organization could answer all five questions satisfactorily and still produce an irresponsible outcome. One of the strongest challenges involved systems that could be useful, understandable, individually reversible, and apparently well verified while producing cumulative or uneven harm to the people affected by them.

The original question was too focused on what the organization could lose. The revised question forces us to include people, privacy, rights, fairness, trust, and other interests that may not belong to the organization at all.

The person matters on their own.

Trying to break the framework

The adversarial review wasn't an academic exercise. It is part of how we've learned to build AI systems.

We increasingly separate building from verification. We use independent review where consequence warrants it, test whether systems actually followed required processes rather than merely produced the expected result, and deliberately ask reviewers — human and AI — to look for evidence that contradicts the conclusion we want.

We applied the same discipline here. The review found weaknesses.

It challenged our subtraction principle by pointing out that some controls prevent failures we may never see precisely because the controls worked. It challenged our thinking about human approval because repeated approval can deteriorate into ceremony. It exposed risks that occur before any obvious decision point, such as sensitive information entering a model. And it showed that individually acceptable decisions can produce unacceptable patterns over time.

We kept the five questions, but changed what they have to account for.

A governance framework shouldn't be judged by whether it survives criticism unchanged. It should be judged by whether criticism makes it better.

Capability is not authority

Testing these ideas against systems we've actually built produced another useful distinction.

AI can perform substantial work without having authority to turn all of that work into organizational action. A system can research, analyze, draft, code, compare, recommend, or prepare a change while another control determines whether the result becomes official, external, persistent, or difficult to reverse.

We've found it useful to identify those transitions explicitly. We call one such transition a consequential boundary: the point where AI-assisted work becomes materially more authoritative, external, difficult to reverse, or consequential.

Publishing information can be a consequential boundary. So can changing an official record, sending an external communication, moving money, granting access, or deploying software.

But the adversarial review improved this idea too. Sensitive information can be exposed before a boundary is crossed. Excessive permissions can remain dangerous even when unused. Bias can emerge gradually across many individually acceptable decisions.

Some risks happen before the line. Some never cross a line at all.

The boundary is therefore a useful operating tool, not a complete theory of AI risk.

The questions aren't the controls

There is a significant limitation to this approach. Five questions can't create expertise an organization doesn't possess.

A nonprofit director who doesn't know a particular security vulnerability exists can't discover every technical safeguard simply by asking better questions. Applicable laws remain applicable. Security practices still matter. High-consequence systems may require technical expertise, formal assessment, or controls prescribed by standards and regulation.

The questions are an entry point.

Their answers should produce controls appropriate to the work: MFA, backups, restricted permissions, source verification, human approval, independent review, audit samples, monitoring, limits on what information can enter an AI system, or sometimes a decision not to automate at all.

The simplicity belongs in the way the organization begins reasoning about governance. The controls that follow should be as sophisticated as the consequence requires.

Governance for organizations without governance departments

This is where the work becomes particularly relevant to rural organizations.

Small organizations don't inherently have small AI risks. In fact, technology authority can be unusually concentrated. One person may have access to the AI account, database, credentials, communications, and public-facing systems. An overly privileged AI agent can concentrate that authority even further.

The human consequences are also close to home. The person affected by an AI-assisted decision may be a neighbor, student, business owner, grant applicant, constituent, employee, or community partner.

The answer isn't to reproduce an enterprise governance department inside a five-person nonprofit. Nor is it to decide that governance is something only large institutions can afford.

It is to understand the consequence, protect what matters, and build the smallest set of controls capable of carrying that responsibility.

That's the standard we're trying to develop at CRAI.

AI is going to keep changing, and these five questions probably will too. That's part of the design. Governance shouldn't depend on our ability to predict every rule we'll eventually need. It should help us see the system we're actually operating, the authority we've given it, the people it may affect, and the evidence we need before trusting its work.

• Do we actually need this?
• Do we understand what it's doing?
• What happens if it's wrong?
• How do we know it worked?
• What can't we afford to lose — or harm?

Five questions won't govern an AI system by themselves. But we're finding they are a useful place to begin figuring out what will.

And when the answers demand more structure, we add it. When that structure stops earning its place, we subtract it —

without subtracting responsibility.

Jason Strickland is Head of Technology at the Center for Rural AI, where he works on systems governance. The five questions above came out of the tools CRAI actually builds and operates — you can see that work in the Pilot Library and across what we do.

If this framework is useful to your own organization, we'd like to hear about it — and to learn more about CRAI's mission, the about page is a good place to start.