AI: brilliant assistant, terrible boss

Artificial intelligence is having a moment. It writes emails, drafts contracts, summarises meetings and tells us things with such confidence that we forget it might be completely wrong.

AI can be a remarkable productivity tool, but it can also be the digital equivalent of an overenthusiastic intern with access to all your systems and no sense of self-preservation.

The challenge for organisations is not whether to use AI. That debate has largely passed. The challenge is how to manage its risks while still capturing its benefits.

Risk 1: AI hallucinations and human overconfidence

One of the most common AI risks is also one of the oldest: people trusting something that sounds plausible.

An embarrassing way to learn about AI risk is to quote legislation that was written by a chatbot rather than parliament. In a routine court application, a law firm presented AI-generated wording as if it appeared in the Insolvency Rules. The judge checked. The wording did not exist.

When the court asked for an explanation, the law firm sent a second AI-assisted response, which made matters worse. The judge found that the junior lawyer involved had largely outsourced the thinking process to AI and criticised the lack of supervision by senior lawyers.

The irony was that the AI system had repeatedly warned the user to verify its answers against authoritative sources. As the judgment makes clear, AI can be an excellent research assistant, but it should not be promoted to partner, director or decision-maker. Human judgement, verification and accountability remain non-delegable.

Risk 2: Data security and unintended consequences

Many organisations worry about employees putting confidential information into public AI tools, and rightly so. Under UK data protection law, personal data uploaded to third-party AI systems may constitute a transfer to another country or organisation, which requires appropriate safeguards to be in place. But newer risks are also emerging as AI systems become increasingly capable and autonomous. For example, some people’s chats with numerous frontier AI tools were made public, including Claude, ChatGPT and Grok. This included personal data.

Also, a recent security incident at Hugging Face, a major AI platform, demonstrated how an AI agent, operating during a cyber security evaluation, reportedly exploited weaknesses in the system and pursued its objectives with remarkable persistence. The striking aspect was not malicious intent. It was relentless automation combined with access and opportunity.

This highlights a simple truth: AI amplifies existing weaknesses. Poor passwords, excessive privileges, weak governance and unpatched systems become even greater risks when intelligent automation is involved.

Basic cyber hygiene suddenly becomes much more important when the attacker never sleeps.

Risk 3: Dependency on someone else’s AI

Many businesses assume that access to advanced AI models will always be available, affordable and unrestricted. That assumption may prove optimistic.

In June 2026, Anthropic withdrew access to one of its most powerful models after a US government directive restricted availability to foreign nationals. The incident triggered debate about national security, technology control and reliance on foreign-owned AI infrastructure.

At the same time, MPs have been urging the UK government to develop a clearer strategy for ‘technology sovereignty’ to ensure that access to critical technologies cannot be removed at the whim of external actors.

Risk 4: Reputation and trust

AI mistakes rarely remain private. An inaccurate legal filing, a flawed customer communication or a discriminatory automated decision can quickly become a headline.

Under UK data protection law, individuals have the right not to be subject to decisions made entirely by computer systems where those decisions significantly affect them. The Information Commissioner’s Office, the UK’s data protection regulator, has made clear that organisations must be able to explain how AI-assisted decisions are made. Trust takes years to build and minutes to lose.

Risk 5: Transparency and provenance

The EU AI Act’s transparency obligations are now in force, meaning certain AI systems must be clear when people are interacting with AI and requiring some AI-generated or manipulated content to be marked or labelled. The European Commission has published a voluntary Code of Practice on Transparency of AI-generated Content alongside official guidance to support organisations in meeting the transparency mandates. As part of its compliance, Anthropic has said it will now add a watermark to AI-generated content.

It’s important to assess when you use AI and, if you detect AI-generated content, whether you were expecting it and what relevance it has.

Practical steps to manage AI risk

Fortunately, managing AI risk does not require a PhD in machine learning, nor a budget the size of a small nation’s GDP.

Start with the basics:

1. Create an AI policy that defines acceptable use of AI, when it may be used, what uses are prohibited and when AI use must be disclosed or labelled
2. Train employees to understand AI strengths and limitations
3. Protect data by controlling what can be uploaded to AI tools. Keep personal and confidential information to a minimum and, where possible, use AI tools that operate in a secure, isolated environment (sometimes called a ‘sandbox’) where your data cannot leak out or be used to train the AI
4. Control sharing settings. Tell staff not to create public links to AI conversations which contain personal data, confidential information or pricing
5. Maintain human oversight of important decisions and content. Make sure this ‘human in the loop’ has appropriate skills and actively supervises the process and output. Remember, you can’t pin the blame on an AI tool. Record use of AI and the human oversight from a transparency and provenance standpoint
6. Check transparency rules. While the EU AI Act does not directly apply in the UK, identify whether users must be told they are interacting with AI, whether outputs need machine-readable marking and learn to recognise AI-generated output
7. Review suppliers carefully, including their security practices, operational resilience and contractual protections. Check for appropriate limits on liability, clear data handling terms and commitments about service availability and performance. Assess what happens if they change terms or withdraw their service.

Final thoughts

The biggest AI risk is not the machine – it’s the temptation to assume the machine is always right.

Most AI failures occur because organisations overlook governance, not because the technology suddenly becomes self-aware and starts demanding a corner office and shares in the business.

The organisations that will benefit most from AI are unlikely to be those that use it most aggressively. They will be those that use it thoughtfully, understand its limitations and build sensible controls around it.

In other words, treat AI like a very capable colleague: welcome its help, appreciate its speed, but always check its work before signing your name to it.