When AI goes rogue, what’s at stake for CX? 

Frontier AI research may feel far removed from day-to-day CX, but the concerns driving alarm among researchers and regulators are increasingly relevant to the AI agents, copilots, and autonomous workflows CX teams are eager to put into production.

To make sense of the risks, we spoke with TTEC Digital solution architects Ryan Boyer and Nathan Galloway. Their work with AWS and Google Cloud at the forefront of AI-powered CX gives them firsthand insight into the lessons emerging from real-world deployments.  

The following conversation has been edited for length and clarity.

Could the doomsday scenarios in the headlines happen in my CX environment?

Nathan Galloway: Here’s the good news. The most dramatic scenarios, including the extinction headlines, are still largely frontier questions. Lab incidents in sandbox environments shouldn’t be conflated with the risks CX leaders face in their live production environments.  

For CX leaders, the more immediate concern is what’s already in production and whether the right safeguards are in place.  

Ryan Boyer: And in most instances, CX leaders aren't solving the model safeguard problem on their own. I recently saw a stat showing that Anthropic model adoption is still heavily concentrated on platforms like Amazon Bedrock and Vertex AI.

AWS, Google, and Microsoft have earned a lot of trust for their privacy and security controls. Could an open-weight model (that is, AI models whose underlying code and parameters are publicly accessible) behave unexpectedly? Sure. But it’s still surrounded by the traditional security controls these companies provide, which should give leaders some peace of mind.

What AI risks should CX leaders be worried about?  

Nathan: Recent reported incidents offer real lessons, like models using exposed credentials, accessing data, or making up answers when they can’t find information. As we build agents for similar CX tasks, we need to understand those risks and put guardrails in place before production.

Ryan: Exactly. Is your chatbot going to start a real-life Skynet or hack into Hugging Face? Not likely. It’s more likely what Nathan said: “Is my chatbot giving out wrong information? Is it leaking PII or PHI?” That’s what the risk looks like in CX.  

What governance mistakes are you seeing organizations make right now?

Ryan: I see organizations tend to land at two extremes: moving so cautiously they miss AI’s benefits, or so fast that governance isn’t keeping up.

When you're in an overly conservative model, all it does is lead to people doing AI on their own. Frustrated users start installing whatever they want on their personal laptops or running a little Mac Mini sitting right next to their desk. Suddenly, all your corporate data is sitting outside your boundary just because IT said, "We're sticking with Copilot and not letting anyone use Claude because we don't know Claude."

On the “too fast” side, even a small prompt change — like swapping a comma for a colon — can shift an agent’s behavior in ways a one-line code change never would. You need time to test, observe, and adjust.

We’re also seeing a rush to connect AI to every available data source. Model Context Protocol (MCP) makes these connections easier, but easier doesn’t mean safer. You still need to understand what those connections allow and where your data can go.

For example, by allowing MCP, people connect their work Copilot to their personal Gmail accounts. Which means, theoretically, the day they decide to resign, that person can just say, "Hey Copilot, can you make a script that sends all my email contacts to my Gmail real quick? Thanks, bye."  

Four lessons for building AI more responsibly

#1: Grade agents on the right KPIs.

Nathan: If you’re measuring only containment and deflection, for example, you can steer an agent toward hitting a number instead of helping the customer. And that can lead to unintended consequences. Make sure your measures reward the outcomes you actually want.

Look at this inside the contact center: if you start grading your agent purely on containment, it will actively go out and find a way to hit that number. If you grade agents on the wrong numbers, you're going to lead yourself right to the wrong results.

#2: Put governance in the system, not just on paper.

Nathan: Governance boards and intake forms aren’t enough if there’s almost no policy in the agents themselves. Put safeguards into your configuration, logging, and permissions so governance actually controls what the agent can do, not just how it gets built.

#3: Don’t treat go-live as the finish line.

Nathan: An agent doesn’t stay the same — it gets better or worse. That’s why you need to have a robust observability strategy in place so you can catch problems early and understand whether changes are having the desired effects. Not every application needs weekly updates, but launch can’t be where the work stops or you’ll end up with an agent that has drifted in the wrong direction.

#4: Don’t build one agent to do everything.

Ryan: Give your agents a clear purpose, then coordinate how they work together. Otherwise, every single change means testing every scenario. In practice, that means differentiating between orchestration agents at the top of your pyramid and task-based agents at the bottom. You should be able to improve your password-reset agent without disrupting all of your other agents. That differentiation is critical to making ongoing improvement more manageable.

People prototype something working in Claude Desktop or ChatGPT and think going to production with a raw API model will be easy. They don't realize those consumer apps have massive system prompts built behind the scenes. The UI makes it look drop-dead simple, but when you hit the native LLM directly in an enterprise setup, it doesn't perform the same way at all.

Nathan: That’s why the next trend for computing isn't going to be rapid innovation on the raw models themselves. Far more money is about to be spent learning how to protect, sandbox, and put safety measures around the intelligence we already have.  

Ryan: At the end of the day, it’s about finding that right blend of productivity and security. If you’re too conservative or too open, you create vulnerabilities or drive people to do AI on their own. When you get the architecture and governance right, you get real results without slowing down.

[cta-1]