AI Chatbot Security: Considerations for Internal Deployments

Most internal chatbot incidents come from the plumbing around the model: what it can read as the user, who else reads the answer, how replies are rendered, and what happens when someone adds a tool. This post covers four places where internal deployments fail in practice, with documented attacks and incidents.

The model acts with the user's full reach

A chatbot that runs with a signed-in employee's permissions can read everything that employee can. Anyone who can put text in front of the bot inherits that reach. EchoLeak (CVE-2025-32711), disclosed by Aim Security, was a zero-click flaw in Microsoft 365 Copilot where an attacker's email was later pulled in as retrieval context. As Hack The Box's write-up describes it, the attacker got Copilot to return an image reference to an attacker-controlled URL with the data embedded in it, so loading the image sent the data out. Microsoft's cross-prompt-injection classifiers, which should have stopped the hidden instruction, were bypassed with specific phrasings, and Microsoft fixed the issue server-side.

Two lessons carry over. First, the exfiltration channel was output rendering, not the model's reasoning. Do not auto-load remote images or links from model output, and allowlist the destinations your UI will fetch. Second, a classifier in front of the model is a speed bump. A 2025 study of twelve published defenses bypassed most of them with adaptive attacks, which is why testing the boundary with the detector switched off matters. The limits that hold are the ones outside the model: what the token can reach, and which actions need a person.

Check who reads the answer, not just who asked

Row-level filtering by the asker's identity is necessary and still not enough once the bot answers in a shared channel. Draw the guest line at whose credential a guest can reach, not at how sensitive the service sounds. An integration that runs on the requester's own grant exposes only what that person can see. Anything backed by a workspace or shared operator credential is team-only, and a guest gets it only through tools you explicitly allowlist, such as public help-center search. For tools that read HR or finance data, return the result to the asker alone, as a private message, and never post it in the channel. A separate check should decide whether content from one conversation may appear in another: show private material only where everyone present could already read the source, and withhold even public material when any reader is a guest or from another organization (the details are in our RAG access control guide).

For your own deployment, list the audiences a reply can reach (the asker, a channel, a shared transcript, an exported log) and decide per tool which audience is allowed. The HR question "what is the CEO's salary" is solved by an API that enforces row-level security and returns nothing. The harder case is the allowed question asked in the wrong room.

Make every new tool pick a class

Security reviews cover the tools that exist on review day. The next tool someone adds ships with whatever defaults the framework gives it. A test can close that gap: require every registered tool to appear in an explicit write, read-only, approval-required or approval-exempt classification, so a new tool cannot ship with an accidental default. Let admins switch services to read-only or turn off a single tool, and run that check right before the tool executes, so scheduled turns are covered as well.

Treat the transcript store as a database

Chat logs, memory and traces concentrate data from many permission domains in one place. Decide what you need before you capture it. Storing turn metadata (outcome, timing, tool-call IDs) without request or reply text is possible when the source system keeps the conversation and can be read live. Samsung's 2023 ban on staff use of generative AI followed engineers uploading internal source code to ChatGPT, with the company citing that data sent to such services is stored on external servers and is hard to retrieve and delete (Bloomberg via Yahoo Finance). Where text must be kept, redact secrets and card numbers by key name and pattern before it reaches logs or issue reports, set a retention period, and make deletion work. If employees paste sensitive text into the bot, a redaction layer helps, but treat it as best-effort: pattern and NER redaction miss things, so also restrict where the provider may retain or train on inputs under your contract.

The basics that still apply

• Pass the user's token to downstream APIs instead of using a broad service account, so the bot cannot be a confused deputy.
• Filter retrieved chunks by the caller's ACL before they enter the prompt, and strip hidden text and zero-width characters before indexing.
• Require explicit approval for writes, sends and deletes. For how to bind that approval to the exact action, see binding human approval to AI agent actions.
• Use private endpoints or a private network for model traffic, and tie the bot's access to your identity provider so offboarding removes it.
• Send audit logs to your SIEM and alert on bursts of document reads or repeated injection attempts.
• Check data residency and health-data agreements with your provider before processing regulated data.

A one-week audit

Day one, list every tool and its credential: whose is it, and could a guest or contractor reach it. Day two, read your rendering code for anything that fetches a URL from model output. Day three, test with two users and one restricted document in a shared room. Day four, add the unclassified-tool test. Day five, write down what you store from each conversation and why. Fix what the week finds before adding another connector.

References

• Hack The Box: Inside CVE-2025-32711 (EchoLeak)
• The Attacker Moves Second (arXiv 2510.09023)
• Bloomberg (via Yahoo Finance): Samsung bans staff's AI use after spotting ChatGPT data leak

---