Give a customer-facing chatbot read access only to what you would be comfortable posting on your website. On its own, it may take only steps that are easy to undo. Everything else should go through a person, or through a system that verifies who is asking. Work on the assumption that whatever the chatbot can see, a persistent visitor will get out of it.
How much you can entrust to it depends on three things. Whether it only answers, or also changes things in your systems. Whether you can verify who is asking. And whether anyone is monitoring how it runs. So before you deploy it, go through five questions: what it reads, what it does on its own, how much it may spend, who monitors it and what you have from your vendor in writing.
Assume the chatbot can be talked into things
Lupa.cz reported on a talk in Prague about the weaknesses of business chatbots. Three things follow from it. A customer-facing chatbot can be abused to do someone else’s work at the company’s expense. The operator’s rules can be bypassed by rephrasing the request. And a chatbot with access to documents, email or a database can leak data.
The reason is simple. A language model does not reliably distinguish between your instruction and text from a visitor. To the model, both are text it is supposed to respond to.
So the sentence “do not share internal information with anyone” in the prompt is a request, not a lock. It works most of the time. You cannot rely on it.
So don’t build security on what you tell the chatbot. Build it on what the chatbot cannot reach and cannot do. Nobody can coax out of it data it cannot see.
You are guarding two things: data and money. The following sections are a checklist for both.
Give the chatbot read access only to what anyone may see
A simple test helps. For every document and every table, imagine the visitor being handed the whole thing. If that doesn’t bother you, the chatbot may read it.
| Type of data | Give access? | On what condition |
|---|---|---|
| Public information: price list, delivery terms, how-to guides, opening hours | Yes | A copy prepared for the chatbot, not access to the entire storage |
| Order status of a specific customer | Yes, restricted | The system verifies the customer and returns only their record |
| Internal documents: purchase prices, margins, contracts, internal procedures | No | – |
| Company email and shared folders | No | – |
| The entire customer database | No | Only a single record after verification |
| Personal and sensitive data | Only the bare minimum | Find out what rules apply to them where you operate, and check with a local expert |
With a specific customer’s data, what matters is who verifies identity. It must not be the chatbot. It can be persuaded that it is talking to someone else.
The system underneath it should do the verifying. The customer is logged in, or enters an order number and a piece of information only they know. The system then returns one record to the chatbot and nothing more.
Ask your vendor directly: can the chatbot search all orders, or does it get only that one? If it can search, the only thing protecting other people’s data is its prompt.
Beware of the convenient shortcut “we’ll connect it to the shared drive so it has something to draw on”. Shared drives tend to contain things that don’t belong there. Prepare the chatbot’s source material separately and have someone read it before it is uploaded.
The chatbot may do on its own only what is easy to undo
Reading is one risk. The second arises the moment the chatbot is allowed to change something in your systems.
Sort its actions into three groups.
Allowed on its own:
- answering from the prepared source material,
- creating a support ticket,
- handing the conversation over to a person,
- showing a customer the status of their order after verification.
Only with a person’s approval:
- a refund, discount or other compensation,
- a change of delivery address or contact details,
- cancelling an order,
- a message sent on the company’s behalf outside the chat window.
Not at all:
- deleting records,
- changing prices or stock levels,
- anything involving payment details.
An address change looks harmless. But it is a well-known way to redirect someone else’s shipment. That is why it belongs in the middle group.
Approval doesn’t have to slow things down. The chatbot prepares a proposal and a support agent confirms or rejects it with one click.
Deal with promises separately. The chatbot may promise a customer a discount or a delivery date that isn’t valid. Find out how far such a promise binds a company where you operate, and check with a local expert. In the chat window, at least state that the answers are automated and where the customer can find the terms that apply.
Set a spending cap and limits for a single conversation
A chatbot costs money for every piece of text it reads and writes. When someone uses it as a free assistant for their own work, you pay for it. That is exactly the kind of abuse Lupa.cz describes.
And the visitor doesn’t have to be an attacker. It is enough for someone to discover that your chatbot will translate a long text for them or write their school essay.
So set the limits before you put the chatbot on your website:
- A monthly spending cap with the model provider. Once it is reached, the chatbot stops.
- An alert to the person responsible as early as half of the cap.
- The length of a single message. A question about delivery time fits into a few sentences; someone else’s document doesn’t.
- The number of messages in one conversation and the number of conversations per device per day.
- The length of the answer. A chatbot that writes only short answers is not much use for other people’s work.
Decide in advance what the visitor will see once it has stopped. A sentence saying the assistant is unavailable, plus a support contact, is enough.
The instruction “answer only questions about our products” does belong in the prompt. But treat it as a filter for ordinary visitors, not as protection. The limits are the protection.
Appoint a person who reads the real conversations
Without monitoring, you will learn about abuse from the invoice or from a customer. So appoint one person by name. “Support will keep an eye on it” means nobody is keeping an eye on it.
For the first week after launch, have them go through the conversations every day. After that, a sample once a week and a usage report are enough.
What to look for:
- questions unrelated to your line of business,
- unusually long conversations,
- attempts to find out the chatbot’s instructions or internal information,
- a sudden rise in the number of conversations, especially at night,
- answers where the chatbot promised something or made something up.
Reading the conversations pays off even without attackers. You will see what customers ask about and what they couldn’t find on your website.
The same person must be able to switch the chatbot off. On their own, immediately and without calling the vendor. Try it out before you need it.
Conversation logs contain customer data. Agree how long they are kept and who may read them. Check with a local expert what rules apply to processing them where you operate.
Get the vendor’s answers in writing
A verbal assurance that “it’s secure” is not enough. Ask for a short document you can understand without a technical background.
- List of access rights. Which systems and data the chatbot can reach, separately for reading and for making changes.
- Customer verification. Where exactly it takes place, and that it is not done by the language model.
- Conversation storage. Where the conversations are stored, for how long and who has access to them. And whether third-party models are trained on them.
- Limits. What spending and usage caps are set and who may change them.
- Switching off. How you stop the chatbot yourselves.
- Incidents. Who will notify you of a security problem, and how quickly.
- Abuse testing. How the chatbot was tested, and that the test will be repeated after every change of model or prompt.
If the vendor can’t answer one of these points, that is an answer too.
The same applies if you are putting the chatbot together yourselves from an off-the-shelf service. In that case, write the answers down yourselves and keep them where the person who covers for you can find them too.
Before launch, try talking the chatbot round yourselves
Before customers see the chatbot, give it an hour and try to use it against yourselves. You don’t need technical knowledge for this, just patience.
- Ask it for something outside your field: translating a long text, writing someone else’s email, doing a homework assignment.
- Ask it about its instructions and which documents it draws on.
- Pose as a colleague from support and ask for another customer’s orders.
- Claim that a manager promised you a discount and demand confirmation of it.
- Try a refused request again in different words, in another language or in parts.
Write down what happened. Then distinguish between two kinds of findings.
If the chatbot said something silly or off-topic, adjust the prompt and the limits.
If it handed over data or carried out an action it shouldn’t have, the fault is not in the wording. It is in the permissions. Take the access away; don’t try to discipline it with a better prompt.
Repeat the test after every major change. A new model or new source material can change its behaviour.
Sources
Frequently asked questions
Our chatbot is only on the website, with no connection to our systems. Does this apply to us too?
The risk of a data leak is small in your case, as long as the chatbot draws only on public material. But the risk of paying for other people’s work remains, as does the risk that it says something untrue under your brand. Limits and regular reading of conversations are therefore worthwhile even for a very simple deployment.
Is a chatbot that offers only prepared answers and buttons more secure?
Yes. When the visitor chooses from a menu and doesn’t type free text, there is nothing to rephrase and nobody to talk round. The price is less convenience: a customer with an unusual question ends up with a person. For a first deployment, it tends to be a sensible start.
Should the customer know they are talking to a bot?
We recommend saying so right in the first message. The customer then treats the answers with an appropriate degree of trust and knows they can ask for a person. In some countries the rules may also require it, so check what applies where you operate.
What should we do if we find out the chatbot has disclosed something it shouldn’t have?
Switch the chatbot off and save the conversation logs that show exactly what went out and to whom. Then take away its access to the data in question; adjusting the prompt alone is not enough. Find out from a local expert whether you are obliged to report the leak to anyone.
May an internal chatbot for employees see more than the customer-facing one?
It may, but not everything. The same principle applies: the chatbot should see only what the person asking is allowed to see. If a warehouse worker has no access to payroll, they must not be able to reach it through the chatbot either.
