Finding a cool vulnerability

Around three years ago I found a series of vulnerabilities in a chatbot integrated into a web application: following them led from accessing other users’ conversations to extracting their session tokens. I’m leaving out names and identifying details, but the interesting part is how all the pieces fit together to make it possible.

The chatbot used the same access token as the rest of the application. This made sense for sending a message: the user was already logged in, and the backend could use that token to identify them. What I found was that the same token also worked for operations that should have required completely different permissions.

I could list other users’ conversations, send messages to an active conversation by changing its identifier, or close it and receive a generated summary. That summary could also contain personal information shared during the conversation.

The server was using FastAPI with Pydantic, which meant that the error messages from the API were quite helpful: when a request was missing something, the validation errors described the required fields and their types. Useful feedback for a developer, and also for someone making requests they should never have been allowed to make.

Then performing some fuzzying on the API endpoints (swagger wasn’t exposed at least), I found an endpoint with the whole configuration for the chatbot: its system prompt, generation settings, and the rules applied to responses before they reached the frontend, such as a RegExp that replaces any string that looked like a phone number with the actual phone number, to prevent hallucinations. All readable with an ordinary user token. And with the same token it was also possible to PATCH or PUT the configuration.

This meant I could change how the chatbot behaved for everyone using it: changing the system prompt was an obvious possibility if I wanted to implement, say, phishing or similar nefarious behaviour, but I could also replace both the RegExp and what it would return when applied.

So I could choose a string that appeared frequently in the chatbot’s answers and replace it with arbitrary content, after generation, e.g. the name of the company owning the chatbot. The model could produce a perfectly ordinary answer, and the application would add my content to it.

That was quite dangerous on its own (replacing any email for support with one for phishing, for example), but I looked into the frontend to check if there were a couple additional vulnerabilities that would make this vulnerability way more dangerous.

And there they were: responses from the chatbot were rendered using React dangerouslySetInnerHTML, without sanitization sufficient to prevent the payload I tested. The content added by the replacement rule could therefore become active HTML inside the application.

The application also kept its access token in sessionStorage, not a HTTPonly cookie, so any JavaScript code running in that context could read it. Its Content Security Policy did not block the execution and outgoing request used in the proof of concept.

Putting those pieces together, I could modify a response replacement rule, have the chatbot deliver HTML that executed JavaScript, and send the current user’s token to a server I controlled. The user only needed to receive a response containing the string matched by that rule.

I verified the token extraction and documented the result: that token could then be used to impersonate the affected user within the access it granted; I didn’t need to go further to demonstrate the problem.

It was honestly really great from my point of view: finding those vulnerabilities one after the other, each one building up on top of the previous one, and each one necessary to build what I could build. Less so as a customer of that service, but it was fixed a few hours after I let them know, so hopefully no harm for anyone.