Our AI Apps Do More Than We Asked

First published on SGTech Community Sharing · AI Security.

In a study of 807,207 custom AI apps, with 199 tested in depth, nine in ten of the popular apps could be pushed into harmful tasks, seven in ten did jobs they were never built for, and a quarter to a third of tricky inputs made apps worse at their own job.

About Epic Connector. Epic Connector is a Singapore-based, AI-native founder community and incubator that helps AI startups build, launch and scale across borders. It works with leading AI security startups such as TrustAI and Infron AI, with leading AI research institutions including Tsinghua University and the National University of Defense Technology, and with Y Combinator, and is expanding its presence across Singapore, Tokyo and the United States.

Imagine a small bakery sets up an AI assistant on its website to take cake orders. A week later, a student is using the bakery's assistant to write homework. Someone else is using it as a free coding tool. Every one of those conversations runs on the bakery's account and carries the bakery's name.

When we talk about AI security, we usually talk about jailbreaks: tricking a model into saying something harmful. Our study looks at a wider and more everyday problem. An app is built for one job, but nothing firmly stops it from doing others, or from doing its own job badly when fed the wrong input. We call this a blurred capability boundary.

The findings in this article come from our February 2026 study, in collaboration with Yunyi Zhang, Shibo Cui, Baojun Liu and Jingkai Yu of Tsinghua University, Min Zhang and Fan Shi of the National University of Defense Technology, and Han Zheng of Singapore-based TrustAI Pte. Ltd., presented at NDSS 2026.

Building an AI app is mostly about saying no

In traditional software, we write every feature, so the app can only do what we coded. With an AI app built on a large model such as GPT, Claude or ERNIE, the model underneath can already do almost anything. Our real job as builders is to restrict it: to cut that ability down to the one task the app is for.

Upgrade and jailbreak both leave the app's job; the difference is whether the base model would itself refuse the task. Downgrade never leaves the job at all, which makes it the hardest to spot. Each hurts us differently. Upgrade costs us money, as strangers use our app on our bill. Jailbreak causes harm under our name. Downgrade breaks trust in what the product is for.

Who attacks, and what they get

The attacker we consider is simply a normal user. All they can do is type, or upload a file. Our study shows that is enough to get free AI on someone else's bill, get round restrictions, damage the builder's name, or, where the app makes decisions, steal money or advantage.

A translation feature used for everything else (real case, upgrade). In early 2025, when many US TikTok users moved to Rednote, the platform quickly added an AI translation feature. Weak limits meant users could push it well beyond translating.

An auditing bot that waves through theft (scenario, downgrade). In a Web3 setting, an AI bot checks transactions before they go through. Crafted input makes it judge a malicious operation as safe, and user assets are stolen.

A CV that grades itself (demonstrated, downgrade). We hid an instruction inside a CV telling an AI recruitment screener to declare the candidate the best it had seen. The model returned exactly that verdict.

How our study worked

We collected every app we could reach on four no-code platforms: OpenAI's GPTs Store (576,952 apps), ByteDance's Coze (187,115), Baidu's AgentBuilder (22,638) and Poe (20,502). We think the four questions below are the most reusable idea in our study:

1. Target: how clearly does it say what the app is for?
2. Process: does it lay out the steps the app should follow?
3. Capability: does it list what the app can do?
4. Constraint: does it say what the app must refuse, and how firmly?

What we found

Of the 11,176 AgentBuilder prompts, 43% set no constraints at all, and almost half scored below 50 out of 100. On GPTs Store, 19 developers had each published more than 1,000 apps, one of them 8,530.

Upgrade: 144 apps did tasks from 15 or more categories, including all 50 top GPTs. We trace this to platform defaults: GPTs come with web search and image generation switched on.

Jailbreak: 178 apps (89%) carried out harmful tasks using techniques from free tools. More worrying, 17 did so when simply asked, with no technique at all.

Downgrade: on all six open-source models, a quarter to a third of the misleading inputs produced a worse answer. None was immune.

Does a better prompt fix it?

We compared a real app with no limits against one with a well-designed prompt. The better prompt always helped, but by very different amounts: it cut out-of-scope tasks by anywhere from 5.3% to 80%, depending on the platform and base model.

The base model matters too. Identical constraints held more firmly on Baidu's ERNIE than on Claude-3-Haiku. So a prompt we tested on one model tells us little about another.

A clear prompt is the cheapest improvement we can make. It is not a lock: its effect ranged from 5% to 80%, and it depends on a model we may not control.

What we can do about it

When we build AI apps:

1. Write all four parts of the prompt. 43% of apps skipped constraints entirely.
2. Switch off every plugin we don't need.
3. Don't let the model have the last word. Strip hidden characters from uploaded documents and enforce prices, approvals and rankings in ordinary code.
4. Attack our own apps, and again when the model changes.
5. Watch usage and set spending limits.

When we use AI apps built by others: share with it only what we would share with any general AI system. If a single-purpose app answers anything we ask, the builder probably never set limits.

When we run a platform or startup programme: defaults matter more than guidance. Plugins off by default and a starter prompt with constraints already in it would change thousands of apps at once.

Inside one founder stack: Epic Connector

Epic Connector gives its cohort founders a shared technical kit: compute through Nvidia, payments through Bagel Pay, and model access through Infron AI. We focus on the model-access layer, because every prompt and every answer from a founder's app passes through it.

Infron is a model gateway. An app sends requests to one OpenAI- or Anthropic-compatible endpoint, and the gateway forwards each one to one of 300+ models across many providers. Because it sits on the path of every request, its settings decide three things our study shows matter: which model actually answers, what happens to the data, and who pays for misuse.

A gateway only sees text going to and from the model. It does not see the plugins an app connects, the documents it reads, or what it does with the answer. Default plugins, hidden text in documents and apps that let the model decide all sit in that blind spot, and remain our job as builders.

Conclusion

An app's instructions are a request to the model, not a lock. Most apps we studied never wrote limits, most of those that did could be talked past them, and even a well-built app can be misled by what it reads. Writing a clear prompt is the first step, but only the first.

---

Original article: SGTech Community Sharing · AI Security. Research: Zhang et al., NDSS 2026. Prepared by the AI Workgroup, SGTech CSC Chapter.