A comprehensive security audit has exposed significant vulnerabilities in the artificial intelligence supply chain, revealing that sensitive user data is frequently intercepted and stored by third-party services. The findings come from an experiment involving 2 million digital canary tokens planted in prompts sent through a sprawling network of AI services. The test demonstrates that data entrusted to these platforms does not remain private, creating a privacy disaster for users who believe their inputs are secure.
AI Router Security Failures
The investigation, led by HD Moore, chief research officer at security firm Dragos, targeted the complex ecosystem of “AI routers” or “AI gateways.” These services act as brokers for developers, receiving prompts from end users and routing them to the most suitable inference engine based on criteria such as cost or performance. Instead of sending data directly to major providers like OpenAI or Anthropic, developers often hire these routers to manage the process.
Moore assembled a collection of unique, trackable secrets, including AWS access keys tied to empty accounts, SSH passwords, and private git repository references. None of these credentials had ever been used before. He seeded each secret into exactly one prompt sent to a single provider. If any secret was leaked, the canary tokens would allow researchers to identify the specific link in the chain responsible for the breach.
The results showed that while prompts are encrypted in transit, they are decrypted at each hop within the inference reseller network. The AI router reads the plaintext to determine the best provider, often sending the prompt to a pool of available models that may run on different hosting infrastructure. Each player in this chain is likely to log the prompt and store it indefinitely, depending on their policies.
Hosting Provider Breaches and Scraping
The most alarming findings involved the compromise of three separate hosting providers. Attackers who breached one provider reused a couple hundred of the AWS keys planted in the test. Moore noted that some stolen keys came from requests that the provider rejected before a model ever ran, indicating the attacker was copying requests at the provider’s front door rather than deeper in the stack.
In the other two breaches, stolen requests were marked with a Zero Data Retention (ZDR) flag, which failed to protect the data once the attacker was inside the system. The parties behind the hack tested each key within hours of it being seeded, searching for additional credentials in S3 buckets and secrets managers. They also attempted to enable hosted models in AWS Bedrock and added their own IAM users to maintain access. The stolen bait keys were used tens of thousands of times in AWS API calls.

Secrets sent to two of the hacked providers were quickly used together from the same rented server, suggesting the same adversary compromised both. Additionally, the experiment revealed that some providers scrape secret URLs put into prompts, violating privacy terms. A Chinese AI lab was found fetching secret URLs from more than half of the prompts sent to its models between July and October, including roughly two-thirds of requests marked as no data collection.
This scraping was not limited to direct requests. Several US hosting providers resell this lab’s models, and prompts sent to those providers triggered the exact same scraper. A customer who chose a US provider to keep their data in the United States did not receive the promised privacy. Later, a git client on a different cloud attempted to clone private repositories referenced in prompts, but only for those using this specific lab’s models.
Moore identified resellers who offered low prices to lure prompts to their servers, then turned on request logging to capture data. When credentials were reused when sent through one seller but not another, it confirmed the reseller was responsible for the leak. One US hosting provider was found crawling a secret URL more than two months after it was used in a single inference call.
The researcher concluded that the most effective way to prevent leakage is to host models using on-premises servers, though this option is expensive. For those who cannot afford self-hosted systems, Moore recommends receiving service directly from a first-tier provider that offers a single contract and privacy policy. He also advised users to avoid accepting default provider or reseller pools and to pick upstream providers themselves, noting that even direct contracts offer no guarantee of total data protection.
Source: Ars Technica

