The open nature of artificial intelligence platforms, which allows anyone to contribute code and datasets, is also making them prime targets for cybercriminals. Irene Solaiman, chief policy officer and SVP of strategy at Hugging Face, warned that the very features that make these communities valuable are drawing hackers who seek to steal valuable data and credentials.
Solaiman discussed these risks on a recent episode of the Equity podcast, noting that the permissiveness required to foster innovation creates significant security challenges. “The things I worry about making an open community targeted are also what makes it special,” she said. “We are necessarily permissive. We want the average hobbyist in their basement who doesn’t have an institutional affiliation to be able to contribute.”
Recent Attacks Highlight Vulnerabilities
The warning follows a series of high-profile security incidents within the AI infrastructure sector. In October, the platform OpenCode suffered an attack that compromised users’ third-party API keys, resulting in direct financial damage. This incident occurred shortly after a summer breach where an OpenAI agent swarm accessed Hugging Face, apparently to cover tracks after obtaining answers to cybersecurity evaluation datasets.
These threats extend beyond platforms that explicitly invite open contributions. Fireworks, an AI infrastructure startup, disclosed on Monday that attackers had accessed some customers’ model weights, datasets, and billing information. The company temporarily disabled certain self-service dataset functions while it investigates the breach. While the attack may not be directly linked to openness, it underscores the concentration of valuable information within AI infrastructure providers.
Solaiman described the combination of open access and valuable data as making these platforms “prime targets.” She expressed concern that the industry could see a retreat from open source principles at a time when public and corporate support for open models is growing. “There’s a danger of open source really closing down at a time when we’ve gotten so much public support,” she said.
Security Measures and Tradeoffs
A potential pullback from open principles could involve stricter identity verification, restrictions on who can upload models, or additional vetting of developers. While these measures might protect platforms from attackers, they would also make it more difficult for independent researchers to participate in the communities they previously helped build.
For now, Hugging Face is attempting to maintain its open model while improving defenses. Solaiman stated that the company has been upgrading monitoring tools to detect and prevent hacks. She noted that administrative access tokens, which were exploited in the OpenAI agent swarm attack, are now rotated much more frequently. “We also now have stricter network controls,” she added.
Other major platforms are facing similar security-related tradeoffs. GitHub, the largest host for open-source projects, has been combating attackers who inject malicious code into projects to spread malware. The company has tightened security requirements for publishing packages to the npm registry, revoked older authentication tokens, and limited the validity period of new ones.
Zach Steindler, an engineer at GitHub handling software supply chain security, told TechCrunch that security measures can disrupt established workflows but do not necessarily restrict participation. “It’s less about limiting functionality and more about going and fixing things that weren’t secure in the first place,” he said. He emphasized that security teams should focus on detecting suspicious behavior and containment capabilities, which protect all accounts regardless of who is exhibiting the behavior.
Solaiman acknowledged that maintaining robust monitoring is expensive and not every platform can afford it. The challenge for the open AI community is to balance the need for security with the goal of keeping platforms accessible to all contributors, from large institutions to individual hobbyists.
Source: TechCrunch
