SF Orders Apple & Google to Pull AI Nudify Apps from Stores

What happened

San Francisco's City Attorney has issued formal legal demands to Apple and Google, ordering both companies to remove AI-powered "nudify" applications from their respective app stores. The directive, reported by Ars Technica and Wired, targets a category of tools that use generative AI to produce non-consensual intimate imagery (NCII) — digitally stripping clothing from photographs of real people without their knowledge or permission.

The City Attorney's office contends that by hosting and distributing these applications, Apple and Google have been active participants in facilitating harm, not merely passive platforms. Crucially, both companies collect a standard percentage cut of in-app revenue, meaning they have financially benefited from transactions conducted through nudify tools. Official estimates cited in the reporting suggest the two tech giants likely generated millions of dollars in fees from these applications.

The legal action stops short of a court filing for now, but the demand letters carry significant weight: San Francisco has a track record of pursuing tech companies through litigation when voluntary compliance fails. Both Apple and Google have existing policies that nominally prohibit apps designed to harass or generate NCII, making the City Attorney's core argument that enforcement of their own rules has been inconsistent at best.

Why it matters

For customer experience and service-design professionals, this case crystallises a question that platform businesses can no longer defer: what is the operator's moral and legal responsibility for the experiences — including harmful ones — that their ecosystem enables? App stores are, at their core, curated service environments. Apple in particular has built its entire brand proposition around the promise of a safe, trusted marketplace. When harmful products persist inside that environment, the trust architecture that underpins the customer relationship is compromised, regardless of whether the platform wrote the offending code.

From a behavioural economics perspective, the revenue-sharing model creates a structural conflict of interest — a textbook misalignment of incentives. Platforms profit from volume and engagement, which can blunt the urgency of policing edge cases until external pressure, regulatory or reputational, forces the issue. San Francisco's intervention is a signal that regulators are increasingly willing to treat platform fees as evidence of complicity, not merely association. CX leaders inside platform businesses should read this as a prompt to audit their own ecosystems for analogous blind spots before a city attorney does it for them.

By the numbers

• Millions of dollars in app-store fees estimated to have been collected by Apple and Google from nudify application revenue, according to the San Francisco City Attorney's office.
• Two of the world's largest app-store operators — Apple's App Store and Google Play — named in the formal legal demand.

The Renascence take

The instinct in platform governance is to treat harmful-content removal as a moderation problem — a content team's remit, handled reactively. San Francisco's framing rejects that entirely, and rightly so. The more instructive lens here is service design: if you design a distribution system that shares revenue with suppliers, you have designed yourself into accountability for what those suppliers deliver.

Most operators will read this story as a tech-policy matter and move on. They shouldn't. The underlying principle — that revenue participation equals responsibility participation — applies to any platform, marketplace or franchise model that takes a cut of third-party transactions. The behavioral trap is the diffusion of responsibility: each internal team assumes someone else is watching the ecosystem. Customer-obsessed operators should instead assign explicit ownership of ecosystem harm, tie it to the same commercial metrics as growth, and audit their supplier or developer bases with the same rigour they apply to their own products. Waiting for a regulator to define your duty of care is not a strategy; it is a liability.