AI

Stability AI, gunning for a hit, launches an AI-powered music generator

Comment

"play" icon neon sign
Image Credits: Alina Kurianova / Getty Images

A year ago, Stability AI, the London-based startup behind the open source image-generating AI model Stable Diffusion, quietly released Dance Diffusion, a model that can generate songs and sound effects given a text description of the songs and sound effects in question.

Dance Diffusion was Stability AI’s first foray into generative audio, and it signaled a meaningful investment — and acute interest, seemingly — from the company in the nascent field of AI music creation tools. But for nearly a year after Dance Diffusion was announced, all seemed quiet on the generative audio front — at least as far as it concerned Stability’s efforts.

The research organization Stability funded to create the model, Harmonai, stopped updating Dance Diffusion sometime last year. (Historically, Stability has provided resources and compute to outside groups rather than build models entirely in-house.) And Dance Diffusion never gained a more polished release; even today, installing it requires working directly with the source code, as there’s no user interface to speak of.

Now, under pressure from investors to translate over $100 million in capital into revenue-generated products, Stability is recommitting to audio in a big way.

Today marks the release of Stable Audio, a tool that Stability claims is the first capable of creating “high-quality,” 44.1 kHz music for commercial use via a technique called latent diffusion. Trained on audio metadata as well as audio files’ durations — and start times — Stability says that Audio Diffusion’s underlying, roughly 1.2-billion-parameter model affords greater control over the content and length of synthesized audio than the generative music tools released before it.

Stability AI is on a mission to unlock humanity’s potential by building foundational AI models across a number of content types or ‘modalities,’” Ed Newton-Rex, VP of audio for Stability AI, told TechCrunch in an email interview. “We started with Stable Diffusion and have grown to include languages, code and now music. We believe the future of generative AI is multimodality.”

Stable Audio wasn’t developed by Harmonai — or, rather, it wasn’t developed by Harmonai alone. Stability’s audio team, formalized in April, created a new model inspired by Dance Diffusion to underpin Stable Audio, which Harmonai then trained.

Harmonai now serves as Stability’s AI music research arm, Newton-Rex, who joined Stability last year after tenures at TikTok and Snap, tells me.

“Dance Diffusion generated short, random audio clips from a limited sound palette, and the user had to fine-tune the model themselves if they wanted any control. Stable Audio can generate longer audio, and the user can guide generation using a text prompt and by setting the desired duration,” Newton-Rex said. “Some prompts work fantastically, like EDM and more beat-driven music, as well as ambient music, and some generate audio that’s a bit more ‘out there,’ like more melodic music, classical and jazz.”

Stability turned down our repeated requests to try Stable Audio ahead of its launch. For now, and perhaps in perpetuity, Stable Audio can only be used through a web app, which wasn’t live until this morning. In a move that’s sure to irk supporters of its open research mission, Stability hasn’t announced plans to release the model behind Stable Audio in open source.

But Stability was amenable to sending samples showcasing what the model can accomplish across a range of genres, mainly EDM, given brief prompts.

While they very well could’ve been cherry picked, the samples sound — at least to this reporter’s ears — more coherent, melodic and for lack of a better word musical than many of the “songs” from the audio generation models released so far. (See Meta’s AudioGen and MusicGen, Riffusion, OpenAI’s Jukebox, Google’s MusicLM and so on.) Are they perfect? Clearly not — they’re lacking in creativity, for one. But if I heard the ambient techno track below playing in a hotel lobby somewhere, I probably wouldn’t assume AI was the creator.

As with generative image, speech and video tools, yielding the best output from Stable Audio requires engineering a prompt that captures the nuances of the song you’re attempting to generate — including the genre and tempo, prominent instruments and even the feelings or emotions the song evokes.

For the techno track, Stability tells me they used the prompt “Ambient Techno, meditation, Scandinavian Forest, 808 drum machine, 808 kick, claps, shaker, synthesizer, synth bass, Synth Drones, beautiful, peaceful, Ethereal, Natural, 122 BPM, Instrumental”; for the track below, “Trance, Ibiza, Beach, Sun, 4 AM, Progressive, Synthesizer, 909, Dramatic Chords, Choir, Euphoric, Nostalgic, Dynamic, Flowing.”

And this sample was generated with “Disco, Driving, Drum, Machine, Synthesizer, Bass, Piano, Guitars, Instrumental, Clubby, Euphoric, Chicago, New York, 115 BPM”:

For comparison, I ran the prompt above through MusicLM via Google’s AI Test Kitchen app on the web. The result wasn’t bad necessarily. But MusicLM interpreted the prompt in a very obviously repetitive, reductive way:

One of the most striking things about the songs that Stable Audio produces is the length up to which they’re coherent — about 90 seconds. Other AI models generate long songs. But often, beyond a short duration — a few seconds at the most — they devolve into random, discordant noise.

The secret is the aforementioned latent diffusion, a technique similar to that used by Stable Diffusion to generate images. The model powering Stable Audio learns how to gradually subtract noise from a starting song made almost entirely of noise, moving it closer — slowly but surely, step by step — to the text description.

It’s not just songs that Stable Audio can generate. The tool can replicate the sound of a car passing by, or of a drum solo.

Here’s the car:

And the drum solo:

Stable Audio is far from the first model to leverage latent diffusion in music generation, it’s worth pointing out. But it’s one of the more polished in terms of musicality — and fidelity.

To train Stable Audio, Stability AI partnered with the commercial music library AudioSparx, which supplied a collection of songs — around 800,0000 in total — from its catalog of largely independent artists. Steps were taken to filter out vocal tracks, according to Newton-Rex — presumably over the potential ethical and copyright quandries around “deepfaked” vocals.

Somewhat surprisingly, Stability isn’t filtering out prompts that could land it in legal crosshairs. While tools like Google’s MusicLM throw an error message if you type something like “along the lines of Barry Manilow,” Stable Audio doesn’t — at least not now.

When asked point blank if someone could use Stable Audio to generate songs in the style of popular artists like Harry Styles or The Eagles, Newton-Rex said that the tool’s limited by the music in its training data, which doesn’t include music from major labels. That may be so. But a cursory search of AudioSparx’s library turns up thousands of songs that themselves are “in the style of” artists like The Beatles, AC/DC and so on, which seems like a loophole to me.

Stable Audio is designed primarily to generate instrumental music, so misinformation and vocal deepfakes aren’t likely to be an issue,” Newton-Rex said. “In general, however, we’re actively working to combat emerging risks in AI by implementing content authenticity standards and watermarking in our imaging models so that users and platforms can identify AI-assisted content generated through our hosted services … We plan to implement labeling of this nature in our audio models too.”

Increasingly, homemade tracks that use generative AI to conjure familiar sounds that can be passed off as authentic, or at least close enough, have been going viral. Just last month, a Discord community dedicated to generative audio released an entire album using an AI-generated copy of Travis Scott’s voice — attracting the wrath of the label representing him.

Music labels have been quick to flag AI-generated tracks to streaming partners like Spotify and SoundCloud, citing intellectual property concerns — and they’ve generally been victorious. But there’s still a lack of clarity on whether “deepfake” music violates the copyright of artists, labels and other rights holders.

And unfortunately for artists, it’ll be a while before clarity arrives. A federal judge ruled last month that AI-generated art can’t be copyrighted. But the U.S. Copyright Office hasn’t taken a firm stance yet, only recently beginning to seek public input on copyright issues as they relate to AI.

Stability takes the view that Stable Audio users can monetize — but not necessarily copyright — their works, which is a step short of what other generative AI vendors have proposed. Last week, Microsoft announced that it would extend indemnification to protect commercial customers of its AI tools when they’re sued for copyright infringement based on the tools’ outputs.

Stability AI customers who pay $11.99 per month for the Pro tier of Stable Audio can generate 500 commercializable tracks up to 90 seconds long monthly. Free tier users are limited to 20 non-commercializable tracks at 20 seconds long per month. And users who wish to use AI-generated music from Stable Audio in apps, software or websites with more than 100,000 monthly active users have to sign up for an enterprise plan.

In the Stable Audio terms of service agreement, Stability makes it clear that it reserves the right to use both customers’ prompts and songs, as well as data like their activity on the tool, for a range of purposes, including developing future models and services. Customers agree to indemnify Stability in the event intellectual property claims are made against songs created with Stable Audio.

But, you might be wondering, will the creators of the audio on which Stable Audio was trained see even a small portion of that monthly fee? After all, Stability, as have several of its generative AI rivals, has landed itself in hot water over training models on artists’ work without compensating or informing them.

As with Stability’s more recent image-generating models, Stable Audio does have an opt-out mechanism — although the onus for the most part lies on AudioSparx. Artists had the option to remove their work from the training dataset for the initial release of Stable Audio, and about 10% chose to do so, according to AudioSparx EVP Lee Johnson.

“We support our artists’ decision to participate or not, and we’re happy to provide them with this flexibility,” Johnson said via email.

Stability’s deal with AudioSparx covers revenue sharing between the two companies, with AudioSparx letting musicians on the platform share in the profits generated by Stable Audio if they opted to participate in the initial training or decide to help train future versions of Stable Audio. It’s similar to the model being pursued by Adobe and Shutterstock with their generative AI tools, but Stability wasn’t forthcoming on the particulars of the deal, leaving unsaid how much artists can expect to be paid for their contributions.

Artists have reason to be wary, given Stability CEO Emad Mostaque’s propensity for exaggeration, dubious claims and outright mismanagement.

In April, Semafor reported that Stability AI was burning through cash, spurring an executive hunt to ramp up sales. According to Forbes, the company has repeatedly delayed or outright not paid wages and payroll taxes, leading AWS — which Stability uses for compute to train its models — to threaten to revoke Stability’s access to its GPU instances.

Stability AI recently raised $25 million through a convertible note (i.e. debt that converts to equity), bringing its total raised to over $125 million. But it hasn’t closed new funding at a higher valuation; the startup was last valued at $1 billion. Stability was said to be seeking quadruple that within the next few months, despite stubbornly low revenues and a high burn rate.

Will Stable Audio turn the company’s fortunes around? Maybe. But considering the hurdles Stability has to clear, it’s safe to say it’s a bit of a long shot.

More TechCrunch

Apple has published a technical paper detailing the models that it developed to power Apple Intelligence, the range of generative AI features headed to iOS, macOS and iPadOS over the…

Apple says it took a ‘responsible’ approach to training its Apple Intelligence models

A fireside chat on Monday between Nvidia CEO Jensen Huang and Meta CEO Mark Zuckerberg at the SIGGRAPH 2024 conference in Colorado took a few unexpected turns. It started innocently…

Huang and Zuckerberg swapped jackets at SIGGRAPH 2024 and things got weird

Meta’s machine learning model, Segment Anything, has a sequel: It now takes the model to the video domain, showing how fast the field is moving.

Zuckerberg touts Meta’s latest video vision AI with Nvidia CEO Jensen Huang

Featured Article

The fall of EV startup Fisker: A comprehensive timeline

Here is a timeline of the events that led fledgling automaker Fisker to file for bankruptcy.

The fall of EV startup Fisker: A comprehensive timeline

Hello, and welcome back to TechCrunch Space. In case you missed it, Boeing and NASA decided to keep Starliner docked to the International Space Station for the rest of the…

TechCrunch Space: Catching stars

As failed EV startup Fisker winds its way through bankruptcy, a persistent and tricky question has become a flashpoint of the proceedings: does its only secured lender, Heights Capital Management,…

The question haunting Fisker’s bankruptcy

So-called “unlearning” techniques are used to make a generative AI model forget specific and undesirable info it picked up from training data, like sensitive private data or copyrighted material. But…

Making AI models ‘forget’ undesirable data hurts their performance

Uber is now letting riders in India book up to three rides simultaneously.

Uber now lets users in India book three trips at once

U.S. airports are rolling out facial recognition to scan travelers’ faces before boarding their flights. Americans, at least, can opt out. 

How to opt out of facial recognition at airports (if you’re American)

The promise of AI and large language models (LLMs) is the ability to understand increasingly wider amounts of context and make sense of that information easily, so it makes sense…

Bee AI raises $7M for its wearable AI assistant that learns from your conversations

Featured Article

DEI backlash: Stay up-to-date on the latest legal and corporate challenges

It’s clear that this year will be a turning point for DEI.

DEI backlash: Stay up-to-date on the latest legal and corporate challenges

Bike-taxi startup Rapido, which counts Swiggy among its investors, is the latest Indian firm to become a unicorn.

India’s Rapido becomes a unicorn with fresh $120M funding

Government websites aren’t known for cutting-edge tech. GovWell co-founder and CTO Ben Cohen discovered this while trying to help his dad, a contractor, apply for building permits. Cohen worked as…

GovWell is bringing automation and efficiency to local governments

Critics have long argued that wararantless device searches at the U.S. border are unconstitutional and violate the Fourth Amendment.

US border agents must get warrant before cell phone searches, federal court rules

Featured Article

UK’s Zapp EV plans to expand globally with an early start in India

Zapp is launching its urban electric two-wheeler in India in 2025 as it plans to expand globally.

UK’s Zapp EV plans to expand globally with an early start in India

The first time I saw Google’s latest commercial, I wondered, “Is it just me, or is this kind of bad?” By the fourth or fifth time I saw it, I’d…

Dear Google, who wants an AI-written fan letter?

Featured Article

MatPat, the first big YouTuber to successfully exit his company, is lobbying for creators on Capitol Hill

Though MatPat retired from YouTube, he’s still pretty busy. In fact, he’s been spending a lot of time on Capitol Hill.

MatPat, the first big YouTuber to successfully exit his company, is lobbying for creators on Capitol Hill

Featured Article

A tale of two foldables

Samsung is still foldables’ 500-pound gorilla, but the company successes have made the category significantly less lonely in recent years.

A tale of two foldables

The California Department of Motor Vehicles this week granted Nuro approval to test its third-generation R3 autonomous delivery vehicle in four Bay Area cities, giving the AV startup a positive…

Autonomous delivery startup Nuro is gearing up for a comeback

With Ghostery turning 15 years old this month, TechCrunch caught up with CEO Jean-Paul Schmetz to discuss the company’s strategy and the state of ad tracking.

Ghostery’s CEO says regulation won’t save us from ad trackers

Two years ago, workers at an Apple Store in Towson, Maryland, were the first to establish a formally recognized union at an Apple retail store in the United States. Now…

Apple reaches its first contract agreement with a US retail union

OpenAI is testing SearchGPT, a new AI search experience to compete directly with Google. The feature aims to elevate search queries with “timely answers” from across the internet and allows…

OpenAI comes for Google with SearchGPT

Indian cryptocurrency exchange WazirX announced on Saturday a controversial plan to “socialize” the $230 million loss from its recent security breach among all its customers, a move that has sent…

WazirX to ‘socialize’ $230M security breach loss among customers

Featured Article

Stay up-to-date on the amount of venture dollars going to underrepresented founders

Stay up-to-date on the latest funding news for Black and women founders.

Stay up-to-date on the amount of venture dollars going to underrepresented founders

The National Institute of Standards and Technology (NIST), the U.S. Commerce Department agency that develops and tests tech for the U.S. government, companies and the broader public, has re-released a…

NIST releases a tool for testing AI model risk

Featured Article

Max Space reinvents expandable habitats with a 17th-century twist, launching in 2026

Max Space’s expandable habitats promise to be larger, stronger, and more versatile than anything like them ever launched, not to mention cheaper and lighter by far than a solid, machined structure.

Max Space reinvents expandable habitats with a 17th-century twist, launching in 2026

Payments giant Stripe has acquired a four-year-old competitor, Lemon Squeezy, the latter company announced Friday. Terms of the deal were not disclosed. As a merchant of record, Lemon Squeezy calculates…

Stripe acquires payment processing startup Lemon Squeezy

iCloud Private Relay has not been working for some Apple users across major markets, including the U.S., Europe, India and Japan.

Apple reports iCloud Private Relay global outages for some users

Welcome to Startups Weekly — your weekly recap of everything you can’t miss from the world of startups. To get Startups Weekly in your inbox every Friday, sign up here. This…

Legal tech, VC brawls and saying no to big offers

Apple joins 15 other tech companies — including Google, Meta, Microsoft and OpenAI — that committed to the White House’s rules for developing generative AI.

Apple signs the White House’s commitment to AI safety