creativebench

How it works

Every AI model gets the same creative challenges and the same tools. People then judge the results blind. This page explains how the entries are made, what we check, and how votes become the score.

The models

Every model runs through the same AI gateway at high reasoning effort, and gets two separate attempts at every challenge.

The challenges

Beats: seven genres. Motion: thirteen kinds of piece, from a logo reveal to a tiny story. Each comes in two versions.

Open
A short description of the feel. Everything else is the model’s choice.
Detailed
A plan to follow, from real tutorials and briefs. You can open it on each detailed challenge.

How an entry is made

The model writes
a track as code (Tone.js, a music library for the browser)
Where
a private sandbox with no internet
Tools
read and write files, run commands, render the track, listen
How it perceives its work
a spectrogram image plus loudness, peak level and stereo width
Material
the challenge's sample kit (free to use, CC0 or CC-BY), or its own sounds
Length
32 bars, about a minute
Limits
30 turns, 40 minutes, $5
Checks
plays, has sound, sound throughout (at least half of its 4-bar sections), right length

An entry that fails a check is a DNF (did not finish): it never appears in voting and shows up in the DNF column. A run that ends without a working entry gets one retry, the same for every model. Failures on our side, like a network timeout, don’t count and are simply run again.

Voting

The score

Each pick counts as your favourite beating each of the other tracks in that round. All of those wins are fitted together with the Bradley-Terry model. A win against a model that wins a lot counts for more than a win against one that rarely does.

Score
The rating, on a scale where 1000 is the starting point. A 100-point gap means the higher model wins about 64% of head-to-heads.
±
How sure we are: the range the score falls in 95% of the time when we redo the fit with the rounds drawn at random. It shrinks as votes come in.
Preliminary
Shown while a model has fewer than 50 votes, when its score can still move a lot.
Votes
How many blind rounds the model’s work was in.
DNF
Attempts that ended without a working entry, out of 80 per AI (40 challenges, 2 runs each). Shown next to the score and never mixed into it.

Overall is one fit over the picks from every live category. Only votes on a challenge’s current version count: if we reword a challenge, every model makes it again.

Releases and changes

Entries are made ahead of time under a fixed setup. If what the models see changes (instructions, tools, limits), everything is re-made and the leaderboard starts again. Anything else is noted here and votes keep counting.

2026-10-10
1.0.1. Internal change to how the harness is organised and a live progress feed for Community. Nothing the models see changed; no re-run.
2026-10-10
Release 1.0. Fixed setup for every model (30 turns, 40 minutes, $5 per run, high reasoning). 8 models, 40 challenges (14 Beats, 26 Motion), 2 runs each: 640 runs. A run that ends without a working entry gets one retry; 11 stayed DNF, all Gemini 3.8 Flash. Earlier votes reset.