What a $20-per-test loop taught us about short-form video — including the part where our own framework failed.
The first three seconds are the wrong thing to obsess over
Every short-form playbook in circulation opens the same way: win the first three seconds or lose the viewer. It is the most repeated claim in the industry and the most rarely tested.
We tested it. Across 298 controlled experiments on microdrama pilots, three-second retention explained 26% of the variance in our cost outcome. Fifteen-second retention explained 70%.
The hook is the weakest checkpoint on the retention curve, not the strongest. It is where the industry spends most of its creative energy and where the data says the least.
That finding is not the point of this piece. It is an illustration of the point, which is this: almost everything we believed about what makes short-form video work turned out to be either wrong, unmeasurable, or true for reasons different from the ones we'd assumed. We only found that out because we built a machine that could tell us we were wrong, and then had the discipline to run our own playbook through it.
This is what the machine was, what it produced, and where it broke.
The problem: creative decisions made by seniority
In most content organisations, the pilot that goes into production is chosen in a room. Someone senior watches three or four cuts, forms a view, and the view becomes the decision. The reasoning is usually articulate and often wrong, and — critically — nobody ever finds out which, because the alternatives are never made.
This is not a criticism of taste. Taste is real and experienced people have more of it. The problem is that taste produces a point estimate with no error bar, and short-form video has an error bar you would not believe.
Here is the number that ended the argument internally. Within a single production batch — same show, same writers, same cast, same week, several pilot cuts of the same episode — the worst cut cost a median of 2.5x more per qualified view than the best cut. In the top quartile of batches, the spread was 3.5x.
Same story. Same team. Same budget. Two and a half times the cost, depending on which cut you happened to pick.
No taste is that good. Nobody's is.
The system
The design was deliberately cheap and deliberately blind.
Fixed budget. Roughly $20–25 per test. Small enough that nobody had to approve it, which mattered more than any other design decision. Cheap experiments get run; expensive ones get debated.
Unbranded distribution. Every pilot went out on a Facebook page with no show identity, no channel following, no prior audience. This is the part most teams skip and it is the part that makes the results mean anything. Test on your own channel and you measure your existing audience's loyalty. Test cold and you measure the creative.
Fixed window. 24 hours. Long enough for the retention curve to stabilise, short enough to keep the production calendar moving.
One outcome number. We used cost per deep completion — internally, "C95." Views are vanity in this format; a viewer who abandons at eight seconds tells you nothing about whether they'd install an app. Cost per viewer who watched nearly to the end is the closest single proxy we found for downstream intent. Lower is better.
Iterate, promote, repeat. Each show's launch arc got an average of 7.3 pilot tests. The winner became the new baseline and the next arc's cuts were tested against it.
That's the whole system. There is no proprietary technology in it. It costs less than a day of a senior creative's time to run a full batch.
What it produced
Testing is worth about a third of your acquisition cost. Choosing the tested winner rather than a pilot selected on judgment reduced cost per deep completion by a median of 34% (mean 33%, interquartile range 22–43%). That is not a marginal gain. On a media budget of any size it is the difference between a channel that works and one that doesn't.
The team got 40% better. Over 298 tests, cost per deep completion fell about 40%, controlling for video duration, show identity, and genre. The raw decline was 46%, but part of that was mix — later titles were inherently easier — and the honest number after controls is 40%.
Every show improved. Of the eleven shows with at least eight tests, all eleven trended toward lower cost over time (sign test p = 0.001). This is the finding I'd stake the most on. A programme-level average can be dragged by one lucky title. Eleven for eleven is a team learning.
Four things the data said that we didn't expect
1. A big hook that doesn't hold is worse than a modest hook that does.
The steepness of the drop between three and five seconds correlated positively with cost (rho = +0.44, p < 0.0001). Our single worst-performing test in nine months held 78% of viewers at three seconds — top-quartile hook — and 13% at thirty. It cost more per qualified view than anything else we ever ran.
A hook is a promise. The cost is not paid at the moment you make it. It is paid at the moment the viewer works out you can't keep it, and by then the algorithm has already learned something about your content that you cannot unlearn.
2. Genre difficulty is mostly an illusion.
One of our genres looked roughly 60% more expensive than the others on raw averages. This was accepted internally as a category tax — that genre is just harder.
Control for thirty-second retention and the genre effect vanishes entirely (t = −0.93, not significant). The genre wasn't harder. Our cuts in that genre simply retained worse. That reframes it from a budgeting problem into a creative problem, which is the more useful kind, because creative problems are fixable and taxes are not.
3. Predictive power plateaus at fifteen seconds.
Retention at 3s explains 26% of outcome variance; 5s, 39%; 10s, 59%; 15s, 70%. After that it flattens — 20s, 25s and 30s all sit around 74%. If a cut is still holding at fifteen seconds, the remaining information is mostly already in.
This has a direct operational consequence: fifteen seconds is where creative review time should concentrate. Not the first three.
4. Longer costs more, monotonically.
Median cost per deep completion rose steadily with runtime across every duration band we tested. Some of this is arithmetic — a longer video has a further finish line — but the practical implication holds regardless: every additional second of runtime must earn its place against a rising cost curve, and most don't.
The audit: our own playbook failed
Alongside the test log we maintained a creative checklist — twenty-four elements across five segments of the video, from opening hook through cliffhanger. Shocking visual reveal. Establish antagonist. Powerful flashback to trauma. Unresolved conflict. Open ends. The standard vocabulary of the format, formalised into a grid.
The checklist was the artefact everyone wanted. It was the thing that felt like knowledge.
We coded twenty-seven videos against all twenty-four elements, retrospectively, from raw files, then tested each element against actual outcome.
Nothing survived.
With twenty-four simultaneous tests, the corrected significance threshold is p < 0.0021. The best-performing element came in at p = 0.014. Not one element cleared the bar.
Worse than null: the direction was wrong. Of the sixteen elements with enough variation to evaluate, fifteen showed videos containing the element performing worse. Powerful flashback to trauma: +45% cost. Protagonist in poor situation: +70%. Shocking visual reveal: +63%. Teases upcoming conflict: +43%. The number of checklist boxes a video ticked correlated positively with cost (rho = +0.32) — more elements, worse result, though not at significance.
Four elements appeared in zero of twenty-seven videos, including both call-to-action rows. They had been on the checklist for months and had never once been observed.
Two caveats, both of which I think strengthen rather than weaken the finding.
Twenty-seven videos is a small sample and this test is underpowered against subtle effects. Real but modest element-level effects could be hiding in the noise. What can be ruled out is a large effect, and a large effect is what a creative checklist implicitly promises.
Second: the coding was retrospective, done by people who already knew how each video had performed. That is normally a fatal flaw — it biases coding toward the result. Here it cuts the other way. Any such bias would have pushed the elements toward looking predictive. They still didn't.
Why we published the negative result
The commercially convenient version of this piece writes itself. Nine elements that drive completion. A framework with a name. A downloadable grid.
The problem is that it would be false, and the first person to ask "did you code the videos that failed?" would end the conversation.
There's a deeper reason. The checklist and the loop are different kinds of object, and conflating them is the central error in how this industry talks about virality. The checklist is a theory — a claim about what causes what. The loop is a decision system — a way of choosing between options without needing the theory to be right.
Our theory was wrong. Our system delivered a 40% improvement anyway.
That's not a paradox. It's the entire point. The loop works because it doesn't require anyone to be right in advance. It only requires that the alternatives actually get made and actually get measured. Every organisation that skips the loop and invests in the framework has it exactly backwards: they're buying the fragile thing and skipping the robust one.
What I'd do differently
Code the checklist prospectively. Score each cut before the test runs, then compare predicted rank to actual rank. That is the only design that can ever validate a creative framework, and we never ran it. Everything in our checklist tab is measurement dressed as prediction.
Include a genuine control arm. Every cut we tested was made by a team already steeped in the house style. We never tested a pilot built to deliberately violate the playbook, so we could never observe what the elements were worth.
Separate the outcome metric from the retention curve. Cost per deep completion is close to arithmetically determined by deep retention. That makes it an excellent ranking device and a poor discovery device — it can tell you which of two cuts is better, but it can't tell you much you didn't already see in the curve.
Fix the metric's blind spot on duration. Cost per deep completion is not comparable across formats. Long-form content that was demonstrably successful in market scored badly on it in our reverse tests, purely because the finish line was further away. Within a fixed duration band it's reliable. Across bands it lies.
The one-line version
If you take a single thing from this: stop trying to know which cut will win, and build the cheapest possible machine for finding out.
Ours cost $20 a test. It ran 298 times. It made every show we touched cheaper to launch. And when we finally pointed it at our own creative theory, it told us the theory was wrong — which is the most valuable thing it ever did, and the only reason I trust the rest of the numbers in this piece.
Figures in this piece are drawn from 298 controlled tests conducted over a nine-month period. Show titles, budgets and platform-level spend are omitted. Statistical work: Spearman rank correlations for bivariate relationships, OLS on log-transformed cost outcomes with duration, show and genre controls for the trend estimate, Welch's t-tests with Bonferroni correction across the twenty-four checklist elements.