Test on what policies are worth, and stop before it costs you.

Experiments is where your marketing tests are designed, run and enforced. Every test is judged on policy economics, carries its own limits and lands in the ledger with its result, win or lose.

Texas comparison sites, bids up 15%
Split by ZIP code against a holdout, day 11 of 28
Stopped by its limits
Extra binds
+38
Cost per bound policy
+$41
Projected loss ratio
0.69
Projected loss ratio, test arm over holdout, in points
Limit set by Maria: stop if projected loss ratio passes 0.68.

Judged on what the policies are worth.

A test that wins on cost per lead can still lose money. Experiments in Fountlabs are judged on the measures that matter in insurance, cost per bound policy, quote to bind, retention, projected loss ratio and expected contribution, using the profitability layer to read a result long before the claims come in.

Every test carries limits you set up front. Cross one and the test stops, the budget goes back, and the reason goes in the ledger.

Experiments
Every test in market, and what it is judged on
New test
Safe Driver Search, new ad copyJudged on 3-year value per policyRunning, day 9
Florida home, call back inside 5 minutesJudged on quote to bindWon, rolled out
Texas comparison sites, bids up 15%Judged on projected loss ratioStopped by its limits
Meta Texas, broad audienceJudged on cost per bound policyBlocked, overlaps a live test
Home, affiliate price testJudged on 12-month retentionWaiting for sign-off

Tests that stay honest to the end.

Most marketing tests fall apart in the middle. Someone moves the budget halfway through, a second test lands on the same audience, or a result gets called on four days of data. Fountlabs holds every test to its design until it's done.

  • Limits on spend, cost per policy or projected loss ratio
  • Tests that would collide with a live one wait until it ends
  • Results are called when there are enough policies to call them
  • Winners roll out with a sign-off, and the whole thing goes in the ledger
Limits on this test
Set by Maria before launch, checked every hour
Projected loss ratioStop if the test arm passes 0.68Tripped on day 11
SpendNo more than $60k over the test$31k used
OverlapNo other test on Texas comparison trafficClear
Enough to call itAt least 400 bound policies per arm212 so far
Tests worth running

The agents spot the opening. And set up the test.

When the Monday analysis spots an opening, say Texas comparison sites writing better business than their bids suggest, an agent proposes the test: how to split it, how long it needs to run, what to judge it on and where the limits should sit. A person approves it before anything changes in market.

Bids and budgets

Raise bids on a comparison site in one state and see whether the extra policies hold their margin.

Lead partners and prices

Try a new lead partner, or a new price with an existing one, with a cap on what you're willing to lose finding out.

Speed to lead

Call back inside five minutes in one region and measure what it does to binds and retention.

Audiences and creative

Put a new lookalike seed or a new message up against what you run today, judged on 3-year value per policy.

Where it sits

Part of one system.

Experiments read their results from the profitability layer, so a test can be judged on what its policies are worth months before the claims arrive. Every test, and what it taught you, goes in the ledger.

Test boldly. Lose small.

See Fountlabs