On Beeminder.com, a new user is currently show the "classic" button style, of yellow letters, all caps, on a black background. I've set up an A/B test framework to try a "modern" style that is black text on yellow background, mixed case, with more rounded corners on the buttons. Will the new buttons be statistically significantly better than the old ones? (We use a Monte-Carlo simulation sampling from the posterior beta distributions for conversion rates for classic and modern styles, and "significantly better" means the MC simulation for one style beats the other 95%+ of the time)
Update 2025-12-29 (PST) (AI summary of creator comment): The market will resolve based on Expected Loss (EL) in addition to probability thresholds:
Resolves YES when:
P(modern being best) > 95%, OR
EL(modern) < 0.25%
Resolves NO when:
P(modern being best) < 5%, OR
EL(classic) < 0.25%
Threshold of Caring (TOC) is set at 0.25% (5% of the baseline ~5% conversion rate).
1,000@EspenJohannesen Great question!
Here's what we are actually going to do: the Monte-Carlo simulation also gives us an "Expected Loss" (EL) for each arm of the experiment, which is the amount of conversion rate that we would be giving up by choosing this arm of the experiment, multiplied by the probability of it not being the actual best arm. We can stop the experiment and resolve YES or NO based on the EL, as well as on the P(modern) versus P(classic). This will happen when the EL is less than some amount, called the "Threshold of Caring" (TOC). Right now, the baseline conversion rate is about 5%, so let's set the TOC to 0.25%, or 5% of the current conversion rate.
It resolves YES when P(modern being best) > 95% or EL(modern) < 0.25%
It resolves NO when P(modern being best) < 5% or EL(classic) < 0.25%