NOTE (edited): I'm running this experiment with n1.tools, which is now a fully functional MVP! The tool is free to use, so feel free to register and try it out.
--------
I have been conducting a blinded trial to see whether 1.5mg of melatonin decreases the amount of time it takes me to get to sleep, as measured by my Oura ring (sleep latency).
1. Take 1.5mg melatonin or a placebo (blinded) before going to bed.
2. Sleep, check what I took the next morning.
3. Check the sleep latency metric in the Oura app and record alongside whether I took the placebo/supplement.
3. Repeat for 20 days.
Resolution criteria: Resolves YES if a naive model (i.e. that doesn't include confounders etc.) suggests at least an 80% probability that 1.5mg melatonin reduces my sleep latency at all.
Here's an example experiment that shows a 81% chance of a reduction in my post-coffee anxiety from L-Theanine - P(decrease) 0.81
Extra notes:
- I've actually already done this experiment (last day was today, March 27th), but I haven't released the results.
- I won't bet on this market.
- I haven't done the experiment on days when I stayed out later than usual, or consumed alcohol.
-----------
Model details (for those interested)
I'll model the posterior distribution of sleep latency with the placebo and with 1.5mg melatonin, and then merge those to create a posterior distribution for the absolute difference in sleep latency between the two.
I'll be using a Bayesian model for this, with the assumptions laid out below. Importantly, I assume that the prior and posterior sleep latency observations are normally distributed. Skipping this assumption would mean I have to use a sampling algorithm like MCMC, which would take too long to be feasible in the n1.tools app (I could assume other forms for the distributions and use the corresponding equation to update, but for now I'm sticking to the above).
Priors:
Mean Melatonin ~ Normal(mean_supplement_data, var_supplement_data)
Mean Placebo ~ Normal(mean_placebo_data, var_placebo_data)
Variance Melatonin ~ Variance of supplement_data
Variance Placebo ~ Variance of placebo_data
Likelihoods:
Data Melatonin ~ Normal(Mean Melatonin, sqrt(5))
Data Placebo ~ Normal(Mean Placebo, sqrt(5))
Posterior Inference:
Probability(Absolute Difference ≤ 0) ≥ 0.80
1,000🏅 Top traders
| # | Name | Total profit |
|---|---|---|
| 1 | Ṁ37 | |
| 2 | Ṁ21 | |
| 3 | Ṁ21 | |
| 4 | Ṁ16 | |
| 5 | Ṁ14 |
People are also trading
So this was a blowout for team melatonin; on average I fell asleep 7.5 minutes sooner and basically the entire posterior distribution of possible effect sizes is in the "slept sooner" zone.
Note: n1.tools was approved by Oura, so it's now possible to connect and run experiments with your Oura ring!
@JoeBoyle I needed to make an assumption about the variance in order to make use of conjugate priors to update the posterior, and 5 seemed reasonable (implies a std dev of around 2.24). Other than that it's a purely subjective choice.
Just FYI: 1.5 melatonin is generally higher than is optimal for adjusting sleep schedules (i.e. moving them earlier or later). For that, small doses taken a couple hours earlier are counterintuitively more effective.
You can google the Melatonin Response Curve, esp. with respect to circadian disorders for more info. This will also visualize the equally surprising (to most people) fact that melatonin dosed at the wrong times will keep you up longer .
If you're happy with when you're falling asleep and only want to reduce sleep latency (i.e., you want to be able to get in bed later than your current time, and still manage to fall asleep by the same time you currently tend to), 1.5 mg is probably great, but might lose efficacy in the long term (this part is more anecdotal).
@Vergissfunktor Thanks for the info! Not trying to change my bedtime, more keen to fall asleep faster. There's a few things I'd like to test to get to an ideal dosage of melatonin (different dosages on subjective wakefulness the next day, etc.), so possibly a similar market in future.
@LuisCostigan What was the basis for using 1.5mg? Based on what is commonly sold in my part of 'murica, it's more like 10mg as a standard dose size. Is that crazy-high based on what you know? I take 10mg occasionally (maybe 1 or 2x per week). I did note a meaningful decline in how effective it felt after having taking it for a few months.
@AlQuinn a combination of 1) I already had some 3mg pills lying around that I can cut in two, 2) Some friends told me 3mg made them feel drowsy the next day, and 3) some cursory research seemed to suggest 1-3 mg is common (such as here)
10mg does sound like a lot, especially given point 2 (but to each their own, hence the experiment!)
@AlQuinn I am not a doctor, but here is Scott Alexander (a doctor) talking about Melatonin: https://slatestarcodex.com/2018/07/10/melatonin-much-more-than-you-wanted-to-know/
In it he suggests taking 0.3mg of melatonin ~9 hours after you wake up:
2. What is the right dose of melatonin?
0.3 mg.
“But my local drugstore sells 10 mg pills! When I asked if they had anything lower, they looked through their stockroom and were eventually able to find 3 mg pills! And you’re saying the correct dose is a third of a milligram?!”
Yes. Most existing melatonin tablets are around ten to thirty times the correct dose.
A meta-analysis of dose-response relationships concurred, finding a plateau effect around 0.3 mg, with doses after that having no more efficacy, but worse side effects
3. Timing:
The general heuristic is that melatonin drags your sleep time towards the direction of when you take the melatonin.
So if you want to go to sleep (and wake up) earlier, you want to take melatonin early in the day. How early? Van Geijlswijk et al sums up the research as saying it is most effective “5 hours prior to both the traditionally determined [dim light melatonin onset] (circadian time 9)”. If you don’t know your own melatonin cycle, your best bet is to take it 9 hours after you wake up (which is presumably about seven hours before you go to sleep).
@Nikos that's great advice, and is what works best in my personal experience.
One of the reasons that it's sold in 5mg and 10mg doses in the US is that it's classified as a health supplement, and is therefore not regulated. Consumers assume more is better and that companies wouldn't harm them, so it's easy to outcompete the healthy dosages by offering 10mg pills.
@Nikos interesting, didn't know Scott did a write-up on that!. I started taking melatonin when I got covid because: 1) I vaguely remember a study (which probably won't replicate) that it can reduce the severity of covid, and 2) omicron covid gave me that strange symptom of lower back pain, so I had trouble sleeping for the first time in my life.
I'm a very good sleeper normally, so I take melatonin (10mg) occasionally simply because I often feel better rested if I take some or because I'm traveling/flying and want to be rested when I arrive, but have never tried a lower dose. I actually find "non-drowsy" cetirizine knocks me out much better than melatonin, though it takes longer to manifest (~1 hour for 5mg cetirizine vs 15 minutes for 10mg melatonin).
n1.tools looks cool! I kind of doubt that people will pay for it, since as you say in the FAQ, you can do the same thing yourself with a little extra legwork. I'm not a business person, but I'd recommend making it free at first and finding out what people like the most about your product and adapting accordingly. Then you could add a premium plan with extra features once you have a solid customer base
@CDBiddulph Thanks for your kind words! I do intend to experiment a little with pricing, but advice I often hear for new products is something like "get to an MVP and see if people will pay for it", so I'm trying to weigh that up against the approach you suggested.
I'll say a few things in defence of the current pricing structure:
- I've since added the % chance of an increase/decrease in the outcome from the intervention (on top of the p-value), which wasn't completely trivial.
- The interface is hopefully easy to use, and much easier to deal with (particularly on the go) than Excel or a Python notebook.
- It's affordable at 4.99 with a 7 day trial, and super easy to cancel from the dashboard.
I've already built an Oura integration (which I actually used for this experiment), I would need to get the app approved before I release it publicly. I think another quick win would be an integration with RescueTime
to get objective measurements on productivity.