Randomise everything not glued down

Almost every decision your product makes should have some amount of randomness in it.

A lot of product decisions are very arbitrary:

• "Send this notification after 24 hours"
• "Do X at 9AM every day"
• "Show the top 20 recommended content items"

Each of these contains a magic number chosen once at some point in the build process and never revisited. Why is 24 hours better than 23? Why 9AM instead of 9:30AM?

Instead of treating them as fixed, treat them as a policy. Decide what the reasonable boundaries are for that value:

• "Send this notification after 12-36 hours"
• "Do X between 6AM and 11 AM every day"
• "Show the top 10-30 recommended content items"

Obviously don’t randomise your security policies or other invariants.

Then randomly sample from that distribution. Log all those decisions along with the propensities - the probability of that choice at decision time.

Free A/B testing (almost)#

If you randomise and log your decisions, you can estimate the value of any given policy by inverse-propensity weighting:

$$V(\pi) = \frac{1}{N} \sum_i{\frac{\pi(a_i | x_i)}{\pi_0(a_i | x_i)}r_i}$$

An A/B test is just estimating the value of the policy "always A" vs the policy "always B":

$$V(\pi_A) = \frac{1}{N} \sum_i{\frac{\mathbb{1}[a_i = A]}{\pi_0(a_i | x_i)}r_i}$$

When A is assigned with equal probability to everyone, this is just the mean reward on the decisions where A was chosen.

A/B testing is just a special case of randomisation that only compares two fixed policies (A or B). By randomising, you get A/B testing for free, plus the ability to look back and evaluate any policy after the fact - even policies you didn't consider at the time.

Caveat: There is a variance trade-off here

Encode uncertainty#

Every decision we make in our product encodes a prior belief we have - “this is the best choice of parameter”. Most of the time we don’t know this for certain.

We can encode our uncertainty into the product instead.

Let’s say we have a reasonable suspicion that in fact 9AM is the best time to do something. That may be the case, but we're never going to be 100% confident in that. We can sample 9AM more frequently than others.

$$A = \begin{cases} 8AM & \text{with probability 0.25}\\ 9AM & \text{with probability 0.5}\\ 10AM & \text{with probability 0.25} \end{cases}$$

On the other hand, if we have no idea which parameters are better than others, we can keep our distribution completely flat - sample all values with equal probability. Logging our propensities lets us evaluate later down the line.

Everything is Bayesian#

Every hard-coded magic number in our product is really a Bayesian statement. Choosing exactly 24 hours as our notification window is really a statement saying “I’m 100% confident that 24 hours is the best choice”. That’s almost never the right statement.

Magic numbers are certainty where most product decisions aren’t.