Experimental Mind #143

I hope you had a good weekend. It was a long one here, so that’s why this newsletter is a bit late. But the setup is the same: a curated overview of interesting reads, events and jobs for the experimental mind (that’s you).

Everything is handpicked by me, so you don’t have to. Enjoy!

🔎 What I’ve been reading

Picking Metrics 101: do's & don'ts
Picking Metrics 101: do’s & don’tsblog.statsig.com

Picking the right metrics for an experiment is not easy. The Statsig team is giving some good advice, including clear do’s and don’ts.

Paper: A/B Testing Intuition Busters

Ronny Kohavi, Alex Deng and Lukas Vermeer have written a paper on Common Misunderstandings in Online Controlled Experiments: A/B Testing Intuition Busters. They are still open for review and comments.

A/B Testing: Ruling Out To Conclude
A/B Testing: Ruling Out To Concludeconductrics.com

Matt Gershoff explains two important concepts from A/B testing.

There are two things we are trying to rule out when we run AB Tests: confounding & sampling variability/error.

A Survey of Causal Inference Applications at Netflix

Netflix relies heavily on A/B testing to measure causal effects. In this blog post we get a peek into the recent held internal Causal Inference and Experimentation Summit.

The weeklong conference brought speakers from across the content, product, and member experience teams to learn about methodological developments and applications in estimating causal effects. We covered a wide range of topics including difference-in-difference estimation, double machine learning, Bayesian AB testing, and causal inference in recommender systems among many others.

🚀 Job opportunities

All featured roles:

Also checkout the job board for more open roles in the field of experimentation, CRO and analytics.

📅 Upcoming events

Experimentation Culture Awards 2022
Experimentation Culture Awards 2022experimentationcultureawards.com

I’ll be one of the jury members at the 2022 Experimentation Culture Awards. Join the award session and listen to inspiration stories.

All upcoming events in the next months:

Second batch of tickets will release soon:

See the full overview of events to find other events and conferences for you and your team.

💬 Quote of the week

“The old is dead, and I don’t know what the new is. The only way to find the new is to start different things and see if there’s something that can come out of experimentation.” — David Lynch

😉 Fun of the week

The latest Marketoonist on The Innovation Funnel:

… this Darwinian process of bringing ideas to life doesn’t necessarily lead to survival of the fittest ideas. If we’re not careful, the innovation funnel leads to survival of the safest ideas.  

☕ Thanks for reading

If you’re enjoying the Experimental Mind newsletter and want to say thank you? Buy me a coffee (or a beer). Or share this newsletter with a friend or colleague.

Have a great week — and keep experimenting.