Is this the future of growth experiments?
Running experiments is now part of every growth process in a startup — If you are not doing it, it means that something wrong. Send me a note and we can solve this.
But how has this process, and especially the technology enabling it evolved, since the birth of A/B testing? How are experiments for growth-focused startups going to be run in the future?
First, let’s take a look back at how early-stage companies used to run web-based experiments.
Back to the 90's
The early days
In the early days of the Internet, the primary medium was text, and traffic was much more difficult to acquire. This meant that launching an experiment took a lot longer, because you had to wait to reach statistical significance.
Google engineers started early, as they ran their first A/B test in 2000 to determine the optimum number of results to display on a search engine results page.
Then by 2011 the had run 7000 tests that year alone. Now that’s probably the number of tests they run in a week…
A/B Testing takes off
When surfing the Internet became a bit faster. It started to make more sense to test different designs as users could actually load the page with images.
You could just create 2 different pages and send traffic randomly to both. Then the highest converting one was winning. But what was a conversion back then? There were no Facebook pixels or whatsoever.
Though with the democratization of payments on the internet. There was a strong business interest to make sure you would optimize for conversion. Google Adwords conversion tracking was launched in 2007.
The birth of WYSIWYG experimentation
Then some companies realized this could be their product: allow anyone to create an experiment and measure the impact to pick a winner. All of this with only limited coding skills.
Optimizely was founded in 2009 and is now a leader in the space. Then AB Tasty, VWO and others jumped on the bandwagon.
When the Internet shifted to mobile sites and apps, a second wave of companies with a similar vision but focused on mobile experimentation came in: Apptimize, or Taplytics…
And, of course, all the ESPs (Email Service Providers) started to have features to A/B test email content(Customer.io, Mailchimp, Drip…) but this is for another post.
Here is an example of the pre-order page built with Optimizely for SimCity circa 2013. 43% increase in checkouts without the top Pre-order banner!
Current phase: Dynamically created
The phase we are currently in is the dynamically created page. By using data and pre-formatted elements. You can pull together the perfect page in milliseconds. Note, this is not to create the experiment itself, but to create the page used for the experiment. This is important for later in the post.
Here is a great example from Guillaume Cabane. He shared in the video here that at Drift, they are creating a landing page based on 4.2 billion potential combinations.
They are doing this by combining multiple sources of data including the IP address, sending it to Clearbit for enrichment, then to Madkudu for scoring which comes back to the page to generate the perfect one for each user.
I wonder how long it took to create this system and similarly how much time do they need to make changes to the experiment.
Though, it is for sure the best way to display the best content to each user right now. 👏
Then similarly to what the WYSIWYG platforms did a decade ago, new ones will arise to offer exactly the same product as the custom-built system we have seen above, and make it accessible to everyone.
RightMessage is the first that I am aware of. But for sure more to come and most probably some new features coming shortly to your favorite AB testing SaaS platform.
The future: Every step of the process in real-time
So it takes milliseconds to create the perfect landing page. But what about the step before the creation? When you lay down the idea of the experiment you want to test.
Well, here is how designers at Airbnb are working to solve this problem. With a clear vision:
The time required to test an idea should be zero.
Starting with this vision statement, they started to work on different solutions. Including the one they shared below.
This prototype is using about a dozen hand-drawn components as training data, open source machine learning algorithms, and a small amount of intermediary code to render components from the design system into the browser.
I am excited to see how far they can push this. You can read more about their vision and work in this article.
The perfect experimentation process
From there how do we create the perfect experimentation process? When I say perfect, it’s mostly based on the speed of the process.
Because make no mistake. The faster you experiment, the faster you learn. Therefore the faster you grow.
Let’s recap on where we are to have this perfect process:
- having the idea (who’s working on this?)
- making this idea come to life (what Airbnb designers are working on)
- creating the experiment (still, takes time and dependent on the above)
- generating the perfect landing page (currently available)
- analyzing the result (in some case this is already automatic)
What is your team doing to experiment faster? Are you already in a full real-time growth process?
And beyond
First ones who are fully done with the checklist above can start working on the future painted in Minority report…
