aByes is a Python package for Bayesian A/B Testing, which supports two main decision rules: A lot of the underlying theory is discussed in this blog post. In your target
folder, clone the repository with the command:aByes
Installation
git clone //github.com/cbellei/abyes.git
Then, inside the same folder [as always, it is advisable to use a virtual environment]:
To check that the package has been installed, in the Python shell type:
If everything works correctly, the package will be imported without errors.
Dependencies
- aByes is tested on Python 3.5 and depends on NumPy, Scipy, Matplotlib, Pymc3 [see
requirements.txt
for version
information].
How to use aByes
The main steps to run the analysis of an A/B experiment are:
- Aggregate the data for the "A" and "B" variations in a List of numpy arrays
- Decide how to do the analysis. Options are: 1. analytic solution; 2. MCMC solution [using PyMC3]; 3. compare the analytic and MCMC solutions
- Set decision rule. Options are: 1. ROPE method; 2. Expected Loss method
- Set parameter to use for the decision. Options are: 1. Lift [difference in means]; 2. Effect size
These and many more examples and instructions can be found in this blogpost.
Example
In IPython, type:
import abyes as ab import numpy as np data = [np.random.binomial[1, 0.4, size=10000], np.random.binomial[1, 0.5, size=10000]] exp = ab.AbExp[method='analytic', decision_var = 'lift', rule='rope', rope=[-0.01,0.01], plot=True] exp.experiment[data]
This will plot the posterior distribution:
It will then give the following result:
*** abyes *** Method = analytic Decision Rule = rope Alpha = 0.95 Rope = [-0.01, 0.01] Decision Variable = lift Result is conclusive: B variant is winner!
There are many more examples available in the file
example.py
, which can be run from the root directory with the command:python abyes/examples/examples.py
Limitations
Currently, aByes:
- only focuses on conversion rate experiments
- allows for only two variants at a time to be tested
These shortcomings may be improved in future versions of aByes. [Feel free to fork the project and make these improvements yourself!]
Licence
Apache License, Version 2.0
Bayesian Tests are a Bayesian alternative to classical hypothesis testing, you can read more about it
here. Simply install from Suppose you run an AB test between two populations, the outcome being one set of measures for each group [could be conversion, revenue, latency, etc]. We'll call these two data sets Python Bayesian Tests
Setup
pip
:Distributions currently available
Example
control
[A group] and variant
[B group]:>>> control
array[[ 0.00892857, 0.0699088 , -0.101626 , ..., 0.323944 ,
0.5 , 0.236842 ]]
>>> variant
array[[ 0.0986842, -0.0176 , 0.0571429, ..., 0.452941 , -0.304348 ,
0.289474 ]]
You simply need to create a BABTest
object, specifying the data sets and the model you want to use. All available models can be found in babtest.py
. Here we'll assume the Student's t-distribution is a good description of our metric.
bt = BABTest[control, variant, model='student']
bt.run[]
bt.plot[]
This will typically produce the following plots:
Data and model
control
and variant
sets [red bars], as well as a few plots of distributions estimated in the Monte Carlo sampling [blue lines].In our example, one can eyeball that the model is a good fit for the data.
Posterior knowledge of distribution parameters
control
, variant
], distribution parameters are estimated. This "knowledge" of parameters is described as a likelihood, or
in other words denstity of credibility, which is represented in the above graphs. In our example, it can be seen that the parameters of the two sets are extremely likely to be different, as their distributions are shifted and- barely overlap in case of the mean,
- do not overlap in case of the other two parameters.
One could conclude that A and B groups are different here.
Troubleshooting
MacOS
Installing requirements
In order to install the pymc
requirement, you'll need a fortran compiler. On MacOS simply install with brew install gcc
Matplotlib runtime error
You may run into this matplotlib
issue:
RuntimeError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework.
Simply create a
~/.matplotlib/matplotlibrc
file with the following option: backend: TkAgg
, which will change the matplotlib
backend and solve the issue.
Requirements
numpy>=1.11.3
scipy>=0.18.1
pymc>=2.3.6
matplotlib>=1.5.3