Bayesian ab testing python github

aByes

aByes is a Python package for Bayesian A/B Testing, which supports two main decision rules:

• Region Of Practical Equivalence (as in the paper Bayesian estimation supersedes the t-test, J. K. Kruschke, Journal of Experimental Psychology, 2012)
• Expected Loss (as discussed in Bayesian A/B Testing at VWO, C. Stucchio)

A lot of the underlying theory is discussed in this blog post.

Installation

• In your target folder, clone the repository with the command:

  git clone https://github.com/cbellei/abyes.git

• Then, inside the same folder (as always, it is advisable to use a virtual environment):

• To check that the package has been installed, in the Python shell type:

• If everything works correctly, the package will be imported without errors.

Dependencies

• aByes is tested on Python 3.5 and depends on NumPy, Scipy, Matplotlib, Pymc3 (see requirements.txt for version

information).

How to use aByes

The main steps to run the analysis of an A/B experiment are:

• Aggregate the data for the "A" and "B" variations in a List of numpy arrays
• Decide how to do the analysis. Options are: 1. analytic solution; 2. MCMC solution (using PyMC3); 3. compare the analytic and MCMC solutions
• Set decision rule. Options are: 1. ROPE method; 2. Expected Loss method
• Set parameter to use for the decision. Options are: 1. Lift (difference in means); 2. Effect size

These and many more examples and instructions can be found in this blogpost.

Example

• In IPython, type:

  import abyes as ab
  import numpy as np
  
  data = [np.random.binomial(1, 0.4, size=10000), np.random.binomial(1, 0.5, size=10000)]
  exp = ab.AbExp(method='analytic', decision_var = 'lift', rule='rope', rope=(-0.01,0.01), plot=True)
  exp.experiment(data)

• This will plot the posterior distribution:

• It will then give the following result:

  *** abyes ***
  
  Method = analytic
  Decision Rule = rope
  Alpha = 0.95
  Rope = (-0.01, 0.01)
  Decision Variable = lift
  
  Result is conclusive: B variant is winner!

• There are many more examples available in the file example.py, which can be run from the root directory with the command:

  python abyes/examples/examples.py

Limitations

Currently, aByes:

• only focuses on conversion rate experiments
• allows for only two variants at a time to be tested

These shortcomings may be improved in future versions of aByes. (Feel free to fork the project and make these improvements yourself!)

Licence

Apache License, Version 2.0

Python Bayesian Tests

Bayesian Tests are a Bayesian alternative to classical hypothesis testing, you can read more about it here.

Setup

Simply install from pip:

Distributions currently available

• Bernoulli distribution
• Exponential distribution
• Lognormal distribution
• Normal distribution
• Poisson distribution
• Student's t-distribution

Example

Suppose you run an AB test between two populations, the outcome being one set of measures for each group (could be conversion, revenue, latency, etc). We'll call these two data sets control (A group) and variant (B group):

>>> control
array([ 0.00892857,  0.0699088 , -0.101626  , ...,  0.323944  ,
        0.5       ,  0.236842  ])
>>> variant
array([ 0.0986842, -0.0176   ,  0.0571429, ...,  0.452941 , -0.304348 ,
        0.289474 ])

You simply need to create a BABTest object, specifying the data sets and the model you want to use. All available models can be found in babtest.py. Here we'll assume the Student's t-distribution is a good description of our metric.

bt = BABTest(control, variant, model='student')
bt.run()
bt.plot()

This will typically produce the following plots:

Data and model

In our example, one can eyeball that the model is a good fit for the data.

Posterior knowledge of distribution parameters

• barely overlap in case of the mean,
• do not overlap in case of the other two parameters.

One could conclude that A and B groups are different here.

Troubleshooting

MacOS

Installing requirements

In order to install the pymc requirement, you'll need a fortran compiler. On MacOS simply install with brew install gcc

Matplotlib runtime error

You may run into this matplotlib issue:

RuntimeError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework.

Simply create a ~/.matplotlib/matplotlibrc file with the following option: backend: TkAgg, which will change the matplotlib backend and solve the issue.

Requirements

• numpy>=1.11.3
• scipy>=0.18.1
• pymc>=2.3.6
• matplotlib>=1.5.3