A Bayesian analysis of the data from a simple experimental design is straightforward. Recall that the theorem implies that the posterior is the likelihood weighted by the prior. Because a hypothesis corresponds to a hypothesized probability distribution, you therefore have to, for each probability in your expected distribution, multiply it by the corresponding probability in your obtaineddistribution. For example, when trying to estimate a population, a “hypothesis” could be “height is normally distributed and the mean height is 170 cm and SD=5”. This would entail that, for example, 5 random samples all being above 200 cm has a very small probability, such that the likelihood would cause the posterior distribution to shift the mean to the right. Let us take it step by step:
To apply Bayes' theorem to a simple within-subjects design, we first make a prior distribution where we associate each possible effect-size with the hypothesized probability. Then, from the likelihood (sample mean distribution) we simply multiply the corresponding probabilities.
- Suppose you have a between-group design with a treatment group, to which administered a certain dose of a drug, and a control group, that received no potent substance.
- You assume a priori, based on previous literature, that it will be roughly normally distributed, so that values vary in their probability in a symmetrical manner. You also believe, a priori, that the central value will be 3 and that a value of +-1 would have a 68% chance of occurring (i.e. the standard deviation is 1). This defines your prior distribution. (Note that prior distributions are in practice difficult to construct. Psychological theories, for example, rarely imply a particular P(effect|theory) distribution. )
- You obtain your sample distribution, with its own mean and standard deviation (the standard error). This is your likelihood distribution. Each value on its horizontal axis is its own hypothesis of the mean, so the evidence clearly favors the hypothesis population mean=sample mean the most.
- You multiply prior and likelihood for every parameter value (and adjust to make sure that the posterior distribution’s area is equal to 1). This is your posterior distribution. Note that there are simple formulae available to skip the hassle of calculating posteriors for each parameter value.
- The posterior distribution can be summarized by a “credibility interval” – the range that has a 95% probability of including the true effect, which, if we assume it is centered on the mean, is (M1 – 1.96xS1) to (M1 + 1.96xS1).
- If we are interested in different hypotheses – that is, different possible mean values – we can compare them using a ratio (the “Bayes factor”): P(H1|D) / P(H0|D) = P(D|H1) / P(D|H0) x P(H1)/P(H0). If more than 1, it supports the numerator hypothesis more than the denominator hypothesis, nudging your belief in the former’s direction. It is often informative to report how the Bayes factor depends on different priors, so that other researchers can choose Bayes factor value based on their own priors.
As you collect more data, the posterior will approach the "true" distribution, and consequently the interval will shrink in width (precision). You may thus use this as a stopping rule, running until it excludes the alternative hypothesis.
Before proceeding, we should highlight a few conceptual points:
- Lacking any prior means that you have no idea of the mean. In terms of prior distribution, this means that its standard deviation should be infinite. We call this a “flat prior”. There is a mathematical caveat to this, which is that when we apply a non-linear transformation of that variable (for example, its inverse) it will not remain flat. For example, if our variable is the male:female ratio of a country’s population, and we assume it as flat out of complete ignorance, then the female:male ratio will not be uniformly distributed. Hence, complete ignorance should mean an inability to assign any kind of prior (and in science, information corresponding to the prior is typically unavailable). There is an approach called “objective Bayesian” that tries to solve this.
- If your prior is wide (i.e. diffuse), the posterior will be dominated by the likelihood, and if it were flat, the posterior would be the same as the likelihood.
- The credibility interval of the posterior will normally be narrower than that of the prior, and will continue to do so as you collect more data. With more data, you therefore gain in precision, and different priors will converge on the same posterior.
- A Bayes factor around 1 means that the experiment was not very effective in differentiating between the hypotheses, in picking up a difference (it is insensitive). It may be as a result of vague priors. Intuitively, this penalty makes sense, for in a vague theory, data very different from it can support it, and because the Bayes factor is low, you will not be inclined to accept it.
- Your posterior only concerns hypotheses you have conceived of. As we learned in the introduction, this does not exhaust the space of possible theories.
But the most important take-home point is that, in the Bayesian approach, beliefs are adjusted continuously. You may continue to collect data for as long as you want to, until the credibility interval has the precision you require. The only information you need to update your belief is the likelihood (a controversial idea known as the “likelihood principle”). As a consequence, you can quantify the subjective probability of the hypotheses you have considered, but have no black-and-white criteria to support your decision making. However, you could stipulate, somewhat arbitrarily, that “to confirm a theory, the Bayes factor must be 4” and then collect data until you have 4 (or ¼, in support of the other hypothesis).