Fhyujk

While studying Bayesian statistics, somehow I am facing a problem to understand the differences between prior distribution and prior predictive distribution. Prior distribution is sort of fine to understand but I have found it vague to understand the use of prior predictive distribution and why it is different from prior distribution.

Let $Y$ be a random variable representing the (maybe future) data. We have a (parametric) model for $Y$ with $Y sim f(y mid theta), quad theta in Theta$, $Theta$ the parameter space. Then we have a prior distribution represented by $pi(theta)$. Given an observation of $Y$, the prior distribution of $theta$ is
$$
f(theta mid y) =frac{f(ymidtheta) pi(theta)}{int_Theta f(ymidtheta) pi(theta); dtheta} $$
The prior predictive distribution of $Y$ is then the (modeled) distribution of $Y$ marginalized over the prior, that is, integrated over $pi(theta)$:
$$
f(y) = int_Theta f(ymidtheta) pi(theta); dtheta
$$ that is, the denominator in Bayes theorem above. This is also called the preposterior distribution of $Y$. This tells you what data (that is $Y$) you expect to see before learning more about $theta$. This have many uses, for instance in design of experiments, for an example, see Experimental Design on Testing Proportions or Intersections of chemistry and statistics.

Another use is as a way to understand the prior distribution better. Say you are interested in modeling the variation in weight of elephants, and your prior distribution leads to a prior predictive with substantial probability over 20 tons. Then you might want to rethink, typical weight of largest elephants is seldom above 6 tons, so a substantial probability over 20 tons seem wrong. One interesting paper in this direction is Gelman (which do not use the terminology ...)

Finally, preposterior concepts are typically not useful with uninformative priors, they require prior modeling taken serious. One example is the following: Let $Y sim mathcal{N}(theta, 1)$ with a flat prior $pi(theta)=1$. Then the prior predictive of $Y$ is
$$
f(y)= int_{-infty}^infty frac1{sqrt{2pi}} e^{-frac12 (y-theta)^2}; dtheta = 1
$$
so is itself uniform, so not very useful.

Predictive here means predictive for observations. The prior distribution is a distribution for the parameters whereas the prior predictive distribution is a distribution for the observation.

If $X$ denotes observation and we use the model (or likelihood) $p(x mid theta)$ then a prior distribution is a distribution for $theta$, for example $p_beta(theta)$ where $beta$ is a set of hyperparameters. Note that there's no conditioning on $beta$ , and therefore the hyperparameters are considered fixed, which is not the case in hierarchical models but this not the point here.

The prior predictive distribution is the distribution of $X$ "averaged" over $theta$,

$$
p_beta(x) = int p(x mid theta) p_beta(theta) dtheta
$$

This distribution is prior as it does not rely on any observations.

We can also define the same way the posterior predictive distribution, that is if we have a sample $X = (X_1, dots, X_n)$ the posterior predictive distribution is

begin{align*}
p_beta(x mid X) &= int p(x mid X, theta) p_beta(theta) dtheta \
&= int p(x mid theta) p(X mid theta) p_beta(theta) dtheta \
&= int p(x mid theta) p_beta(theta mid X)dtheta
end{align*}

thus the posterior predictive distribution is constructed the same way as the prior predictive distribution but while in the latter we weight with $p_beta(theta)$ is the former we weight with $p_beta(theta mid X)$ that is with our "updated" knowledge about $theta$.

Example : Beta-Binomial

Suppose our model is $X mid theta sim Bin(n_1,theta)$ i.e $P(X = x mid theta) = theta^x(1-theta)^{n_1-x}$.

We suppose a beta prior distribution for $theta$, $beta(a,b)$ where $(a,b)$ is the set of hyper parameters.

Then the prior predictive distribution for $theta$ is the beta-binomial distribution of parameter $(n_1,a,b)$. This discrete distribution gives the probability of $k$ successes out of $n_1$ trials given hyper-parameter $(a,b)$ on the probability of success.

Now suppose we observe $n_1$ draws $(x_1, dots, x_{n_1})$ whith $x$ successes.

Since the binomial and beta distributions are conjugate distributions we have:
begin{align*}
p(theta mid X=x)
&propto theta^x (1 - theta)^{n_1-x} times theta^{a-1}(1-theta)^{b-1}\
&propto theta^{a+x-1}(1-theta)^{n_1+b-x-1} \
&propto beta(a+x,n_1+b-x)
end{align*}

Thus $theta mid x$ also follows a beta distribution. Then, $p(x mid x, a,b)$ follows a beta-binomial but this time of parameters $(a+x,b+n_1-x)$ rather than $(a,b)$

Upon a $beta(a,b)$ prior distribution and a $Bin(n_1,theta)$ likelihood, if we observe $x$ successes out of $n_1$ trials the posterior predictive distribution is a beta-binomial of parameters $(n_2,a+x,b+n_1-x)$. Note that $n_2$ and $n_1$ play differents roles, since here the posterior predictive is about:

Given my current knowledge on $theta$ after observing $x$ successes out of $n_1$ trials, i.e $beta(n_1,a+x,n+b-x)$, what probability I have of observing $k$ successes out of $n_2$ additional trials.

I hope this is useful and clear