Model selection - Information criteria, part I

We covered a paper by Schadt and others last week that dealt with selecting the best model for the data. We want to select the simplest model that explains the most data, and there is a tradeoff between model fit and complexity.

In practice, people use various information criteria to formalize the tradeoff, usually of the form
where is the information, is the probability of the data given the model, and is a penalty for model complexity.

Matti asked me last week what the rationale behind choosing is. I'll write about the 2 widely used forms (Akaike Information Criterion (AIC) this time, and Bayesian Information Criterion (BIC) the next), but don't really know the answer to the weights of compared to either :)

Here's the intuition for AIC (the idea is presented in both MacKay and Bishop books): we have a set of models , and want to select the one with highest probability after seeing the data . This is given by If we take prior over models to be uniform, we just need to evaluate the evidence for each model.

Pick one model , and say it has tunable parameters. Select one of them, , and let's assume it's prior distribution is flat with width

We have

Suppose is sharply peaked around , and the width of the peak is Then the probability of will be all from that region, and given by , since the integral drops to 0 outside the peak. Combining that with the prior, and taking the log we get



Now repeating the similar argument over all parameters (taking integrals), we get



The first part of the expression is the fit of the model of the data, and the second part is a linear penalty of the number of parameters, scaled by the log of the fold-difference between the size of the prior and posterior parameter space.

The Gaussian

I did not go deep enough in my math studies as an undergrad to get a glimpse of calculus of variations. It looks fascinating, but I won't pick it up, at least not yet.

Calculus of variations can be used to show why the Gaussian is interesting. It's a limiting distribution of several families, sum of IID variables is approximately Gaussian - but it is also a distribution that conveys our ignorance of the data. Given a distribution has mean and variance , normal distribution is the one that conveys the least extra information, i.e, it has the maximum entropy among all possible .

The way to show it involves calculus of variations and Lagrange multipliers. We encode the 3 conditions of the distribution function (integrates to 1, mean and variance given), and combine it with the entropy in the Lagrangian:



Now differentiating with respect to , and finding the maximum, we get

, from where we instantly get the form of the Gaussian:



Completing the square and solving for the Lagrange multipliers using the constraints for mean and variance, we arrive at the Gaussian distribution. This holds similarly for the multivariate case.

So having a Gaussian as a prior distribution for observed data is equivalent to saying that we know nothing about the data except its mean and variance. Once again - cool :)

Ion channels are amazing

I just realised that so far it's been quite math heavy. So now for something different, inspired by Molecular Biology of the Cell (Alberts et. al), 5th edition.

Cells rely on different gradients (voltage, concentration) to drive processes. The gradients are maintained by active and passive transport across the membranes. K+ leak channel takes care of a part of them. The cool bit is in its selectivity - this channel conducts K+ 10000 times better than Na+, while both ions are almost uniform spheres of similar diameter.

Firstly, the channel is selective for cations by the virtue of negatively charged amino acids at its entrance. Then there is a maximum size limit of how fat of an ion can fit through, the skinny ones make it past the first hurdle into the vestibule. But how do you make sure smaller cations do not slip by?

All cations are associated with the polar water molecules. To get through the vestibule, the ion has to shed the water molecules. For K+, this is exactly balanced by bonds created with carbonyl oxygens. Na+, however, is too small to create the bonds, thus it is energetically unfavorable for it to pass on from the vestibule. Neat!

Least squares intuition

The ideas for these will probably keep coming from Chris Bishop's book. Today, I liked the intuition behind the least squares solution for the (a bit generalized) linear regression problem.

The general problem is this: given a set of data points , we want to find a predictor function of the form


that minimizes the mean-squared error

The are basis functions that allow for richer models, and the are weights of the basis. For ordinary linear regression, we can take , but in general, we can try to match the output with any basis functions - gaussians, sinusoids, sigmoids, etc.

Now, consider a -dimensional space whose axes are given by the regression targets . Then any basis function evaluated at the data points is also a point in this space:

If the number of basis functions is less than the number of data points , then the linear combinations of the basis function values define a linear subspace of .
Particularly, is a point in this subspace for any choice of .

Now for the cherry on the cake - the choice of weights that minimize error corresponds to the choice of that is the projection of the given data vector onto the subspace spanned by the basis functions.

Perhaps obvious (and proof omitted), but I thought it was nice that the world is consistent :)