Not Quite a Perfect Model Stack

In model building, the power of the majority can be a great thing. For those scholars of democracy, this does not refer to Alexis de Tocqueville’s tyranny in the power of majority. I apologize as that is probably a poor pun and may be a bit of a nerdy reference. Applying the power of the majority in machine learning allows for a model to combine the outputs of many learners. The concept of model stacking is built on ensemble modeling. Ensemble modeling usually combines many weak learners to arrive at a final prediction that is an improvement on any single weak learner. Random forest and gradient boosting machines employ methods of ensemble modeling. Stacking is an enhancement to the ensemble methods. Stacking combines the outputs of the ensemble models using some other learning algorithm to combine the predictions. Like any algorithm that is seeking to minimize error, the method used to stack the outputs of the ensemble models is itself trained to minimize the error of prediction. In this article, we’ll go over how to make a (not quite) perfect model stack.

[Related Article: Ensemble Models Demystified]

The above image lays out a simple example of a stacked model. The first step is to develop the…