stacking.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stacking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* * *\n",
    "![Alt text](./images/h2oEnsemble.png \"Super Learner\")\n",
    "* * *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction \n",
    "\n",
    "[Stacking](https://en.wikipedia.org/wiki/Ensemble_learning#Stacking) (sometimes called \"stacked generalization\") involves training a learning algorithm to combine the predictions of several other learning algorithms. First, all of the other algorithms are trained using the available data, then a combiner algorithm, the *metalearner*, is trained to make a final prediction using all the predictions of the other algorithms as additional inputs. If an arbitrary metalearning algorithm is used, then stacking can theoretically represent any of the ensemble techniques described in this article, although in practice, a single-layer [logistic regression](https://en.wikipedia.org/wiki/Logistic_regression) model is often used for metalearning.\n",
    "\n",
    "Stacking typically yields performance better than any single one of the trained models in the ensemble."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Background\n",
    "\n",
    "[Leo Breiman](https://en.wikipedia.org/wiki/Leo_Breiman), known for his work on classification and regression trees and the creator of the Random Forest algorithm, formalized stacking in his 1996 paper, [\"Stacked Regressions\"](http://statistics.berkeley.edu/sites/default/files/tech-reports/367.pdf). Although the idea originated with David Wolpert in 1992 under the name \"Stacked Generalization\", the modern form of stacking that uses internal k-fold cross-validation was Dr. Breiman's contribution.\n",
    "\n",
    "However, it wasn't until 2007 that the [theoretical background](http://dx.doi.org/10.2202/1544-6115.1309) for stacking was developed, which is when the algorithm took on the name, \"Super Learner\". Until this time, the mathematical reasons for why stacking worked were unknown and stacking was considered a \"black art.\" The Super Learner algorithm learns the optimal combination of the base learner fits. In an article titled, \"Super Learner\", by Mark van der Laan et al., proved that the Super Learner ensemble represents an asymptotically optimal system for learning.\n",
    "\n",
    "\n",
    "## Common Types of Ensemble Methods\n",
    "\n",
    "In statistics and machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained by any of the constituent algorithms.\n",
    "\n",
    "### Bagging\n",
    "\n",
    "- Reduces variance and increases accuracy\n",
    "- Robust against outliers or noisy data\n",
    "- Often used with Decision Trees (i.e. Random Forest)\n",
    "\n",
    "### Boosting\n",
    "\n",
    "- Also reduces variance and increases accuracy\n",
    "- Not robust against outliers or noisy data\n",
    "- Flexible - can be used with any loss function\n",
    "\n",
    "### Stacking\n",
    "\n",
    "- Used to ensemble a diverse group of strong learners\n",
    "- Involves training a second-level machine learning algorithm called a \"metalearner\" to learn the optimal combination of the base learners"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Super Learner Algorithm\n",
    "\n",
    "A common task in machine learning is to perform model selection by specifying a number of models with different parameters.  An example of this is Grid Search.  The first phase of the Super Learner algorithm is computationally equivalent to performing model selection via cross-validation.  The latter phase of the Super Learner algorithm (the metalearning step) is just training another single model (no cross-validation) on the *level one* data.  \n",
    "\n",
    "\n",
    "### 1. Set up the ensemble\n",
    "\n",
    "- Specify a list of $L$ base algorithms (with a specific set of model parameters).  These are also called *base learners*.\n",
    "- Specify a metalearning algorithm (just another algorithm).\n",
    "\n",
    "### 2. Train the ensemble\n",
    "\n",
    "#### Cross-validate Base Learners\n",
    "- Perform k-fold cross-validation on each of these learners and collect the cross-validated predicted values from each of the $L$ algorithms.\n",
    "- The $N$ cross-validated predicted values from each of the $L$ algorithms can be combined to form a new $N \\times L$ matrix. This matrix, along wtih the original response vector, is called the \"level-one\" data.\n",
    "\n",
    "![Alt text](./images/stacking_cv.png \"Stacking\")\n",
    "\n",
    "#### Metalearning\n",
    "- Train the metalearning algorithm on the level-one data.\n",
    "- Train each of the $L$ base algorithms on the full training set.\n",
    "- The \"ensemble model\" consists of the $L$ base learning models and the metalearning model, which can then be used to generate predictions on a test set.\n",
    "\n",
    "### 3. Predict on new data\n",
    "\n",
    "- To generate ensemble predictions, first generate predictions from the base learners.\n",
    "- Feed those predictions into the metalearner model to generate the ensemble prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stacking Software in R\n",
    "\n",
    "Stacking is a broad class of algorithms that involves training a second-level \"metalearner\" to ensemble a group of base learners.  The three packages in the R ecosystem which implement the Super Learner algorithm (stacking on cross-validated predictions) are [SuperLearner](https://github.com/ecpolley/SuperLearner), [subsemble](https://github.com/ledell/subsemble) and [h2oEnsemble](https://github.com/h2oai/h2o-3/tree/master/h2o-r/ensemble).  \n",
    "\n",
    "Among ensemble software in R, there is also [caretEnsemble](https://github.com/zachmayer/caretEnsemble), but it implements a boostrapped (rather than cross-validated) version of stacking via the `caretStack()` function.  The bootstrapped version will train faster since bootrapping (with a train/test) is a fraction of the work as k-fold cross-validation, however the the ensemble performance suffers as a result of this shortcut."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SuperLearner\n",
    "\n",
    "Authors: Eric Polley, Erin LeDell, Mark van der Laan\n",
    "\n",
    "Backend: R with constituent algorithms written in a variety of languages\n",
    "\n",
    "The original Super Learner implemenation is the [SuperLearner](https://github.com/ecpolley/SuperLearner) R package (2010).\n",
    "\n",
    "Features:\n",
    "- Implements the Super Learner prediction method (stacking) and contains a library of prediction algorithms to be used in the Super Learner.\n",
    "- Provides a clean interface to 30+ algorithms in R and defines a consistent API for extensibility.\n",
    "- GPL-3 Licensed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### subsemble\n",
    "\n",
    "Authors: Erin LeDell, Stephanie Sapp\n",
    "\n",
    "Backend: R with constituent algorithms written in a variety of languages\n”,
    "\n",
    "[Subsemble](https://github.com/ledell/subsemble) is a general subset ensemble prediction method, which can be used for small, moderate, or large datasets. Subsemble partitions the full dataset into subsets of observations, fits a specified underlying algorithm on each subset, and uses a unique form of k-fold cross-validation to output a prediction function that combines the subset-specific fits. An oracle result provides a theoretical performance guarantee for Subsemble. \n",
    "\n",
    "Features:\n",
    "- Implements the Subsemble Algorithm.\n",
    "- Implements the Super Learner Algorithm (stacking).\n",
    "- Uses the SuperLearner wrapper interface for defining base learners and metalearners.\n",
    "- Multicore and multi-node cluster support via the [snow](https://cran.r-project.org/web/packages/snow/index.html) R package.\n",
    "- Improved parallelization over the SuperLearner package.\n",
    "- Apache 2.0 Licensed.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### H2O Ensemble\n",
    "\n",
    "Authors: Erin LeDell\n",
    "\n",
    "Backend: Java\n",
    "\n",
    "H2O Ensemble has been implemented as a stand-alone R package called [h2oEnsemble](https://github.com/h2oai/h2o-3/tree/master/h2o-r/ensemble). The package is an extension to the [h2o](https://cran.r-project.org/web/packages/h2o/index.html) R package that allows the user to train an ensemble in the H2O cluster using any of the supervised machine learning algorithms H2O. \n",
    "\n",
    "Features:\n",
    "- Uses data-distributed and parallelized Java-based algorithms for the ensemble.\n",
    "- All training and data processing are performed in the high-performance H2O cluster rather than in R memory.\n",
    "- Supports regression and binary classification. \n",
    "- Multi-class support in development.\n",
    "- Code refactor  underway (moving from a separate R package into H2O \"proper\") so that the H2O Ensemble interface can be exposed in Python and Scala.\n",
    "- Apache 2.0 Licensed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Skipping install of 'h2oEnsemble' from a github remote, the SHA1 (612d4489) has not changed since last install.\n",
      "  Use `force = TRUE` to force installation\n"
     ]
    }
   ],
   "source": [
    "# Install from GitHub\n",
    "#install.packages(\"devtools\")\n",
    "#install.packages(\"h2o\")\n",
    "library(devtools)\n",
    "install_github(\"h2oai/h2o-3/h2o-r/ensemble/h2oEnsemble-package\")\n",
    "library(h2oEnsemble)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Higgs Demo\n",
    "\n",
    "This is an example of binary classification using the `h2o.ensemble()` function, the main stacking function in h2oEnsemble. \n",
    "\n",
    "This demo uses a subset of the [HIGGS dataset](https://archive.ics.uci.edu/ml/datasets/HIGGS), which has 28 numeric features and a binary response. The machine learning task in this example is to distinguish between a signal process which produces Higgs bosons (Y = 1) and a background process which does not (Y = 0). The dataset contains approximately the same number of positive vs negative examples. In other words, this is a balanced, rather than imbalanced, dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Start H2O Cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "H2O is not running yet, starting it now...\n",
      "\n",
      "Note:  In case of errors look at the following log files:\n",
      "    /var/folders/2j/jg4sl53d5q53tc2_nzm9fz5h0000gn/T//RtmpcfKpYn/h2o_me_started_from_r.out\n",
      "    /var/folders/2j/jg4sl53d5q53tc2_nzm9fz5h0000gn/T//RtmpcfKpYn/h2o_me_started_from_r.err\n",
      "\n",
      "\n",
      "Starting H2O JVM and connecting: . Connection successful!\n",
      "\n",
      "R is connected to the H2O cluster: \n",
      "    H2O cluster uptime:         1 seconds 306 milliseconds \n",
      "    H2O cluster version:        3.8.2.6 \n",
      "    H2O cluster name:           H2O_started_from_R_me_rdu560 \n",
      "    H2O cluster total nodes:    1 \n",
      "    H2O cluster total memory:   3.56 GB \n",
      "    H2O cluster total cores:    8 \n",
      "    H2O cluster allowed cores:  8 \n",
      "    H2O cluster healthy:        TRUE \n",
      "    H2O Connection ip:          localhost \n",
      "    H2O Connection port:        54321 \n",
      "    H2O Connection proxy:       NA \n",
      "    R Version:                  R version 3.3.0 (2016-05-03) \n",
      "\n"
     ]
    }
   ],
   "source": [
    "h2o.init(nthreads = -1)  # Start an H2O cluster with nthreads = num cores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Load Data into H2O Cluster\n",
    "\n",
    "First, import a sample binary outcome train and test set into the H2O cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n"
     ]
    }
   ],
   "source": [
    "# Import a sample binary outcome train/test set into R\n",
    "train <- h2o.importFile(\"data/higgs_train_10k.csv\")\n",
    "test <- h2o.importFile(\"data/higgs_test_5k.csv\")\n",
    "\n",
    "y <- \"response\"\n",
    "x <- setdiff(names(train), y)\n",
    "family <- \"binomial\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For binary classification, the response should be encoded as factor (also known as the [enum](https://docs.oracle.com/javase/tutorial/java/javaOO/enum.html) type in Java). The user can specify column types in the `h2o.importFile()` command, or you can convert the response column as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#For binary classification, response should be a factor\n",
    "train[,y] <- as.factor(train[,y])  \n",
    "test[,y] <- as.factor(test[,y])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Specify Base Learners & Metalearner"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this example, we will use the default base learner library for `h2o.ensemble`, which includes the default H2O GLM, Random Forest, GBM and Deep Neural Net (all using default model parameter values). We will also use the default metalearner, the H2O GLM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Specify the base learner library & the metalearner\n",
    "learner <- c(\"h2o.glm.wrapper\", \"h2o.randomForest.wrapper\", \n",
    "             \"h2o.gbm.wrapper\", \"h2o.deeplearning.wrapper\")\n",
    "metalearner <- \"h2o.deeplearning.wrapper\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Train an Ensemble"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the ensemble (using 5-fold internal CV) to generate the level-one data. Note that more CV folds will take longer to train, but should increase performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "  |                                                                            \r",
      "  |                                                                      |   0%\r",
      "  |                                                                            \r",
      "  |======================================================================| 100%\n",
      "[1] \"Cross-validating and training base learner 1: h2o.glm.wrapper\"\n",
      "\r",
      "  |                                                                            \r",
      "  |                                                                      |   0%\r",
      "  |                                                                            \r",
      "  |====================================                                  |  51%\r",
      "  |                                                                            \r",
      "  |======================================================================| 100%\n",
      "[1] \"Cross-validating and training base learner 2: h2o.randomForest.wrapper\"\n",
      "\r",
      "  |                                                                            \r",
      "  |                                                                      |   0%\r",
      "  |                                                                            \r",
      "  |==                                                                    |   3%\r",
      "  |                                                                            \r",
      "  |=====                                                                 |   7%\r",
      "  |                                                                            \r",
      "  |=======                                                               |  11%\r",
      "  |                                                                            \r",
      "  |==========                                                            |  14%\r",
      "  |                                                                            \r",
      "  |===========                                                           |  16%\r",
      "  |                                                                            \r",
      "  |============                                                          |  17%\r",
      "  |                                                                            \r",
      "  |===============                                                       |  22%\r",
      "  |                                                                            \r",
      "  |==================                                                    |  26%\r",
      "  |                                                                            \r",
      "  |=====================                                                 |  30%\r",
      "  |                                                                            \r",
      "  |=======================                                               |  33%\r",
      "  |                                                                            \r",
      "  |===========================                                           |  39%\r",
      "  |                                                                            \r",
      "  |==============================                                        |  43%\r",
      "  |                                                                            \r",
      "  |=================================                                     |  47%\r",
      "  |                                                                            \r",
      "  |===================================                                   |  50%\r",
      "  |                                                                            \r",
      "  |=======================================                               |  56%\r",
      "  |                                                                            \r",
      "  |==========================================                            |  60%\r",
      "  |                                                                            \r",
      "  |=============================================                         |  64%\r",
      "  |                                                                            \r",
      "  |===============================================                       |  67%\r",
      "  |                                                                            \r",
      "  |===================================================                   |  73%\r",
      "  |                                                                            \r",
      "  |======================================================                |  77%\r",
      "  |                                                                            \r",
      "  |========================================================              |  80%\r",
      "  |                                                                            \r",
      "  |==========================================================            |  83%\r",
      "  |                                                                            \r",
      "  |=============================================================         |  87%\r",
      "  |                                                                            \r",
      "  |================================================================      |  92%\r",
      "  |                                                                            \r",
      "  |===================================================================   |  96%\r",
      "  |                                                                            \r",
      "  |======================================================================|  99%\r",
      "  |                                                                            \r",
      "  |======================================================================| 100%\n",
      "[1] \"Cross-validating and training base learner 3: h2o.gbm.wrapper\"\n",
      "\r",
      "  |                                                                            \r",
      "  |                                                                      |   0%\r",
      "  |                                                                            \r",
      "  |====                                                                  |   6%\r",
      "  |                                                                            \r",
      "  |========                                                              |  12%\r",
      "  |                                                                            \r",
      "  |=============                                                         |  18%\r",
      "  |                                                                            \r",
      "  |=================                                                     |  25%\r",
      "  |                                                                            \r",
      "  |===========================                                           |  39%\r",
      "  |                                                                            \r",
      "  |===============================                                       |  45%\r",
      "  |                                                                            \r",
      "  |===================================                                   |  51%\r",
      "  |                                                                            \r",
      "  |=======================================                               |  56%\r",
      "  |                                                                            \r",
      "  |===============================================                       |  68%\r",
      "  |                                                                            \r",
      "  |====================================================                  |  74%\r",
      "  |                                                                            \r",
      "  |=======================================================               |  78%\r",
      "  |                                                                            \r",
      "  |=========================================================             |  82%\r",
      "  |                                                                            \r",
      "  |=============================================================         |  88%\r",
      "  |                                                                            \r",
      "  |==================================================================    |  94%\r",
      "  |                                                                            \r",
      "  |===================================================================== |  99%\r",
      "  |                                                                            \r",
      "  |======================================================================| 100%\n",
      "[1] \"Cross-validating and training base learner 4: h2o.deeplearning.wrapper\"\n",
      "\r",
      "  |                                                                            \r",
      "  |                                                                      |   0%\r",
      "  |                                                                            \r",
      "  |==                                                                    |   3%\r",
      "  |                                                                            \r",
      "  |=====                                                                 |   7%\r",
      "  |                                                                            \r",
      "  |=======                                                               |  11%\r",
      "  |                                                                            \r",
      "  |===========                                                           |  16%\r",
      "  |                                                                            \r",
      "  |=============                                                         |  19%\r",
      "  |                                                                            \r",
      "  |===============                                                       |  21%\r",
      "  |                                                                            \r",
      "  |===================                                                   |  27%\r",
      "  |                                                                            \r",
      "  |======================                                                |  32%\r",
      "  |                                                                            \r",
      "  |========================                                              |  35%\r",
      "  |                                                                            \r",
      "  |==========================                                            |  37%\r",
      "  |                                                                            \r",
      "  |=============================                                         |  41%\r",
      "  |                                                                            \r",
      "  |=================================                                     |  47%\r",
      "  |                                                                            \r",
      "  |===================================                                   |  51%\r",
      "  |                                                                            \r",
      "  |=====================================                                 |  53%\r",
      "  |                                                                            \r",
      "  |========================================                              |  57%\r",
      "  |                                                                            \r",
      "  |===========================================                           |  61%\r",
      "  |                                                                            \r",
      "  |================================================                      |  68%\r",
      "  |                                                                            \r",
      "  |=================================================                     |  69%\r",
      "  |                                                                            \r",
      "  |===================================================                   |  73%\r",
      "  |                                                                            \r",
      "  |=======================================================               |  79%\r",
      "  |                                                                            \r",
      "  |===========================================================           |  84%\r",
      "  |                                                                            \r",
      "  |=============================================================         |  87%\r",
      "  |                                                                            \r",
      "  |==============================================================        |  88%\r",
      "  |                                                                            \r",
      "  |================================================================      |  92%\r",
      "  |                                                                            \r",
      "  |===================================================================   |  95%\r",
      "  |                                                                            \r",
      "  |======================================================================| 100%\n",
      "[1] \"Metalearning\"\n",
      "\r",
      "  |                                                                            \r",
      "  |                                                                      |   0%\r",
      "  |                                                                            \r",
      "  |==============                                                        |  20%\r",
      "  |                                                                            \r",
      "  |============================                                          |  40%\r",
      "  |                                                                            \r",
      "  |=================================================                     |  70%\r",
      "  |                                                                            \r",
      "  |======================================================================| 100%\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "   user  system elapsed \n",
       "  2.377   0.108  83.434 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Train the ensemble using 5-fold CV to generate level-one data\n",
    "# More CV folds will take longer to train, but should increase performance\n",
    "system.time(fit <- h2o.ensemble(x = x,\n",
    "                                y = y,\n",
    "                                training_frame = train,\n",
    "                                family = family,\n",
    "                                learner = learner,\n",
    "                                metalearner = metalearner,\n",
    "                                cvControl = list(V = 5, shuffle = TRUE)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Prediction & Model Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `predict()` method for an `h2o.ensemble()` fit will return a list of two objects. The `pred$pred` object contains the ensemble predictions, and `pred$basepred` is a matrix of predictions from each of the base learners. \n",
    "\n",
    "However, a more useful function is `h2o.ensemble_performance()` which will compute the performance metrics for the base learners and ensemble on a test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "\n",
      "Base learner performance, sorted by specified metric:\n",
      "                   learner       AUC\n",
      "1          h2o.glm.wrapper 0.6871217\n",
      "4 h2o.deeplearning.wrapper 0.7272816\n",
      "2 h2o.randomForest.wrapper 0.7683471\n",
      "3          h2o.gbm.wrapper 0.7817065\n",
      "\n",
      "\n",
      "H2O Ensemble Performance on <newdata>:\n",
      "----------------\n",
      "Family: binomial\n",
      "\n",
      "Ensemble performance (AUC): 0.786520264970981\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Compute test set performance:\n",
    "perf <- h2o.ensemble_performance(fit, newdata = test)\n",
    "print(perf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Base learner performance, sorted by specified metric:\n",
      "                   learner       MSE\n",
      "1          h2o.glm.wrapper 0.2216865\n",
      "4 h2o.deeplearning.wrapper 0.2215248\n",
      "2 h2o.randomForest.wrapper 0.1972327\n",
      "3          h2o.gbm.wrapper 0.1900497\n",
      "\n",
      "\n",
      "H2O Ensemble Performance on <newdata>:\n",
      "----------------\n",
      "Family: binomial\n",
      "\n",
      "Ensemble performance (MSE): 0.190136448554782\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# To print the results for a particular metric, like MSE, do the following:\n",
    "print(perf, metric = \"MSE\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "0.786520264970981"
      ],
      "text/latex": [
       "0.786520264970981"
      ],
      "text/markdown": [
       "0.786520264970981"
      ],
      "text/plain": [
       "[1] 0.7865203"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# To access results directly in the ensemble performance object: \n",
    "perf$ensemble@metrics$AUC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Specifying Custom Learners\n",
    "\n",
    "Here is an example of how to generate a custom learner wrappers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Here is an example of how to generate a base learner library using custom base learners:\n",
    "\n",
    "h2o.randomForest.1 <- function(..., ntrees = 1000, nbins = 100, seed = 1) {\n",
    "  h2o.randomForest.wrapper(..., ntrees = ntrees, nbins = nbins, seed = seed)\n",
    "}\n",
    "\n",
    "h2o.deeplearning.1 <- function(..., hidden = c(500,500), activation = \"Rectifier\", seed = 1) {\n",
    "  h2o.deeplearning.wrapper(..., hidden = hidden, activation = activation, seed = seed)\n",
    "}\n",
    "\n",
    "h2o.deeplearning.2 <- function(..., hidden = c(200,200,200), activation = \"Tanh\", seed = 1) {\n",
    "  h2o.deeplearning.wrapper(..., hidden = hidden, activation = activation, seed = seed)\n",
    "}\n",
    "\n",
    "learner <- c(\"h2o.randomForest.1\", \"h2o.deeplearning.1\", \"h2o.deeplearning.2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stacking with Random Grids\n",
    "\n",
    "This example uses the technique of generating a random grid of base learners for maximum diversity, along with the `h2o.stack()` function.  The `h2o.stack()` function produces the exact same output as the `h2o.ensemble()` function, however you can use `h2o.stack()` to combine (via metalearning) a set of exsiting H2O models (where cross-validation was used and the predictions were saved)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n"
     ]
    }
   ],
   "source": [
    "#library(h2oEnsemble)\n",
    "#h2o.init(nthreads = -1)\n",
    "\n",
    "# Import a sample binary outcome train/test set into R\n",
    "train <- h2o.importFile(\"data/higgs_train_10k.csv\")\n",
    "test <- h2o.importFile(\"data/higgs_test_5k.csv\")\n",
    "\n",
    "y <- \"response\"\n",
    "x <- setdiff(names(train), y)\n",
    "family <- \"binomial\"\n",
    "\n",
    "#For binary classification, response should be a factor\n",
    "train[,y] <- as.factor(train[,y])  \n",
    "test[,y] <- as.factor(test[,y])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Random Grid Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Random Grid Search (e.g. 120 second maximum)\n",
    "# This is set to run fairly quickly, increase max_runtime_secs \n",
    "# or max_models to cover more of the hyperparameter space.\n",
    "# Also, you can expand the hyperparameter space of each of the \n",
    "# algorithms by modifying the hyper param code below.\n",
    "\n",
    "search_criteria <- list(strategy = \"RandomDiscrete\", \n",
    "                        max_runtime_secs = 120)\n",
    "nfolds <- 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### GBM Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "  |                                                                            \r",
      "  |                                                                      |   0%\r",
      "  |                                                                            \r",
      "  |=                                                                     |   1%\r",
      "  |                                                                            \r",
      "  |=                                                                     |   2%\r",
      "  |                                                                            \r",
      "  |==                                                                    |   3%\r",
      "  |                                                                            \r",
      "  |===                                                                   |   4%\r",
      "  |                                                                            \r",
      "  |====                                                                  |   5%\r",
      "  |                                                                            \r",
      "  |====                                                                  |   6%\r",
      "  |                                                                            \r",
      "  |=====                                                                 |   7%\r",
      "  |                                                                            \r",
      "  |=====                                                                 |   8%\r",
      "  |                                                                            \r",
      "  |======                                                                |   8%\r",
      "  |                                                                            \r",
      "  |=======                                                               |   9%\r",
      "  |                                                                            \r",
      "  |=======                                                               |  10%\r",
      "  |                                                                            \r",
      "  |========                                                              |  11%\r",
      "  |                                                                            \r",
      "  |========                                                              |  12%\r",
      "  |                                                                            \r",
      "  |=========                                                             |  13%\r",
      "  |                                                                            \r",
      "  |=========                                                             |  14%\r",
      "  |                                                                            \r",
      "  |==========                                                            |  14%\r",
      "  |                                                                            \r",
      "  |===========                                                           |  15%\r",
      "  |                                                                            \r",
      "  |===========                                                           |  16%\r",
      "  |                                                                            \r",
      "  |============                                                          |  17%\r",
      "  |                                                                            \r",
      "  |============                                                          |  18%\r",
      "  |                                                                            \r",
      "  |=============                                                         |  19%\r",
      "  |                                                                            \r",
      "  |==============                                                        |  19%\r",
      "  |                                                                            \r",
      "  |==============                                                        |  20%\r",
      "  |                                                                            \r",
      "  |===============                                                       |  21%\r",
      "  |                                                                            \r",
      "  |===============                                                       |  22%\r",
      "  |                                                                            \r",
      "  |================                                                      |  23%\r",
      "  |                                                                            \r",
      "  |=================                                                     |  24%\r",
      "  |                                                                            \r",
      "  |=================                                                     |  25%\r",
      "  |                                                                            \r",
      "  |==================                                                    |  25%\r",
      "  |                                                                            \r",
      "  |==================                                                    |  26%\r",
      "  |                                                                            \r",
      "  |===================                                                   |  27%\r",
      "  |                                                                            \r",
      "  |====================                                                  |  28%\r",
      "  |                                                                            \r",
      "  |====================                                                  |  29%\r",
      "  |                                                                            \r",
      "  |=====================                                                 |  30%\r",
      "  |                                                                            \r",
      "  |======================                                                |  31%\r",
      "  |                                                                            \r",
      "  |=======================                                               |  32%\r",
      "  |                                                                            \r",
      "  |=======================                                               |  33%\r",
      "  |                                                                            \r",
      "  |========================                                              |  34%\r",
      "  |                                                                            \r",
      "  |========================                                              |  35%\r",
      "  |                                                                            \r",
      "  |=========================                                             |  36%\r",
      "  |                                                                            \r",
      "  |==========================                                            |  37%\r",
      "  |                                                                            \r",
      "  |===========================                                           |  38%\r",
      "  |                                                                            \r",
      "  |===========================                                           |  39%\r",
      "  |                                                                            \r",
      "  |============================                                          |  40%\r",
      "  |                                                                            \r",
      "  |============================                                          |  41%\r",
      "  |                                                                            \r",
      "  |=============================                                         |  41%\r",
      "  |                                                                            \r",
      "  |==============================                                        |  42%\r",
      "  |                                                                            \r",
      "  |==============================                                        |  43%\r",
      "  |                                                                            \r",
      "  |===============================                                       |  44%\r",
      "  |                                                                            \r",
      "  |===============================                                       |  45%\r",
      "  |                                                                            \r",
      "  |================================                                      |  46%\r",
      "  |                                                                            \r",
      "  |=================================                                     |  47%\r",
      "  |                                                                            \r",
      "  |==================================                                    |  48%\r",
      "  |                                                                            \r",
      "  |==================================                                    |  49%\r",
      "  |                                                                            \r",
      "  |===================================                                   |  50%\r",
      "  |                                                                            \r",
      "  |====================================                                  |  51%\r",
      "  |                                                                            \r",
      "  |====================================                                  |  52%\r",
      "  |                                                                            \r",
      "  |=====================================                                 |  52%\r",
      "  |                                                                            \r",
      "  |=====================================                                 |  53%\r",
      "  |                                                                            \r",
      "  |======================================                                |  54%\r",
      "  |                                                                            \r",
      "  |=======================================                               |  55%\r",
      "  |                                                                            \r",
      "  |=======================================                               |  56%\r",
      "  |                                                                            \r",
      "  |========================================                              |  57%\r",
      "  |                                                                            \r",
      "  |========================================                              |  58%\r",
      "  |                                                                            \r",
      "  |=========================================                             |  58%\r",
      "  |                                                                            \r",
      "  |=========================================                             |  59%\r",
      "  |                                                                            \r",
      "  |==========================================                            |  60%\r",
      "  |                                                                            \r",
      "  |===========================================                           |  61%\r",
      "  |                                                                            \r",
      "  |===========================================                           |  62%\r",
      "  |                                                                            \r",
      "  |============================================                          |  63%\r",
      "  |                                                                            \r",
      "  |=============================================                         |  64%\r",
      "  |                                                                            \r",
      "  |==============================================                        |  65%\r",
      "  |                                                                            \r",
      "  |==============================================                        |  66%\r",
      "  |                                                                            \r",
      "  |===============================================                       |  67%\r",
      "  |                                                                            \r",
      "  |===============================================                       |  68%\r",
      "  |                                                                            \r",
      "  |================================================                      |  69%\r",
      "  |                                                                            \r",
      "  |=================================================                     |  69%\r",
      "  |                                                                            \r",
      "  |=================================================                     |  70%\r",
      "  |                                                                            \r",
      "  |==================================================                    |  71%\r",
      "  |                                                                            \r",
      "  |==================================================                    |  72%\r",
      "  |                                                                            \r",
      "  |===================================================                   |  73%\r",
      "  |                                                                            \r",
      "  |====================================================                  |  74%\r",
      "  |                                                                            \r",
      "  |=====================================================                 |  75%\r",
      "  |                                                                            \r",
      "  |=====================================================                 |  76%\r",
      "  |                                                                            \r",
      "  |======================================================                |  77%\r",
      "  |                                                                            \r",
      "  |=======================================================               |  78%\r",
      "  |                                                                            \r",
      "  |=======================================================               |  79%\r",
      "  |                                                                            \r",
      "  |========================================================              |  80%\r",
      "  |                                                                            \r",
      "  |=========================================================             |  81%\r",
      "  |                                                                            \r",
      "  |=========================================================             |  82%\r",
      "  |                                                                            \r",
      "  |==========================================================            |  83%\r",
      "  |                                                                            \r",
      "  |===========================================================           |  84%\r",
      "  |                                                                            \r",
      "  |===========================================================           |  85%\r",
      "  |                                                                            \r",
      "  |============================================================          |  85%\r",
      "  |                                                                            \r",
      "  |============================================================          |  86%\r",
      "  |                                                                            \r",
      "  |=============================================================         |  87%\r",
      "  |                                                                            \r",
      "  |==============================================================        |  88%\r",
      "  |                                                                            \r",
      "  |==============================================================        |  89%\r",
      "  |                                                                            \r",
      "  |===============================================================       |  90%\r",
      "  |                                                                            \r",
      "  |===============================================================       |  91%\r",
      "  |                                                                            \r",
      "  |================================================================      |  91%\r",
      "  |                                                                            \r",
      "  |=================================================================     |  92%\r",
      "  |                                                                            \r",
      "  |=================================================================     |  93%\r",
      "  |                                                                            \r",
      "  |==================================================================    |  94%\r",
      "  |                                                                            \r",
      "  |==================================================================    |  95%\r",
      "  |                                                                            \r",
      "  |===================================================================   |  96%\r",
      "  |                                                                            \r",
      "  |====================================================================  |  97%\r",
      "  |                                                                            \r",
      "  |===================================================================== |  98%\r",
      "  |                                                                            \r",
      "  |===================================================================== |  99%\r",
      "  |                                                                            \r",
      "  |======================================================================| 100%\n"
     ]
    }
   ],
   "source": [
    "# GBM Hyperparamters\n",
    "learn_rate_opt <- c(0.01, 0.03) \n",
    "max_depth_opt <- c(3, 4, 5, 6, 9)\n",
    "sample_rate_opt <- c(0.7, 0.8, 0.9, 1.0)\n",
    "col_sample_rate_opt <- c(0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8)\n",
    "hyper_params <- list(learn_rate = learn_rate_opt,\n",
    "                     max_depth = max_depth_opt, \n",
    "                     sample_rate = sample_rate_opt,\n",
    "                     col_sample_rate = col_sample_rate_opt)\n",
    "\n",
    "gbm_grid <- h2o.grid(\"gbm\", x = x, y = y,\n",
    "                     training_frame = train,\n",
    "                     ntrees = 100,\n",
    "                     seed = 1,\n",
    "                     nfolds = nfolds,\n",
    "                     fold_assignment = \"Modulo\",\n",
    "                     keep_cross_validation_predictions = TRUE,\n",
    "                     hyper_params = hyper_params,\n",
    "                     search_criteria = search_criteria)\n",
    "gbm_models <- lapply(gbm_grid@model_ids, function(model_id) h2o.getModel(model_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### RF Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "  |                                                                            \r",
      "  |                                                                      |   0%\r",
      "  |                                                                            \r",
      "  |=                                                                     |   1%\r",
      "  |                                                                            \r",
      "  |=                                                                     |   2%\r",
      "  |                                                                            \r",
      "  |==                                                                    |   3%\r",
      "  |                                                                            \r",
      "  |===                                                                   |   4%\r",
      "  |                                                                            \r",
      "  |====                                                                  |   5%\r",
      "  |                                                                            \r",
      "  |====                                                                  |   6%\r",
      "  |                                                                            \r",
      "  |=====                                                                 |   7%\r",
      "  |                                                                            \r",
      "  |=====                                                                 |   8%\r",
      "  |                                                                            \r",
      "  |======                                                                |   8%\r",
      "  |                                                                            \r",
      "  |=======                                                               |   9%\r",
      "  |                                                                            \r",
      "  |=======                                                               |  10%\r",
      "  |                                                                            \r",
      "  |========                                                              |  11%\r",
      "  |                                                                            \r",
      "  |========                                                              |  12%\r",
      "  |                                                                            \r",
      "  |=========                                                             |  13%\r",
      "  |                                                                            \r",
      "  |=========                                                             |  14%\r",
      "  |                                                                            \r",
      "  |==========                                                            |  14%\r",
      "  |                                                                            \r",
      "  |===========                                                           |  15%\r",
      "  |                                                                            \r",
      "  |===========                                                           |  16%\r",
      "  |                                                                            \r",
      "  |============                                                          |  17%\r",
      "  |                                                                            \r",
      "  |============                                                          |  18%\r",
      "  |                                                                            \r",
      "  |=============                                                         |  19%\r",
      "  |                                                                            \r",
      "  |==============                                                        |  19%\r",
      "  |                                                                            \r",
      "  |==============                                                        |  20%\r",
      "  |                                                                            \r",
      "  |===============                                                       |  21%\r",
      "  |                                                                            \r",
      "  |===============                                                       |  22%\r",
      "  |                                                                            \r",
      "  |================                                                      |  23%\r",
      "  |                                                                            \r",
      "  |=================                                                     |  24%\r",
      "  |                                                                            \r",
      "  |=================                                                     |  25%\r",
      "  |                                                                            \r",
      "  |==================                                                    |  25%\r",
      "  |                                                                            \r",
      "  |==================                                                    |  26%\r",
      "  |                                                                            \r",
      "  |===================                                                   |  27%\r",
      "  |                                                                            \r",
      "  |====================                                                  |  28%\r",
      "  |                                                                            \r",
      "  |====================                                                  |  29%\r",
      "  |                                                                            \r",
      "  |=====================                                                 |  30%\r",
      "  |                                                                            \r",
      "  |======================                                                |  31%\r",
      "  |                                                                            \r",
      "  |=======================                                               |  32%\r",
      "  |                                                                            \r",
      "  |=======================                                               |  33%\r",
      "  |                                                                            \r",
      "  |========================                                              |  34%\r",
      "  |                                                                            \r",
      "  |========================                                              |  35%\r",
      "  |                                                                            \r",
      "  |=========================                                             |  36%\r",
      "  |                                                                            \r",
      "  |==========================                                            |  36%\r",
      "  |                                                                            \r",
      "  |==========================                                            |  37%\r",
      "  |                                                                            \r",
      "  |===========================                                           |  38%\r",
      "  |                                                                            \r",
      "  |===========================                                           |  39%\r",
      "  |                                                                            \r",
      "  |============================                                          |  40%\r",
      "  |                                                                            \r",
      "  |============================                                          |  41%\r",
      "  |                                                                            \r",
      "  |=============================                                         |  42%\r",
      "  |                                                                            \r",
      "  |==============================                                        |  42%\r",
      "  |                                                                            \r",
      "  |==============================                                        |  43%\r",
      "  |                                                                            \r",
      "  |===============================                                       |  44%\r",
      "  |                                                                            \r",
      "  |===============================                                       |  45%\r",
      "  |                                                                            \r",
      "  |================================                                      |  46%\r",
      "  |                                                                            \r",
      "  |=================================                                     |  47%\r",
      "  |                                                                            \r",
      "  |==================================                                    |  48%\r",
      "  |                                                                            \r",
      "  |==================================                                    |  49%\r",
      "  |                                                                            \r",
      "  |===================================                                   |  50%\r",
      "  |                                                                            \r",
      "  |====================================                                  |  51%\r",
      "  |                                                                            \r",
      "  |====================================                                  |  52%\r",
      "  |                                                                            \r",
      "  |=====================================                                 |  53%\r",
      "  |                                                                            \r",
      "  |======================================                                |  54%\r",
      "  |                                                                            \r",
      "  |=======================================                               |  55%\r",
      "  |                                                                            \r",
      "  |=======================================                               |  56%\r",
      "  |                                                                            \r",
      "  |========================================                              |  57%\r",
      "  |                                                                            \r",
      "  |========================================                              |  58%\r",
      "  |                                                                            \r",
      "  |=========================================                             |  58%\r",
      "  |                                                                            \r",
      "  |=========================================                             |  59%\r",
      "  |                                                                            \r",
      "  |==========================================                            |  60%\r",
      "  |                                                                            \r",
      "  |===========================================                           |  61%\r",
      "  |                                                                            \r",
      "  |===========================================                           |  62%\r",
      "  |                                                                            \r",
      "  |============================================                          |  63%\r",
      "  |                                                                            \r",
      "  |============================================                          |  64%\r",
      "  |                                                                            \r",
      "  |=============================================                         |  64%\r",
      "  |                                                                            \r",
      "  |==============================================                        |  65%\r",
      "  |                                                                            \r",
      "  |==============================================                        |  66%\r",
      "  |                                                                            \r",
      "  |===============================================                       |  67%\r",
      "  |                                                                            \r",
      "  |===============================================                       |  68%\r",
      "  |                                                                            \r",
      "  |================================================                      |  69%\r",
      "  |                                                                            \r",
      "  |=================================================                     |  69%\r",
      "  |                                                                            \r",
      "  |=================================================                     |  70%\r",
      "  |                                                                            \r",
      "  |==================================================                    |  71%\r",
      "  |                                                                            \r",
      "  |==================================================                    |  72%\r",
      "  |                                                                            \r",
      "  |===================================================                   |  73%\r",
      "  |                                                                            \r",
      "  |====================================================                  |  74%\r",
      "  |                                                                            \r",
      "  |=====================================================                 |  75%\r",
      "  |                                                                            \r",
      "  |=====================================================                 |  76%\r",
      "  |                                                                            \r",
      "  |======================================================                |  77%\r",
      "  |                                                                            \r",
      "  |=======================================================               |  78%\r",
      "  |                                                                            \r",
      "  |=======================================================               |  79%\r",
      "  |                                                                            \r",
      "  |========================================================              |  80%\r",
      "  |                                                                            \r",
      "  |=========================================================             |  81%\r",
      "  |                                                                            \r",
      "  |=========================================================             |  82%\r",
      "  |                                                                            \r",
      "  |==========================================================            |  83%\r",
      "  |                                                                            \r",
      "  |===========================================================           |  84%\r",
      "  |                                                                            \r",
      "  |===========================================================           |  85%\r",
      "  |                                                                            \r",
      "  |============================================================          |  85%\r",
      "  |                                                                            \r",
      "  |============================================================          |  86%\r",
      "  |                                                                            \r",
      "  |=============================================================         |  87%\r",
      "  |                                                                            \r",
      "  |==============================================================        |  88%\r",
      "  |                                                                            \r",
      "  |==============================================================        |  89%\r",
      "  |                                                                            \r",
      "  |===============================================================       |  90%\r",
      "  |                                                                            \r",
      "  |===============================================================       |  91%\r",
      "  |                                                                            \r",
      "  |================================================================      |  91%\r",
      "  |                                                                            \r",
      "  |=================================================================     |  92%\r",
      "  |                                                                            \r",
      "  |=================================================================     |  93%\r",
      "  |                                                                            \r",
      "  |==================================================================    |  94%\r",
      "  |                                                                            \r",
      "  |==================================================================    |  95%\r",
      "  |                                                                            \r",
      "  |===================================================================   |  96%\r",
      "  |                                                                            \r",
      "  |====================================================================  |  96%\r",
      "  |                                                                            \r",
      "  |====================================================================  |  97%\r",
      "  |                                                                            \r",
      "  |===================================================================== |  98%\r",
      "  |                                                                            \r",
      "  |===================================================================== |  99%\r",
      "  |                                                                            \r",
      "  |======================================================================| 100%\n"
     ]
    }
   ],
   "source": [
    "# RF Hyperparamters\n",
    "mtries_opt <- 8:20 \n",
    "max_depth_opt <- c(5, 10, 15, 20, 25)\n",
    "sample_rate_opt <- c(0.7, 0.8, 0.9)\n",
    "col_sample_rate_per_tree_opt <- c(0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8)\n",
    "hyper_params <- list(mtries = mtries_opt,\n",
    "                     max_depth = max_depth_opt,\n",
    "                     sample_rate = sample_rate_opt,\n",
    "                     col_sample_rate_per_tree = col_sample_rate_per_tree_opt)\n",
    "\n",
    "rf_grid <- h2o.grid(\"randomForest\", x = x, y = y,\n",
    "                    training_frame = train,\n",
    "                    ntrees = 200,\n",
    "                    seed = 1,\n",
    "                    nfolds = nfolds,\n",
    "                    fold_assignment = \"Modulo\",\n",
    "                    keep_cross_validation_predictions = TRUE,                    \n",
    "                    hyper_params = hyper_params,\n",
    "                    search_criteria = search_criteria)\n",
    "rf_models <- lapply(rf_grid@model_ids, function(model_id) h2o.getModel(model_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Deeplearning Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "  |                                                                            \r",
      "  |                                                                      |   0%\r",
      "  |                                                                            \r",
      "  |=                                                                     |   1%\r",
      "  |                                                                            \r",
      "  |=                                                                     |   2%\r",
      "  |                                                                            \r",
      "  |==                                                                    |   3%\r",
      "  |                                                                            \r",
      "  |===                                                                   |   4%\r",
      "  |                                                                            \r",
      "  |====                                                                  |   5%\r",
      "  |                                                                            \r",
      "  |====                                                                  |   6%\r",
      "  |                                                                            \r",
      "  |=====                                                                 |   7%\r",
      "  |                                                                            \r",
      "  |=====                                                                 |   8%\r",
      "  |                                                                            \r",
      "  |======                                                                |   8%\r",
      "  |                                                                            \r",
      "  |=======                                                               |   9%\r",
      "  |                                                                            \r",
      "  |=======                                                               |  10%\r",
      "  |                                                                            \r",
      "  |========                                                              |  11%\r",
      "  |                                                                            \r",
      "  |========                                                              |  12%\r",
      "  |                                                                            \r",
      "  |=========                                                             |  13%\r",
      "  |                                                                            \r",
      "  |==========                                                            |  14%\r",
      "  |                                                                            \r",
      "  |===========                                                           |  15%\r",
      "  |                                                                            \r",
      "  |===========                                                           |  16%\r",
      "  |                                                                            \r",
      "  |============                                                          |  17%\r",
      "  |                                                                            \r",
      "  |============                                                          |  18%\r",
      "  |                                                                            \r",
      "  |=============                                                         |  19%\r",
      "  |                                                                            \r",
      "  |==============                                                        |  20%\r",
      "  |                                                                            \r",
      "  |===============                                                       |  21%\r",
      "  |                                                                            \r",
      "  |===============                                                       |  22%\r",
      "  |                                                                            \r",
      "  |================                                                      |  23%\r",
      "  |                                                                            \r",
      "  |=================                                                     |  24%\r",
      "  |                                                                            \r",
      "  |=================                                                     |  25%\r",
      "  |                                                                            \r",
      "  |==================                                                    |  25%\r",
      "  |                                                                            \r",
      "  |==================                                                    |  26%\r",
      "  |                                                                            \r",
      "  |===================                                                   |  27%\r",
      "  |                                                                            \r",
      "  |====================                                                  |  28%\r",
      "  |                                                                            \r",
      "  |====================                                                  |  29%\r",
      "  |                                                                            \r",
      "  |=====================                                                 |  30%\r",
      "  |                                                                            \r",
      "  |=====================                                                 |  31%\r",
      "  |                                                                            \r",
      "  |======================                                                |  31%\r",
      "  |                                                                            \r",
      "  |=======================                                               |  32%\r",
      "  |                                                                            \r",
      "  |=======================                                               |  33%\r",
      "  |                                                                            \r",
      "  |========================                                              |  34%\r",
      "  |                                                                            \r",
      "  |========================                                              |  35%\r",
      "  |                                                                            \r",
      "  |=========================                                             |  36%\r",
      "  |                                                                            \r",
      "  |==========================                                            |  36%\r",
      "  |                                                                            \r",
      "  |==========================                                            |  37%\r",
      "  |                                                                            \r",
      "  |===========================                                           |  38%\r",
      "  |                                                                            \r",
      "  |===========================                                           |  39%\r",
      "  |                                                                            \r",
      "  |============================                                          |  40%\r",
      "  |                                                                            \r",
      "  |=============================                                         |  41%\r",
      "  |                                                                            \r",
      "  |=============================                                         |  42%\r",
      "  |                                                                            \r",
      "  |==============================                                        |  42%\r",
      "  |                                                                            \r",
      "  |==============================                                        |  43%\r",
      "  |                                                                            \r",
      "  |===============================                                       |  44%\r",
      "  |                                                                            \r",
      "  |===============================                                       |  45%\r",
      "  |                                                                            \r",
      "  |================================                                      |  46%\r",
      "  |                                                                            \r",
      "  |=================================                                     |  47%\r",
      "  |                                                                            \r",
      "  |==================================                                    |  48%\r",
      "  |                                                                            \r",
      "  |==================================                                    |  49%\r",
      "  |                                                                            \r",
      "  |===================================                                   |  50%\r",
      "  |                                                                            \r",
      "  |====================================                                  |  51%\r",
      "  |                                                                            \r",
      "  |====================================                                  |  52%\r",
      "  |                                                                            \r",
      "  |=====================================                                 |  53%\r",
      "  |                                                                            \r",
      "  |======================================                                |  54%\r",
      "  |                                                                            \r",
      "  |=======================================                               |  55%\r",
      "  |                                                                            \r",
      "  |=======================================                               |  56%\r",
      "  |                                                                            \r",
      "  |========================================                              |  57%\r",
      "  |                                                                            \r",
      "  |========================================                              |  58%\r",
      "  |                                                                            \r",
      "  |=========================================                             |  59%\r",
      "  |                                                                            \r",
      "  |==========================================                            |  59%\r",
      "  |                                                                            \r",
      "  |==========================================                            |  60%\r",
      "  |                                                                            \r",
      "  |===========================================                           |  61%\r",
      "  |                                                                            \r",
      "  |===========================================                           |  62%\r",
      "  |                                                                            \r",
      "  |============================================                          |  63%\r",
      "  |                                                                            \r",
      "  |=============================================                         |  64%\r",
      "  |                                                                            \r",
      "  |==============================================                        |  65%\r",
      "  |                                                                            \r",
      "  |==============================================                        |  66%\r",
      "  |                                                                            \r",
      "  |===============================================                       |  67%\r",
      "  |                                                                            \r",
      "  |===============================================                       |  68%\r",
      "  |                                                                            \r",
      "  |================================================                      |  69%\r",
      "  |                                                                            \r",
      "  |=================================================                     |  70%\r",
      "  |                                                                            \r",
      "  |==================================================                    |  71%\r",
      "  |                                                                            \r",
      "  |==================================================                    |  72%\r",
      "  |                                                                            \r",
      "  |===================================================                   |  73%\r",
      "  |                                                                            \r",
      "  |====================================================                  |  74%\r",
      "  |                                                                            \r",
      "  |====================================================                  |  75%\r",
      "  |                                                                            \r",
      "  |=====================================================                 |  75%\r",
      "  |                                                                            \r",
      "  |=====================================================                 |  76%\r",
      "  |                                                                            \r",
      "  |======================================================                |  77%\r",
      "  |                                                                            \r",
      "  |=======================================================               |  78%\r",
      "  |                                                                            \r",
      "  |=======================================================               |  79%\r",
      "  |                                                                            \r",
      "  |========================================================              |  80%\r",
      "  |                                                                            \r",
      "  |========================================================              |  81%\r",
      "  |                                                                            \r",
      "  |=========================================================             |  81%\r",
      "  |                                                                            \r",
      "  |==========================================================            |  82%\r",
      "  |                                                                            \r",
      "  |==========================================================            |  83%\r",
      "  |                                                                            \r",
      "  |===========================================================           |  84%\r",
      "  |                                                                            \r",
      "  |===========================================================           |  85%\r",
      "  |                                                                            \r",
      "  |============================================================          |  86%\r",
      "  |                                                                            \r",
      "  |=============================================================         |  86%\r",
      "  |                                                                            \r",
      "  |=============================================================         |  87%\r",
      "  |                                                                            \r",
      "  |==============================================================        |  88%\r",
      "  |                                                                            \r",
      "  |==============================================================        |  89%\r",
      "  |                                                                            \r",
      "  |===============================================================       |  90%\r",
      "  |                                                                            \r",
      "  |================================================================      |  91%\r",
      "  |                                                                            \r",
      "  |================================================================      |  92%\r",
      "  |                                                                            \r",
      "  |=================================================================     |  92%\r",
      "  |                                                                            \r",
      "  |=================================================================     |  93%\r",
      "  |                                                                            \r",
      "  |==================================================================    |  94%\r",
      "  |                                                                            \r",
      "  |==================================================================    |  95%\r",
      "  |                                                                            \r",
      "  |===================================================================   |  96%\r",
      "  |                                                                            \r",
      "  |====================================================================  |  97%\r",
      "  |                                                                            \r",
      "  |====================================================================  |  98%\r",
      "  |                                                                            \r",
      "  |===================================================================== |  98%\r",
      "  |                                                                            \r",
      "  |===================================================================== |  99%\r",
      "  |                                                                            \r",
      "  |======================================================================| 100%\n"
     ]
    }
   ],
   "source": [
    "# Deeplearning Hyperparamters\n",
    "activation_opt <- c(\"Rectifier\", \"RectifierWithDropout\", \n",
    "                    \"Maxout\", \"MaxoutWithDropout\") \n",
    "hidden_opt <- list(c(10,10), c(20,15), c(50,50,50))\n",
    "l1_opt <- c(0, 1e-3, 1e-5)\n",
    "l2_opt <- c(0, 1e-3, 1e-5)\n",
    "hyper_params <- list(activation = activation_opt,\n",
    "                     hidden = hidden_opt,\n",
    "                     l1 = l1_opt,\n",
    "                     l2 = l2_opt)\n",
    "\n",
    "dl_grid <- h2o.grid(\"deeplearning\", x = x, y = y,\n",
    "                    training_frame = train,\n",
    "                    epochs = 15,\n",
    "                    seed = 1,\n",
    "                    nfolds = nfolds,\n",
    "                    fold_assignment = \"Modulo\",\n",
    "                    keep_cross_validation_predictions = TRUE,                    \n",
    "                    hyper_params = hyper_params,\n",
    "                    search_criteria = search_criteria)\n",
    "dl_models <- lapply(dl_grid@model_ids, function(model_id) h2o.getModel(model_id))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### GLM Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "  |                                                                            \r",
      "  |                                                                      |   0%\r",
      "  |                                                                            \r",
      "  |=                                                                     |   1%\r",
      "  |                                                                            \r",
      "  |=                                                                     |   2%\r",
      "  |                                                                            \r",
      "  |==                                                                    |   3%\r",
      "  |                                                                            \r",
      "  |===                                                                   |   4%\r",
      "  |                                                                            \r",
      "  |====                                                                  |   5%\r",
      "  |                                                                            \r",
      "  |====                                                                  |   6%\r",
      "  |                                                                            \r",
      "  |=====                                                                 |   7%\r",
      "  |                                                                            \r",
      "  |=====                                                                 |   8%\r",
      "  |                                                                            \r",
      "  |======                                                                |   8%\r",
      "  |                                                                            \r",
      "  |=======                                                               |   9%\r",
      "  |                                                                            \r",
      "  |=======                                                               |  10%\r",
      "  |                                                                            \r",
      "  |========                                                              |  11%\r",
      "  |                                                                            \r",
      "  |========                                                              |  12%\r",
      "  |                                                                            \r",
      "  |=========                                                             |  13%\r",
      "  |                                                                            \r",
      "  |=========                                                             |  14%\r",
      "  |                                                                            \r",
      "  |==========                                                            |  14%\r",
      "  |                                                                            \r",
      "  |==========                                                            |  15%\n"
     ]
    }
   ],
   "source": [
    "# GLM Hyperparamters\n",
    "alpha_opt <- seq(0,1,0.1)\n",
    "lambda_opt <- c(0,1e-7,1e-5,1e-3,1e-1)\n",
    "hyper_params <- list(alpha = alpha_opt,\n",
    "                     lambda = lambda_opt)\n",
    "\n",
    "glm_grid <- h2o.grid(\"glm\", x = x, y = y,\n",
    "                     training_frame = train,\n",
    "                     family = \"binomial\",\n",
    "                     nfolds = nfolds,\n",
    "                     fold_assignment = \"Modulo\",\n",
    "                     keep_cross_validation_predictions = TRUE,                    \n",
    "                     hyper_params = hyper_params,\n",
    "                     search_criteria = search_criteria)\n",
    "glm_models <- lapply(glm_grid@model_ids, function(model_id) h2o.getModel(model_id))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Specify Base Learners & Metalearner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create a list of all the base models\n",
    "models <- c(gbm_models, rf_models, dl_models, glm_models)\n",
    "\n",
    "# Specify a defalt GLM as the metalearner\n",
    "metalearner <- \"h2o.glm.wrapper\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] \"Metalearning\"\n",
      "\r",
      "  |                                                                            \r",
      "  |                                                                      |   0%\r",
      "  |                                                                            \r",
      "  |======                                                                |   8%\r",
      "  |                                                                            \r",
      "  |======================================================================| 100%\n"
     ]
    }
   ],
   "source": [
    "# Let's stack!\n",
    "stack <- h2o.stack(models = models, \n",
    "                   response_frame = train[,y],\n",
    "metalearner = metalearner)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "##### Compute the Performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "\n",
      "Base learner performance, sorted by specified metric:\n",
      "                                                                   learner\n",
      "78          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_15\n",
      "79          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_35\n",
      "80          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_53\n",
      "76          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_14\n",
      "77          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_29\n",
      "75          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_23\n",
      "74           Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_7\n",
      "23 Grid_DeepLearning_RTMP_sid_a34a_24_model_R_1467014209182_10118_model_17\n",
      "20 Grid_DeepLearning_RTMP_sid_a34a_24_model_R_1467014209182_10118_model_16\n",
      "73          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_48\n",
      "72           Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_5\n",
      "71           Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_9\n",
      "70          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_16\n",
      "36          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_11\n",
      "35          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_18\n",
      "34          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_41\n",
      "33          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_43\n",
      "32          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_26\n",
      "31          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_50\n",
      "30          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_45\n",
      "29          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_19\n",
      "28          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_42\n",
      "26          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_46\n",
      "27           Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_2\n",
      "42          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_31\n",
      "38          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_49\n",
      "43          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_28\n",
      "37          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_22\n",
      "50          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_47\n",
      "48           Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_6\n",
      "49          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_36\n",
      "52          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_38\n",
      "51          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_17\n",
      "39          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_32\n",
      "44          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_52\n",
      "46          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_20\n",
      "53           Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_4\n",
      "57          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_54\n",
      "58          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_27\n",
      "45          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_51\n",
      "59          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_21\n",
      "60          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_25\n",
      "61          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_39\n",
      "62           Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_3\n",
      "63          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_40\n",
      "64          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_30\n",
      "65          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_33\n",
      "66          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_34\n",
      "67          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_12\n",
      "68          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_24\n",
      "69          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_13\n",
      "54          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_37\n",
      "55           Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_1\n",
      "47          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_44\n",
      "40           Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_8\n",
      "56          Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_10\n",
      "41           Grid_GLM_RTMP_sid_a34a_24_model_R_1467014209182_10625_model_0\n",
      "19 Grid_DeepLearning_RTMP_sid_a34a_24_model_R_1467014209182_10118_model_12\n",
      "18  Grid_DeepLearning_RTMP_sid_a34a_24_model_R_1467014209182_10118_model_4\n",
      "14  Grid_DeepLearning_RTMP_sid_a34a_24_model_R_1467014209182_10118_model_9\n",
      "16  Grid_DeepLearning_RTMP_sid_a34a_24_model_R_1467014209182_10118_model_2\n",
      "25 Grid_DeepLearning_RTMP_sid_a34a_24_model_R_1467014209182_10118_model_15\n",
      "12 Grid_DeepLearning_RTMP_sid_a34a_24_model_R_1467014209182_10118_model_11\n",
      "13  Grid_DeepLearning_RTMP_sid_a34a_24_model_R_1467014209182_10118_model_1\n",
      "11  Grid_DeepLearning_RTMP_sid_a34a_24_model_R_1467014209182_10118_model_7\n",
      "15  Grid_DeepLearning_RTMP_sid_a34a_24_model_R_1467014209182_10118_model_8\n",
      "24  Grid_DeepLearning_RTMP_sid_a34a_24_model_R_1467014209182_10118_model_0\n",
      "17  Grid_DeepLearning_RTMP_sid_a34a_24_model_R_1467014209182_10118_model_5\n",
      "22 Grid_DeepLearning_RTMP_sid_a34a_24_model_R_1467014209182_10118_model_10\n",
      "21 Grid_DeepLearning_RTMP_sid_a34a_24_model_R_1467014209182_10118_model_13\n",
      "9   Grid_DeepLearning_RTMP_sid_a34a_24_model_R_1467014209182_10118_model_6\n",
      "3             Grid_GBM_RTMP_sid_a34a_24_model_R_1467014209182_2066_model_4\n",
      "10 Grid_DeepLearning_RTMP_sid_a34a_24_model_R_1467014209182_10118_model_14\n",
      "4             Grid_GBM_RTMP_sid_a34a_24_model_R_1467014209182_2066_model_1\n",
      "8   Grid_DeepLearning_RTMP_sid_a34a_24_model_R_1467014209182_10118_model_3\n",
      "5             Grid_GBM_RTMP_sid_a34a_24_model_R_1467014209182_2066_model_3\n",
      "2             Grid_GBM_RTMP_sid_a34a_24_model_R_1467014209182_2066_model_0\n",
      "6             Grid_DRF_RTMP_sid_a34a_24_model_R_1467014209182_8155_model_1\n",
      "7             Grid_DRF_RTMP_sid_a34a_24_model_R_1467014209182_8155_model_0\n",
      "1             Grid_GBM_RTMP_sid_a34a_24_model_R_1467014209182_2066_model_2\n",
      "         AUC\n",
      "78 0.5000000\n",
      "79 0.5000000\n",
      "80 0.5000000\n",
      "76 0.5389248\n",
      "77 0.5389345\n",
      "75 0.5417750\n",
      "74 0.5822301\n",
      "23 0.5960455\n",
      "20 0.6108904\n",
      "73 0.6154459\n",
      "72 0.6354336\n",
      "71 0.6506383\n",
      "70 0.6610600\n",
      "36 0.6863669\n",
      "35 0.6863711\n",
      "34 0.6864484\n",
      "33 0.6865243\n",
      "32 0.6865479\n",
      "31 0.6866241\n",
      "30 0.6867065\n",
      "29 0.6867256\n",
      "28 0.6867815\n",
      "26 0.6868012\n",
      "27 0.6868321\n",
      "42 0.6870867\n",
      "38 0.6870935\n",
      "43 0.6870962\n",
      "37 0.6871113\n",
      "50 0.6871120\n",
      "48 0.6871134\n",
      "49 0.6871156\n",
      "52 0.6871173\n",
      "51 0.6871189\n",
      "39 0.6871217\n",
      "44 0.6871233\n",
      "46 0.6871237\n",
      "53 0.6871253\n",
      "57 0.6871295\n",
      "58 0.6871317\n",
      "45 0.6871319\n",
      "59 0.6871365\n",
      "60 0.6871365\n",
      "61 0.6871365\n",
      "62 0.6871365\n",
      "63 0.6871365\n",
      "64 0.6871365\n",
      "65 0.6871365\n",
      "66 0.6871365\n",
      "67 0.6871365\n",
      "68 0.6871365\n",
      "69 0.6871365\n",
      "54 0.6871369\n",
      "55 0.6871384\n",
      "47 0.6871406\n",
      "40 0.6871426\n",
      "56 0.6871431\n",
      "41 0.6871481\n",
      "19 0.6910824\n",
      "18 0.6950987\n",
      "14 0.6951873\n",
      "16 0.6968837\n",
      "25 0.7026618\n",
      "12 0.7055578\n",
      "13 0.7126669\n",
      "11 0.7139075\n",
      "15 0.7154103\n",
      "24 0.7230679\n",
      "17 0.7238602\n",
      "22 0.7263660\n",
      "21 0.7298324\n",
      "9  0.7394220\n",
      "3  0.7396723\n",
      "10 0.7409898\n",
      "4  0.7419225\n",
      "8  0.7481582\n",
      "5  0.7516993\n",
      "2  0.7552767\n",
      "6  0.7619598\n",
      "7  0.7633915\n",
      "1  0.7790918\n",
      "\n",
      "\n",
      "H2O Ensemble Performance on <newdata>:\n",
      "----------------\n",
      "Family: binomial\n",
      "\n",
      "Ensemble performance (AUC): 0.788529105381067\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Compute test set performance:\n",
    "perf <- h2o.ensemble_performance(stack, newdata = test)\n",
    "print(perf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Refit the Metalearner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] \"Metalearning\"\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "\n",
      "\n",
      "H2O Ensemble Performance on <newdata>:\n",
      "----------------\n",
      "Family: binomial\n",
      "\n",
      "Ensemble performance (AUC): 0.777667064847103\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Now let's refit the metalearner using a DL and GLM-NN\n",
    "# Note: DL is not usually the best metalearner\n",
    "\n",
    "stack2 <- h2o.metalearn(stack, metalearner = \"h2o.deeplearning.wrapper\")\n",
    "perf2 <- h2o.ensemble_performance(stack2, newdata = test, score_base_models = FALSE)\n",
    "print(perf2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Restrict to Non-Negative Weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] \"Metalearning\"\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "  |======================================================================| 100%\n",
      "\n",
      "\n",
      "H2O Ensemble Performance on <newdata>:\n",
      "----------------\n",
      "Family: binomial\n",
      "\n",
      "Ensemble performance (AUC): 0.786483342785091\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# It's always a good idea to try a GLM restricted to non-negative weights as a metalearner\n",
    "# There have been a lot of empircal studies that show that non-negative weights can lead to better performance\n",
    "\n",
    "h2o.glm_nn <- function(..., non_negative = TRUE) h2o.glm.wrapper(..., non_negative = non_negative)\n",
    "stack3 <- h2o.metalearn(stack, metalearner = \"h2o.glm_nn\")\n",
    "perf3 <- h2o.ensemble_performance(stack3, newdata = test, score_base_models = FALSE)\n",
    "print(perf3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The test set of the top base learner was 0.779, where as the best ensemble got 0.788.  The gain is more than enough to jump 100+ spots on a Kaggle competition. :-)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.3.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}