Recurrent Neural Networks/LSTM-Language-Modelling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Applying LSTM for Language Modelling\n",
    "In this notebook, we will go over the topic of what Language Modelling is and create a Recurrent Neural Network model based on the Long Short-Term Memory unit to train and be benchmarked by the Penn Treebank."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Objective\n",
    "By now, you should have an understanding of how Recurrent Networks work - a specialized model to process sequential data by keeping track of the \"state\" or context. In this notebook, we go over a TensorFlow code snippet for creating a model focused on **Language Modelling** - a very relevant task that is the cornerstone of many different linguistic problems such as **Speech Recognition, Machine Translation and Image Captioning**. For this, we will be using the Penn Treebank, which is an often-used dataset for benchmarking Language Modelling models.\n",
    "\n",
    "## What exactly is Language Modelling?\n",
    "Language Modelling, to put it simply, **is the task of assigning probabilities to sequences of words**. This means that, given a context of one or a few words in the language the model was trained on, the model should have a knowledge of what are the most probable words or sequence of words for the sentence. Language Modelling is one of the tasks under Natural Language Processing, and one of the most important.\n",
    "\n",
    "<img src=\"https://ibm.box.com/shared/static/1d1i5gub6wljby2vani2vzxp0xsph702.png\"/>\n",
    "<center>*Example of a sentence being predicted*</center>\n",
    "\n",
    "In this example, one can see the predictions for the next word of a sentence, given the context \"This is an\". As you can see, this boils down to a sequential data analysis task - you are given a word or a sequence of words (the input data), and, given the context (the state), you need to find out what is the next word (the prediction). This kind of analysis is very important for language-related tasks such as **Speech Recognition, Machine Translation, Image Captioning, Text Correction** and many other very relevant problems. \n",
    "\n",
    "<img src=\"https://ibm.box.com/shared/static/az39idf9ipfdpc5ugifpgxnydelhyf3i.png\"/>\n",
    "<center>*The above example schematized as an RNN in execution*</center>\n",
    "\n",
    "As the above image shows, Recurrent Network models fit this problem like a glove. Alongside LSTM and its capacity to maintain the model's state for over one thousand time steps, we have all the tools we need to undertake this problem. The goal is to create a model that can reach **low levels of perplexity** on our desired dataset.\n",
    "\n",
    "For Language Modelling problems, **perplexity** is the way to gauge efficiency. Perplexity is simply a measure of how well a probabilistic model is able to predict its sample. A higher-level way to explain this would be saying that **low perplexity means a higher degree of trust in the predictions the model makes**. Therefore, the lower perplexity is, the better.\n",
    "\n",
    "## The Penn Treebank dataset\n",
    "Historically, datasets big enough for Natural Language Processing are hard to come by. This is in part due to the necessity of the sentences to be broken down and tagged with a certain degree of correctness - or else the models trained on it won't be able to be correct at all. This means that we need a **large amount of data, annotated by or at least corrected by humans**. This is, of course, not an easy task at all.\n",
    "\n",
    "The Penn Treebank, or PTB for short, is a dataset maintained by the University of Pennsylvania. It is *huge* - there are over **four million and eight hundred thousand** annotated words in it, all corrected by humans. It is composed of many different sources, from abstracts of Department of Energy papers to texts from the Library of America. Since it is verifiably correct and of such a huge size, the Penn Treebank has been used time and time again as a benchmark dataset for Language Modelling.\n",
    "\n",
    "The dataset is divided in different kinds of annotations, such as Piece-of-Speech, Syntactic and Semantic skeletons. For this example, we will simply use a sample of clean, non-annotated words (with the exception of one tag - `<unk>`, which is used for rare words such as uncommon proper nouns) for our model. This means that we just want to predict what the next words would be, not what they mean in context or their classes on a given sentence. \n",
    "<br/>\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "the percentage of lung cancer deaths among the workers at the west `<unk>` mass. paper factory appears to be the highest for any asbestos workers studied in western industrialized countries he said \n",
    " the plant which is owned by `<unk>` & `<unk>` co. was under contract with `<unk>` to make the cigarette filters \n",
    " the finding probably will support those who argue that the u.s. should regulate the class of asbestos including `<unk>` more `<unk>` than the common kind of asbestos `<unk>` found in most schools and other buildings dr. `<unk>` said\n",
    "</div>\n",
    "<center>*Example of text from the dataset we are going to use, `ptb.train`*</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Word Embeddings</h2><br/>\n",
    "\n",
    "For better processing, in this example, we will make use of [**word embeddings**]( [https://www.tensorflow.org/tutorials/word2vec/), which are **a way of representing sentence structures or words as n-dimensional vectors (where n is a reasonably high number, such as 200 or 500) of real numbers**. Basically, we will assign each word a randomly-initialized vector, and input those into the network to be processed. After a number of iterations, these vectors are expected to assume values that help the network to correctly predict what it needs to - in our case, the probable next word in the sentence. This is shown to be very effective in Natural Language Processing tasks, and is a commonplace practice.\n",
    "<br/><br/>\n",
    "<strong>\n",
    "$$Vec(\"Example\") = [0.02, 0.00, 0.00, 0.92, 0.30,...]$$\n",
    "</strong>\n",
    "<br/>\n",
    "Word Embedding tends to group up similarly used words *reasonably* together in the vectorial space. For example, if we use T-SNE (a dimensional reduction visualization algorithm) to flatten the dimensions of our vectors into a 2-dimensional space and use the words these vectors represent as their labels, we might see something like this:\n",
    "\n",
    "<img src=\"https://ibm.box.com/shared/static/bqhc5dg879gcoabzhxra1w8rkg3od1cu.png\" width=\"800\"/>\n",
    "<center>*T-SNE Mockup with clusters marked for easier visualization*</center>\n",
    "\n",
    "As you can see, words that are frequently used together, in place of each other, or in the same places as them tend to be grouped together - being closer together the higher these correlations are. For example, \"None\" is pretty semantically close to \"Zero\", while a phrase that uses \"Italy\" can probably also fit \"Germany\" in it, with little damage to the sentence structure. A vectorial \"closeness\" for similar words like this is a great indicator of a well-built model.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to import the necessary modules for our code. We need **`numpy` and `tensorflow`**, obviously. Additionally, we can import directly the **`tensorflow.models.rnn.rnn`** model, which includes the function for building RNNs, and **`tensorflow.models.rnn.ptb.reader`** which is the helper module for getting the input data from the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "\n",
    "import time\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from resources.ptb import reader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the LSTM model for Language Modeling\n",
    "Now that we know exactly what we are doing, we can start building our model using TensorFlow. Additionally, for the sake of making it easy to play around with the model's hyperparameters, we can declare them beforehand."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initial weight scale.\n",
    "init_scale = 0.1\n",
    "# Initial learning rate.\n",
    "learning_rate = 1.0\n",
    "# Maximum permissible norm for the gradient (For gradient clipping - another measure against Exploding Gradients).\n",
    "max_grad_norm = 5\n",
    "# The number of layers in our model.\n",
    "num_layers = 2\n",
    "# The total number of recurrence steps, also known as the number of layers when our RNN is \"unfolded\".\n",
    "num_steps = 20\n",
    "# The number of processing units (neurons) in the hidden layers.\n",
    "hidden_size = 200\n",
    "# The maximum number of epochs trained with the initial learning rate.\n",
    "max_epoch = 4\n",
    "# The total number of epochs in training.\n",
    "max_max_epoch = 13\n",
    "# The probability for keeping data in the Dropout Layer (this is an optimization).\n",
    "# At 1, we ignore the Dropout Layer wrapping.\n",
    "keep_prob = 1\n",
    "# The decay for the learning rate.\n",
    "decay = 0.5\n",
    "# The size for each batch of data.\n",
    "batch_size = 30\n",
    "# The size of our vocabulary.\n",
    "vocab_size = 10000\n",
    "# Training flag to separate training from testing.\n",
    "is_training = 1\n",
    "# Data directory for our dataset.\n",
    "data_dir = \"./resources/data/simple-examples/data/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some clarifications for LSTM architecture based on the argumants:\n",
    "\n",
    "Network structure:\n",
    "- In this network, the number of LSTM cells are 2. To give the model more expressive power, we can add multiple layers of LSTMs to process the data. The output of the first layer will become the input of the second and so on.\n",
    "- The recurrence steps are 20, that is, when our RNN is \"Unfolded\", the recurrence step is 20.   \n",
    "- the structure is like: \n",
    "     - 200 input units -> [200x200] Weight -> 200 Hidden units (first layer) -> [200x200] Weight matrix  -> 200 Hidden units (second layer) ->  [200] weight Matrix -> 200 unit output\n",
    "\n",
    "Hidden layer:\n",
    "- Each LSTM has 200 hidden units which is equivalant to the dimensianality of the embedding words and output. \n",
    "\n",
    "Input layer: \n",
    "- The network has 200 input units. \n",
    "- Suppose each word is represented by an embedding vector of dimensionality e = 200. The input layer of each cell will have 200 linear units. These e = 200 linear units are connected to each of the h = 200 LSTM units in the hidden layer (assuming there is only one hidden layer, though our case has 2 layers).\n",
    "- The input shape is [batch_size, num_steps], that is [30x20]. It will turn into [30x20x200] after embedding, and then 20x[30x200]\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code is adapted from the PTBModel example bundled with the TensorFlow source code.  \n",
    "#### Train data\n",
    "The story starts from data: \n",
    "- Train data is a list of words, represented by numbers - N = 929589 numbers, e.g. [9971, 9972, 9974, 9975,...]\n",
    "- We read data as mini-batch of size b = 30. Assume the size of each sentence is 20 words (num_steps = 20). Then it will take int(N / b * h) + 1 = 1548 iterations for the learner to go through all sentences once. So, the number of iterators is 1548\n",
    "- Each batch data is read from train dataset of size 600, and shape of [30x20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we start an interactive session:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "session = tf.InteractiveSession()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reads the data and separates it into training data, validation data and testing data.\n",
    "raw_data = reader.ptb_raw_data(data_dir)\n",
    "train_data, valid_data, test_data, _, _ = raw_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Lets just read one mini-batch now and feed our network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "itera = reader.ptb_iterator(train_data, batch_size, num_steps)\n",
    "first_tuple = next(itera)\n",
    "x = first_tuple[0]\n",
    "y = first_tuple[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(30, 20)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets look at 3 sentences of our input x:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[9970, 9971, 9972, 9974, 9975, 9976, 9980, 9981, 9982, 9983, 9984,\n",
       "        9986, 9987, 9988, 9989, 9991, 9992, 9993, 9994, 9995],\n",
       "       [2654,    6,  334, 2886,    4,    1,  233,  711,  834,   11,  130,\n",
       "         123,    7,  514,    2,   63,   10,  514,    8,  605],\n",
       "       [   0, 1071,    4,    0,  185,   24,  368,   20,   31, 3109,  954,\n",
       "          12,    3,   21,    2, 2915,    2,   12,    3,   21]],\n",
       "      dtype=int32)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[0:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "size = hidden_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define 2 placeholders to feed them with mini-batches, that is x and y:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "_input_data = tf.placeholder(tf.int32, [batch_size, num_steps])  # [30#20]\n",
    "_targets = tf.placeholder(tf.int32, [batch_size, num_steps])  # [30#20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets define a dictionary, and use it later to feed the placeholders with our first mini-batch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "feed_dict = {_input_data: x, _targets: y}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, we can use it to feed _input_data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[9970, 9971, 9972, 9974, 9975, 9976, 9980, 9981, 9982, 9983, 9984,\n",
       "        9986, 9987, 9988, 9989, 9991, 9992, 9993, 9994, 9995],\n",
       "       [2654,    6,  334, 2886,    4,    1,  233,  711,  834,   11,  130,\n",
       "         123,    7,  514,    2,   63,   10,  514,    8,  605],\n",
       "       [   0, 1071,    4,    0,  185,   24,  368,   20,   31, 3109,  954,\n",
       "          12,    3,   21,    2, 2915,    2,   12,    3,   21],\n",
       "       [   3,   71,    4,   27,  246,   60,   11,  215,    4,    1, 1846,\n",
       "           9,    3,   71,  546,    2, 6505,  162,    6,  104],\n",
       "       [  93,   25,    6,  261,  681,  251,    0,  278, 3246,   13,  200,\n",
       "           1,    8,  105, 3360,    1,    4,    0,  536,    4],\n",
       "       [  20,    6,  954,   12,    3,   21,   78,   14,  977,  726,    0,\n",
       "          37,   42,   34,    5,  437,  116,  206,  927,    2],\n",
       "       [  18,  296,    7,  201,   76,    4,  182,  560, 3836,   17,  974,\n",
       "         975,    6,  942,    4,  156,   77, 1570,  288,  644],\n",
       "       [  23, 1238,  899,    5,   25,  201,    4,    0,  434,  642,   55,\n",
       "         201,    4,    0, 2423,    2,    1,    1,    1,  483],\n",
       "       [ 379,  706,    9,  413, 8219,   96,   15,    0, 2185, 1758,    1,\n",
       "           1,   37,   13,  834,    5,  852,  222,    7, 1785],\n",
       "       [   2,  179,  940,  117,   38,   59,  677,   14,    1,   10, 1016,\n",
       "         309,   13, 1077, 6360,   16,   23, 4490,    9,  355],\n",
       "       [3572,    4, 3015, 1347,  536,   13,    6, 3949,    5,  438, 9643,\n",
       "           2,   64,   87,   32,  358, 3672, 4103, 1082,   11],\n",
       "       [  71,  178,    3,    8,    3,    2,    0, 1008,  234,   30, 6400,\n",
       "          10,    0,   98,    9,    1,  338,   13,    5,   25],\n",
       "       [1473,   88,   19, 2578, 6591,    8,  629,  563,    8,  223,  184,\n",
       "         127,   18,    6,  828,    1,    2,    0,  324,  158],\n",
       "       [   1,    1,    2,   18,    0, 1844,    4,   73,   39, 2694,    6,\n",
       "        1709,    2,    7,    0, 6509, 1116,   27,    1,    1],\n",
       "       [1055,    5,   25, 8582,   10,  353,  645,   24,    6,  287,    2,\n",
       "        1006,    0, 8861, 2369,   44,    7,    0,    1,  180],\n",
       "       [  36,  501,    5,    6, 1969,    0,   98,   89, 2254,    0,  312,\n",
       "        1641,    4, 1063,    8,  713,    0,  264,  820,    2],\n",
       "       [  32, 2599,  762, 1875,   26, 1402,   45,  516,    2, 2937,   16,\n",
       "        3355, 2062,  251,    0,  529,   24, 1625,  122,   18],\n",
       "       [ 677,  127,    2,   19,   23, 7800, 3592,   14,   64,   87,   32,\n",
       "         350,    0, 3968,    2,   38,   26,  114,   38,   26],\n",
       "       [  25,   45,  769,    2,   23, 2634, 1096, 1175,   19,    6,    1,\n",
       "         154,   23, 1890,   30,    6,    1,    1,    2,  198],\n",
       "       [7736,  391,    5, 5173,  838,    2,  840,    9, 8716,  537, 4132,\n",
       "        2915,    9,    1,    1,   10, 1268,  175,   32,  184],\n",
       "       [   3,   21,    4,    1,  308,  458,   11,   41,   14, 5718,  102,\n",
       "         824,    1,    2,   14,   59,   50,   12,    3,   21],\n",
       "       [   8,    1,   22,   73,   10,  863,   11,  898,  653,  270,    8,\n",
       "         500,  273, 1559,    2,   14, 3019,    5,  585,   84],\n",
       "       [ 483,  762,   87,  108, 1119,    0,    1,   67,    0, 3296,   26,\n",
       "         591,  174,  127,    2,  108,   26, 9821,   11,    6],\n",
       "       [3885,  582,   81,   17, 1834,    2, 1256,   98,  162,  582,  441,\n",
       "         125,   22, 1652,  172,    4,    3,    3,    8,  206],\n",
       "       [  44,   23,    1,    0, 1704,    4,    1,    2,   22,  373,   38,\n",
       "         275,    1, 8017,    2, 2785, 3659, 4359,   80,  634],\n",
       "       [1896,    8,   13, 9468,   17,  752, 4622,    2,   29, 2221,    0,\n",
       "         446, 3552,    4,    0, 2495,  431,  134,  284,  152],\n",
       "       [  48,    7, 1741,  193,    8,  446,  165,  301, 6521, 5122,   15,\n",
       "          12,    3,   21,    4,   10,  161,  783,    8,   79],\n",
       "       [  47, 4447, 1431,    4, 6967, 2121,   24,  452,   18,   43,    3,\n",
       "          48, 1076,   12,    3,   21,   69,   40,    2, 1323],\n",
       "       [  31, 3374,    4, 2108,    1,  134,    8, 6967, 1825, 3306,   14,\n",
       "          13, 3581,    5, 2424, 1583, 6495,    5,    6, 1136],\n",
       "       [  59, 2070, 2433,   28,  517,   20,   23, 4306,    6,   40,  195,\n",
       "           2, 9398,  400, 4908,  673, 1572,  400,    1, 1173]],\n",
       "      dtype=int32)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session.run(_input_data, feed_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this step, we create the stacked LSTM, which is a 2 layer LSTM network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "lstm_cell = tf.contrib.rnn.BasicLSTMCell(hidden_size, forget_bias=0.0)\n",
    "stacked_lstm = tf.contrib.rnn.MultiRNNCell([lstm_cell] * num_layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, we initialize the states of the nework:\n",
    "\n",
    "#### _initial_state\n",
    "\n",
    "For each LSTM, there are 2 state matrics, c_state and m_state.  c_state and m_state represent \"Memory State\" and \"Cell State\". Each hidden layer, has a vector of size 30, which keeps the states. So, for 200 hidden units in each LSTM, we have a matrix of size [30x200]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(LSTMStateTuple(c=<tf.Tensor 'MultiRNNCellZeroState/BasicLSTMCellZeroState/zeros:0' shape=(30, 200) dtype=float32>, h=<tf.Tensor 'MultiRNNCellZeroState/BasicLSTMCellZeroState/zeros_1:0' shape=(30, 200) dtype=float32>),\n",
       " LSTMStateTuple(c=<tf.Tensor 'MultiRNNCellZeroState/BasicLSTMCellZeroState_1/zeros:0' shape=(30, 200) dtype=float32>, h=<tf.Tensor 'MultiRNNCellZeroState/BasicLSTMCellZeroState_1/zeros_1:0' shape=(30, 200) dtype=float32>))"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "_initial_state = stacked_lstm.zero_state(batch_size, tf.float32)\n",
    "_initial_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets look at the states, though they are all zero for now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(LSTMStateTuple(c=array([[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]], dtype=float32), h=array([[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]], dtype=float32)),\n",
       " LSTMStateTuple(c=array([[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]], dtype=float32), h=array([[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]], dtype=float32)))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session.run(_initial_state, feed_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Embeddings\n",
    "We create the embeddings for our input data. Embedding is a dictionary of [10000x200] for all 10000 unique words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding = tf.get_variable(\"embedding\", [vocab_size, hidden_size])  # [10000x200]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.01063205,  0.00814733, -0.00844954, ...,  0.00378895,\n",
       "        -0.02073141,  0.01221155],\n",
       "       [ 0.01041644,  0.0137022 ,  0.00560501, ...,  0.01056347,\n",
       "         0.00471439, -0.02278037],\n",
       "       [-0.02003521,  0.00571358, -0.02156209, ..., -0.00082284,\n",
       "         0.01038924,  0.02189862],\n",
       "       ...,\n",
       "       [-0.00145212,  0.02051926,  0.01961245, ...,  0.01894623,\n",
       "         0.0170134 ,  0.01944283],\n",
       "       [ 0.01519434,  0.02213032,  0.01244638, ..., -0.00379353,\n",
       "        -0.00215455,  0.01070447],\n",
       "       [ 0.01711179, -0.01450142, -0.01781657, ...,  0.00271924,\n",
       "        -0.0067168 ,  0.01134633]], dtype=float32)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session.run(tf.global_variables_initializer())\n",
    "session.run(embedding, feed_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`embedding_lookup` goes to each row of `input_data` and for each word in the row/sentence, finds the correspond vector in embedding.\n",
    "It creates a [30x20x200] matrix, so, the first element of __inputs__ (the first sentence), is a matrix of 20x200, which each row of it, is a vector representing a word in the sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define where to get the data for our embeddings.\n",
    "inputs = tf.nn.embedding_lookup(embedding, _input_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'embedding_lookup:0' shape=(30, 20, 200) dtype=float32>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.0134263 ,  0.0212361 ,  0.02112414, ..., -0.00959834,\n",
       "         0.01385364,  0.00844003],\n",
       "       [-0.02212243, -0.01237901,  0.00419767, ..., -0.01935913,\n",
       "         0.0139115 , -0.00408517],\n",
       "       [ 0.01915921, -0.00510946, -0.00119747, ..., -0.01553603,\n",
       "         0.00980788, -0.00401851],\n",
       "       ...,\n",
       "       [ 0.01955495, -0.01637472, -0.00495487, ...,  0.02245566,\n",
       "         0.0200689 ,  0.01397086],\n",
       "       [-0.023112  , -0.00590596,  0.00579223, ..., -0.01317282,\n",
       "         0.00883655,  0.01173041],\n",
       "       [ 0.00109089, -0.01057535,  0.01938266, ..., -0.00067376,\n",
       "        -0.00868236,  0.02187211]], dtype=float32)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session.run(inputs[0], feed_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Constructing Recurrent Neural Networks\n",
    "`tf.nn.dynamicrnn()` creates a recurrent neural network using `stacked_lstm` which is an instance of RNNCell. \n",
    "\n",
    "The input should be a Tensor of shape: [batch_size, max_time, ...], in our case it would be (30, 20, 200)\n",
    "\n",
    "This method, returns a pair (outputs, new_state) where:\n",
    "- outputs is a length T list of outputs (one for each input), or a nested tuple of such elements.\n",
    "- new_state is the final state\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "outputs, new_state =  tf.nn.dynamic_rnn(stacked_lstm, inputs, initial_state=_initial_state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, lets look at the outputs. The output of the stackedLSTM comes from 200 hidden layer, and in each time step (20), one of them gets activated. We use the linear activation to map the 200 hidden layer to a [?x10 matrix]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'rnn/transpose_1:0' shape=(30, 20, 200) dtype=float32>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 3.4254440e-04,  3.1593619e-05,  1.6462205e-04, ...,\n",
       "        -2.7911790e-04,  3.4073930e-06, -1.8114239e-04],\n",
       "       [ 2.8872013e-04,  2.4893074e-04,  1.4808765e-04, ...,\n",
       "        -2.0268549e-04, -3.9944818e-04, -2.7534069e-04],\n",
       "       [ 2.3651251e-04,  8.9221931e-04, -2.2316233e-05, ...,\n",
       "         2.0748454e-04, -6.9108745e-04, -6.5613375e-04],\n",
       "       ...,\n",
       "       [-7.5111107e-05, -5.8198889e-04, -3.6036398e-04, ...,\n",
       "        -6.7492179e-04, -3.3249514e-04, -5.5762048e-05],\n",
       "       [-5.7119832e-05, -3.1587936e-04, -2.2311052e-04, ...,\n",
       "        -5.4390036e-04, -4.4383504e-04, -3.3675591e-04],\n",
       "       [-6.3285333e-05, -5.0682033e-04,  1.4555137e-04, ...,\n",
       "        -3.8899272e-04, -5.3006998e-04, -2.6523112e-04]], dtype=float32)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session.run(tf.global_variables_initializer())\n",
    "session.run(outputs[0], feed_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets reshape the output tensor from  [30 x 20 x 200] to [600 x 200]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'Reshape:0' shape=(600, 200) dtype=float32>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output = tf.reshape(outputs, [-1, size])\n",
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 3.42544401e-04,  3.15936195e-05,  1.64622048e-04,  4.51625470e-04,\n",
       "        2.50978279e-04,  4.32280300e-04, -7.55842775e-05,  1.37098934e-04,\n",
       "       -1.32023357e-04, -1.63846184e-04, -1.71776890e-04, -1.21824203e-04,\n",
       "        2.14190208e-04, -3.66728491e-04,  3.40574450e-04, -3.29786068e-04,\n",
       "       -3.72775539e-05, -3.53801115e-05, -2.95415757e-05,  2.87478993e-04,\n",
       "       -1.29926004e-04, -1.48746592e-04,  3.20694089e-04, -1.97542555e-04,\n",
       "        2.74078629e-04, -6.54401883e-05, -6.13756885e-04,  1.65539430e-04,\n",
       "        7.61668853e-05,  1.37133844e-04,  5.19918918e-04, -1.45887507e-05,\n",
       "       -1.81402691e-04, -3.14831459e-06,  7.46607257e-05, -2.23519935e-04,\n",
       "        2.44077659e-04,  1.27249819e-04,  2.24282667e-05, -6.26288820e-05,\n",
       "       -1.86371675e-04,  1.79491180e-04,  1.23375103e-05,  4.26117651e-04,\n",
       "       -2.39729525e-05, -9.59906247e-05,  1.57951319e-04,  1.60361713e-04,\n",
       "        3.05357418e-04,  1.13533315e-04,  3.21410160e-04,  8.96532787e-04,\n",
       "       -2.55986350e-04, -1.54176392e-04, -3.82019352e-04,  4.13472560e-04,\n",
       "        6.90435581e-06, -6.03276829e-04,  3.32430354e-05,  5.42513037e-04,\n",
       "       -1.82928113e-06,  3.30085750e-04,  1.56443610e-04, -5.65575727e-04,\n",
       "        6.14310440e-04,  4.51032865e-05,  2.76610226e-04, -4.33879235e-04,\n",
       "       -1.77531576e-04, -2.26424236e-04,  1.64208774e-04,  1.63481905e-04,\n",
       "        4.06252457e-06, -4.01364290e-04,  1.18180840e-04,  6.76177933e-06,\n",
       "       -1.42570716e-04,  9.01318272e-05,  3.51686962e-04, -7.78321773e-05,\n",
       "       -1.80744741e-04,  1.81762603e-04,  1.43860525e-04,  3.01463861e-05,\n",
       "       -2.17945475e-04, -2.00749477e-04,  9.58897508e-05, -1.84449382e-04,\n",
       "        1.78013925e-05, -4.81660565e-04,  3.02458298e-04, -2.52996426e-04,\n",
       "       -1.28949629e-04, -2.74382357e-04, -1.23325835e-05, -2.12302679e-04,\n",
       "        1.77412167e-05, -7.22605182e-05, -9.85606603e-05, -1.45201411e-04,\n",
       "        3.98061420e-05, -9.67911546e-05,  2.40738198e-04, -8.33322883e-06,\n",
       "        3.97994125e-04, -5.10408194e-04,  1.61280041e-05,  1.40952339e-04,\n",
       "        3.88232031e-04,  3.79809644e-04,  4.24231584e-05,  5.17311215e-04,\n",
       "        5.89741685e-04,  1.59482632e-04,  2.41756628e-04,  1.46421735e-04,\n",
       "        1.42315708e-04, -2.02090405e-06, -7.10324501e-04, -4.86488600e-04,\n",
       "       -1.25014994e-04, -6.08225309e-05, -2.03641335e-04,  2.93985067e-04,\n",
       "        2.34519423e-04,  1.00746023e-04,  3.47783411e-04, -4.30017186e-04,\n",
       "        1.79862705e-04, -3.13075070e-05,  2.56400002e-04, -4.52421111e-04,\n",
       "        2.53759004e-04, -1.88306556e-04, -1.13671362e-04,  1.43342448e-04,\n",
       "       -2.26519827e-04,  4.22557903e-04, -5.15865177e-05,  2.03148855e-04,\n",
       "       -4.16267954e-04, -7.97753216e-07,  2.23342518e-04,  9.93102731e-05,\n",
       "        2.02837342e-04, -1.41243319e-04, -4.60071315e-05, -2.05773191e-04,\n",
       "        3.31418094e-04, -1.37659998e-04, -2.59050394e-05, -1.37345764e-04,\n",
       "        2.66423973e-04,  6.98464399e-04, -1.13171765e-04,  6.84430299e-04,\n",
       "        8.99385559e-05,  2.40534035e-04, -1.80619187e-04, -4.27566469e-04,\n",
       "        1.87704718e-05,  7.79978291e-05, -1.50946580e-04, -1.66290964e-04,\n",
       "       -2.86583127e-05,  2.08663696e-04, -2.66765157e-04,  2.97660445e-04,\n",
       "       -4.13441623e-04, -4.16473165e-04,  2.01548726e-04,  2.07242533e-06,\n",
       "       -8.77775674e-05, -3.48536305e-05, -4.56815265e-04,  2.08273035e-04,\n",
       "       -2.65600956e-05,  2.70593882e-04,  4.34169022e-04, -4.97812325e-05,\n",
       "       -5.49540913e-04, -2.33065366e-04, -5.35282830e-04,  3.79081612e-05,\n",
       "        2.12970655e-04, -1.20779907e-04,  4.17896172e-06, -2.82328489e-04,\n",
       "        8.55648104e-05,  3.66759719e-04, -2.84199050e-04, -2.09802674e-04,\n",
       "        5.49255812e-04,  3.64651496e-05,  1.61385644e-04,  1.52614652e-04,\n",
       "       -3.17563565e-04, -2.79117899e-04,  3.40739302e-06, -1.81142386e-04],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session.run(output[0], feed_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic unit\n",
    "Now, we create a logistic unit to return the probability of the output word. That is, mapping the 600.\n",
    "\n",
    "Softmax = [600 x 200] * [200 x 1000] + [1 x 1000] -> [600 x 1000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "softmax_w = tf.get_variable(\"softmax_w\", [size, vocab_size])  # [200x1000]\n",
    "softmax_b = tf.get_variable(\"softmax_b\", [vocab_size])  # [1x1000]\n",
    "logits = tf.matmul(output, softmax_w) + softmax_b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(600, 10000)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session.run(tf.global_variables_initializer())\n",
    "logi = session.run(logits, feed_dict)\n",
    "logi.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10000,)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first_word_output_probablity = logi[0]\n",
    "first_word_output_probablity.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prediction\n",
    "The maximum probablity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4654"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_array = session.run(embedding, feed_dict)\n",
    "np.argmax(first_word_output_probablity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, what is the ground truth for the first word of first sentence? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9971"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y[0][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, you can get it from target tensor, if you want to find the embedding vector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'Placeholder_1:0' shape=(30, 20) dtype=int32>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "_targets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is time to compare logit with target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "targ = session.run(tf.reshape(_targets, [-1]), feed_dict) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9971"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first_word_target_code = targ[0]\n",
    "first_word_target_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1.45170279e-02, -1.79491118e-02,  1.78318582e-02, -2.89530307e-03,\n",
       "       -6.23417646e-03,  7.49334320e-03,  1.64727047e-02, -3.48045118e-03,\n",
       "       -2.36215871e-02,  2.42397189e-04,  1.93719529e-02, -4.46389243e-03,\n",
       "       -1.97104029e-02,  8.94280896e-03,  7.62885809e-03, -2.08076779e-02,\n",
       "       -4.36184928e-04, -1.12350555e-02,  1.85768344e-02, -2.57953256e-03,\n",
       "        6.81799464e-03,  1.92548819e-02,  7.84593821e-03,  7.38065690e-03,\n",
       "       -1.41855124e-02, -1.25131011e-03, -1.14028119e-02,  2.93873623e-03,\n",
       "        4.14362177e-04,  2.34870352e-02,  2.21861750e-02,  7.54712150e-03,\n",
       "       -1.03686983e-02,  3.60180996e-03, -4.58486378e-03,  2.27417387e-02,\n",
       "        1.08637847e-02,  1.07932389e-02, -8.71755835e-03,  1.91723183e-02,\n",
       "       -2.61896849e-03,  1.02394409e-02, -1.27013242e-02,  1.84333883e-03,\n",
       "       -8.36285390e-03, -1.93770174e-02,  2.02140585e-02,  2.42355429e-02,\n",
       "       -1.84150636e-02,  1.14948452e-02,  1.55534223e-02,  8.85790586e-03,\n",
       "       -1.31806647e-02, -4.57906909e-03,  1.12304837e-03,  1.52571946e-02,\n",
       "        2.22918242e-02, -2.20935270e-02, -2.25046687e-02,  2.04432085e-02,\n",
       "       -3.24859098e-03,  1.44779570e-02,  1.53159238e-02, -1.69276036e-02,\n",
       "        1.21327415e-02,  3.82561423e-03,  8.54793936e-03,  1.49579383e-02,\n",
       "       -8.72006733e-03,  1.64578892e-02, -5.55427000e-03,  1.71584822e-02,\n",
       "        1.21538416e-02, -1.92051735e-02, -1.91067271e-02,  1.32356845e-02,\n",
       "        2.23711692e-03, -3.48259136e-03, -9.47632641e-03,  2.63454765e-03,\n",
       "       -7.99367204e-04, -1.65789817e-02,  1.01704560e-02,  1.17629692e-02,\n",
       "       -4.59428318e-03, -1.43900635e-02,  9.88443196e-03,  1.04361512e-02,\n",
       "        7.40706921e-03,  1.28863715e-02, -6.56775758e-03,  2.27475986e-02,\n",
       "        8.43303651e-03,  5.33104315e-03, -1.87673047e-02,  1.80347338e-02,\n",
       "        1.42258070e-02, -1.48792975e-02,  1.62586011e-02,  2.02873498e-02,\n",
       "       -1.33335367e-02, -4.79916856e-03,  9.20233503e-03, -8.89282674e-05,\n",
       "        3.50118242e-03,  1.70103908e-02, -1.70815159e-02,  3.97803076e-03,\n",
       "        8.72310251e-03,  5.54434210e-03,  3.10826115e-03,  5.32886758e-03,\n",
       "        2.20524818e-02,  3.11940350e-03,  1.69881284e-02, -5.91062196e-03,\n",
       "       -2.18681712e-02, -6.44848682e-03, -2.08975319e-02, -1.88802313e-02,\n",
       "        1.35185122e-02, -1.39290821e-02,  2.56298110e-04, -2.16906313e-02,\n",
       "        8.05320591e-03,  1.01521686e-02, -1.76097471e-02,  5.91939501e-03,\n",
       "        3.74237075e-04,  6.68772683e-03, -2.41873991e-02,  9.48289409e-04,\n",
       "        1.74918547e-02, -2.17660144e-03, -3.87720577e-03,  1.82762370e-02,\n",
       "       -1.24457497e-02,  1.12090409e-02,  1.55478045e-02, -9.62398760e-03,\n",
       "       -2.37814039e-02,  1.52057037e-04,  1.52767524e-02,  1.54018030e-02,\n",
       "        5.22391684e-03,  1.11443326e-02,  1.98943019e-02,  1.64460987e-02,\n",
       "       -2.02260911e-04,  8.51721317e-03,  2.29904316e-02,  7.62808695e-03,\n",
       "       -6.50643371e-03,  1.50355063e-02,  2.52978876e-04, -1.26439100e-02,\n",
       "        1.61295384e-02, -2.21609622e-02, -8.50938819e-03,  5.46133891e-03,\n",
       "       -1.11913001e-02,  1.64576136e-02,  3.75596993e-03, -2.41955630e-02,\n",
       "        9.81193036e-03, -1.08633973e-02,  1.44272335e-02, -3.37503105e-03,\n",
       "        1.63126849e-02, -2.01455876e-02, -2.01709382e-02,  2.23122798e-02,\n",
       "        9.45972279e-03, -1.83542036e-02, -5.30282967e-03, -1.86318625e-02,\n",
       "       -1.88035965e-02,  8.19233432e-03,  2.21422687e-02,  3.24522518e-03,\n",
       "        1.72720738e-02, -1.03112552e-02,  6.86577521e-03,  4.03476879e-03,\n",
       "       -1.09565072e-03, -1.35192126e-02, -1.64050963e-02,  7.31300190e-03,\n",
       "        2.17461661e-02,  9.89392772e-03, -7.69362785e-03, -1.44274533e-03,\n",
       "       -3.88269499e-03, -9.21410136e-03,  1.06925257e-02,  5.26031107e-03,\n",
       "       -1.07080862e-03, -1.40373651e-02, -5.39387576e-03, -2.07592547e-02],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first_word_target_vec = session.run(tf.nn.embedding_lookup(embedding, targ[0]))\n",
    "first_word_target_vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Objective function\n",
    "\n",
    "Now we want to define our objective function. Our objective is to minimize loss function, that is, to minimize the average negative log probability of the target words:\n",
    "\n",
    "loss = −1N∑i = 1Nln⁡ptargeti  \n",
    "This function is already implimented and available in TensorFlow through `sequence_loss_by_example` so we can just use it here. `sequence_loss_by_example` is weighted cross-entropy loss for a sequence of logits (per example).  \n",
    "\n",
    "Its arguments:  \n",
    "\n",
    "logits: List of 2D Tensors of shape [batch_size x num_decoder_symbols].  \n",
    "targets: List of 1D batch-sized int32 Tensors of the same length as logits.  \n",
    "weights: List of 1D batch-sized float Tensors of the same length as logits.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss = tf.contrib.legacy_seq2seq.sequence_loss_by_example([logits], [tf.reshape(_targets, [-1])],[tf.ones([batch_size * num_steps])])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`loss` is a 1D batch-sized float Tensor [600x1]: The log-perplexity for each sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([9.199925 , 9.208916 , 9.204049 , 9.219676 , 9.212233 , 9.225138 ,\n",
       "       9.220055 , 9.201657 , 9.196995 , 9.206326 , 9.217281 , 9.195156 ,\n",
       "       9.19926  , 9.215224 , 9.194069 , 9.195595 , 9.202955 , 9.220181 ,\n",
       "       9.204887 , 9.213215 , 9.221146 , 9.214842 , 9.214809 , 9.210042 ,\n",
       "       9.204665 , 9.1955595, 9.216201 , 9.196068 , 9.201439 , 9.199709 ,\n",
       "       9.21013  , 9.201667 , 9.206893 , 9.204727 , 9.218736 , 9.202456 ,\n",
       "       9.206758 , 9.208283 , 9.214172 , 9.210835 , 9.223533 , 9.209924 ,\n",
       "       9.195504 , 9.198727 , 9.2158985, 9.20256  , 9.193649 , 9.219003 ,\n",
       "       9.208118 , 9.222098 , 9.210141 , 9.197668 , 9.208061 , 9.20483  ,\n",
       "       9.207626 , 9.204768 , 9.210172 , 9.197681 , 9.207999 , 9.2099695,\n",
       "       9.225169 , 9.210031 , 9.225235 , 9.197767 , 9.207822 , 9.201268 ,\n",
       "       9.215479 , 9.20993  , 9.204683 , 9.222193 , 9.222103 , 9.19782  ,\n",
       "       9.225204 , 9.221833 , 9.204607 , 9.206292 , 9.223461 , 9.221262 ,\n",
       "       9.196561 , 9.212295 , 9.21061  , 9.221179 , 9.226846 , 9.199456 ,\n",
       "       9.212661 , 9.19551  , 9.1994915, 9.200699 , 9.221355 , 9.215794 ,\n",
       "       9.204836 , 9.208312 , 9.213761 , 9.2265005, 9.204637 , 9.210176 ,\n",
       "       9.195599 , 9.207312 , 9.210248 , 9.208561 , 9.221257 , 9.222272 ,\n",
       "       9.210381 , 9.197721 , 9.208008 , 9.195388 , 9.217271 , 9.222062 ,\n",
       "       9.194772 , 9.195675 , 9.22255  , 9.198426 , 9.216286 , 9.227272 ,\n",
       "       9.20742  , 9.217598 , 9.201512 , 9.217307 , 9.204741 , 9.201652 ,\n",
       "       9.2255535, 9.201638 , 9.211143 , 9.223684 , 9.210063 , 9.2248   ,\n",
       "       9.207975 , 9.206806 , 9.214992 , 9.216443 , 9.224094 , 9.221091 ,\n",
       "       9.2252245, 9.210069 , 9.197964 , 9.197298 , 9.214788 , 9.222821 ,\n",
       "       9.194761 , 9.204626 , 9.208158 , 9.221898 , 9.22722  , 9.210648 ,\n",
       "       9.211217 , 9.210101 , 9.195739 , 9.227446 , 9.203125 , 9.197302 ,\n",
       "       9.211031 , 9.210247 , 9.195741 , 9.196803 , 9.204762 , 9.204657 ,\n",
       "       9.204709 , 9.2047615, 9.221175 , 9.204874 , 9.223067 , 9.222155 ,\n",
       "       9.193698 , 9.21263  , 9.209024 , 9.216487 , 9.195616 , 9.220804 ,\n",
       "       9.201525 , 9.204686 , 9.2047205, 9.222321 , 9.221118 , 9.195976 ,\n",
       "       9.227447 , 9.207256 , 9.217859 , 9.201744 , 9.195591 , 9.197801 ,\n",
       "       9.210994 , 9.21063  , 9.210551 , 9.226704 , 9.198184 , 9.197842 ,\n",
       "       9.217402 , 9.204754 , 9.20246  , 9.214925 , 9.209291 , 9.22137  ,\n",
       "       9.213754 , 9.194295 , 9.207225 , 9.207803 , 9.215716 , 9.22227  ,\n",
       "       9.221024 , 9.196412 , 9.210139 , 9.196777 , 9.202925 , 9.207058 ,\n",
       "       9.221366 , 9.221116 , 9.210444 , 9.227446 , 9.214805 , 9.196525 ,\n",
       "       9.204767 , 9.19604  , 9.20689  , 9.210631 , 9.216659 , 9.198499 ,\n",
       "       9.208168 , 9.216542 , 9.201414 , 9.19791  , 9.200354 , 9.197804 ,\n",
       "       9.208294 , 9.197764 , 9.204735 , 9.195683 , 9.210792 , 9.200977 ,\n",
       "       9.217494 , 9.197089 , 9.202653 , 9.195589 , 9.208147 , 9.22194  ,\n",
       "       9.204813 , 9.203405 , 9.221    , 9.227527 , 9.210615 , 9.2265215,\n",
       "       9.203285 , 9.196524 , 9.213925 , 9.194531 , 9.208324 , 9.2227745,\n",
       "       9.214246 , 9.20828  , 9.226813 , 9.207693 , 9.202957 , 9.206795 ,\n",
       "       9.221131 , 9.209871 , 9.204847 , 9.204827 , 9.195633 , 9.197408 ,\n",
       "       9.216731 , 9.201352 , 9.204756 , 9.204854 , 9.206776 , 9.195617 ,\n",
       "       9.194955 , 9.210216 , 9.193103 , 9.202947 , 9.218921 , 9.2213   ,\n",
       "       9.1990795, 9.204653 , 9.20179  , 9.195603 , 9.220507 , 9.193748 ,\n",
       "       9.22495  , 9.204788 , 9.20479  , 9.204814 , 9.227272 , 9.210576 ,\n",
       "       9.212326 , 9.2024355, 9.227278 , 9.2059355, 9.216071 , 9.221306 ,\n",
       "       9.226093 , 9.204644 , 9.207016 , 9.195564 , 9.202339 , 9.20739  ,\n",
       "       9.199027 , 9.201678 , 9.195755 , 9.204725 , 9.209956 , 9.22005  ,\n",
       "       9.217452 , 9.227279 , 9.22124  , 9.226019 , 9.195531 , 9.208104 ,\n",
       "       9.212591 , 9.194275 , 9.195522 , 9.216736 , 9.210807 , 9.21002  ,\n",
       "       9.217676 , 9.208434 , 9.225022 , 9.195681 , 9.207422 , 9.209416 ,\n",
       "       9.204814 , 9.2039   , 9.226374 , 9.22371  , 9.204805 , 9.215484 ,\n",
       "       9.197318 , 9.19376  , 9.206425 , 9.204717 , 9.209609 , 9.207222 ,\n",
       "       9.207772 , 9.225601 , 9.212839 , 9.195705 , 9.218843 , 9.2160635,\n",
       "       9.196707 , 9.215386 , 9.206824 , 9.221102 , 9.202805 , 9.204721 ,\n",
       "       9.196638 , 9.207714 , 9.19452  , 9.218533 , 9.217167 , 9.196055 ,\n",
       "       9.206989 , 9.210639 , 9.206045 , 9.195745 , 9.197805 , 9.204743 ,\n",
       "       9.22667  , 9.215323 , 9.216428 , 9.226814 , 9.215329 , 9.204765 ,\n",
       "       9.1938   , 9.218279 , 9.204759 , 9.207868 , 9.223506 , 9.212161 ,\n",
       "       9.213321 , 9.196517 , 9.22129  , 9.204702 , 9.226396 , 9.207736 ,\n",
       "       9.207966 , 9.217624 , 9.221041 , 9.204885 , 9.2048855, 9.204763 ,\n",
       "       9.193898 , 9.211019 , 9.222231 , 9.227218 , 9.199341 , 9.20996  ,\n",
       "       9.204893 , 9.212403 , 9.222027 , 9.215422 , 9.204039 , 9.219154 ,\n",
       "       9.207448 , 9.221998 , 9.204985 , 9.204971 , 9.202463 , 9.227423 ,\n",
       "       9.197841 , 9.210799 , 9.2074795, 9.221339 , 9.207983 , 9.209924 ,\n",
       "       9.204521 , 9.19624  , 9.219    , 9.201297 , 9.220756 , 9.217063 ,\n",
       "       9.204361 , 9.221738 , 9.195119 , 9.2046585, 9.204882 , 9.217204 ,\n",
       "       9.198071 , 9.199599 , 9.210278 , 9.197912 , 9.207995 , 9.2099495,\n",
       "       9.204664 , 9.211869 , 9.193058 , 9.202473 , 9.210498 , 9.201373 ,\n",
       "       9.204574 , 9.19354  , 9.20539  , 9.208367 , 9.210172 , 9.201587 ,\n",
       "       9.201923 , 9.204813 , 9.217165 , 9.21445  , 9.227549 , 9.222564 ,\n",
       "       9.212869 , 9.210011 , 9.22376  , 9.206968 , 9.217196 , 9.200971 ,\n",
       "       9.1953335, 9.204599 , 9.221291 , 9.1955595, 9.200456 , 9.215577 ,\n",
       "       9.201793 , 9.202556 , 9.202843 , 9.204732 , 9.217014 , 9.215595 ,\n",
       "       9.204023 , 9.201418 , 9.221137 , 9.217006 , 9.225441 , 9.225547 ,\n",
       "       9.215072 , 9.227182 , 9.204525 , 9.20962  , 9.20796  , 9.223317 ,\n",
       "       9.225693 , 9.223266 , 9.215371 , 9.2118   , 9.226068 , 9.194906 ,\n",
       "       9.210076 , 9.197815 , 9.19785  , 9.208383 , 9.201488 , 9.2004795,\n",
       "       9.207836 , 9.204723 , 9.195747 , 9.227729 , 9.210117 , 9.204637 ,\n",
       "       9.2047615, 9.212028 , 9.216239 , 9.226744 , 9.227589 , 9.204849 ,\n",
       "       9.224662 , 9.204912 , 9.193806 , 9.220023 , 9.213419 , 9.194498 ,\n",
       "       9.221162 , 9.225128 , 9.208312 , 9.221552 , 9.213385 , 9.215138 ,\n",
       "       9.203998 , 9.220434 , 9.204925 , 9.203284 , 9.225924 , 9.195458 ,\n",
       "       9.199099 , 9.211509 , 9.210095 , 9.19549  , 9.210733 , 9.198053 ,\n",
       "       9.193535 , 9.220688 , 9.218308 , 9.209177 , 9.201711 , 9.212749 ,\n",
       "       9.225183 , 9.208149 , 9.199197 , 9.19544  , 9.205755 , 9.195512 ,\n",
       "       9.220913 , 9.21655  , 9.210241 , 9.197795 , 9.208006 , 9.21003  ,\n",
       "       9.202485 , 9.208838 , 9.215726 , 9.2083845, 9.195622 , 9.210653 ,\n",
       "       9.196963 , 9.199452 , 9.210101 , 9.220833 , 9.205618 , 9.215872 ,\n",
       "       9.200316 , 9.206627 , 9.209102 , 9.197855 , 9.202164 , 9.194236 ,\n",
       "       9.210388 , 9.197906 , 9.207941 , 9.213921 , 9.203754 , 9.2047415,\n",
       "       9.193756 , 9.223553 , 9.215808 , 9.210153 , 9.199051 , 9.204646 ,\n",
       "       9.193394 , 9.2084875, 9.220741 , 9.226282 , 9.198719 , 9.217192 ,\n",
       "       9.221326 , 9.194139 , 9.227462 , 9.205092 , 9.200688 , 9.217955 ,\n",
       "       9.227424 , 9.221339 , 9.21514  , 9.224651 , 9.2168665, 9.21881  ,\n",
       "       9.208882 , 9.218771 , 9.1935625, 9.207869 , 9.21234  , 9.221179 ,\n",
       "       9.203524 , 9.198483 , 9.204623 , 9.217544 , 9.224956 , 9.206729 ,\n",
       "       9.217546 , 9.202353 , 9.224927 , 9.204732 , 9.227054 , 9.207174 ],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session.run(loss, feed_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "184.2402"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cost = tf.reduce_sum(loss) / batch_size\n",
    "\n",
    "session.run(tf.global_variables_initializer())\n",
    "session.run(cost, feed_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, lets store the new state as final state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "final_state = new_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "\n",
    "To do gradient clipping in TensorFlow we have to take the following steps:\n",
    "\n",
    "1. Define the optimizer.\n",
    "2. Extract variables that are trainable.\n",
    "3. Calculate the gradients based on the loss function.\n",
    "4. Apply the optimizer to the variables / gradients tuple."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Define Optimizer\n",
    "\n",
    "`GradientDescentOptimizer` constructs a new gradient descent optimizer. Later, we use constructed `optimizer` to compute gradients for a loss and apply gradients to variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a variable for the learning rate.\n",
    "lr = tf.Variable(0.0, trainable=False)\n",
    "# Create the gradient descent optimizer with our learning rate.\n",
    "optimizer = tf.train.GradientDescentOptimizer(lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### 2. Trainable Variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Definining a variable, if you passed `trainable=True`, the `Variable()` constructor automatically adds new variables to the graph collection `GraphKeys.TRAINABLE_VARIABLES`. Now, using `tf.trainable_variables()` you can get all variables created with `trainable=True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Variable 'embedding:0' shape=(10000, 200) dtype=float32_ref>,\n",
       " <tf.Variable 'rnn/multi_rnn_cell/cell_0/basic_lstm_cell/kernel:0' shape=(400, 800) dtype=float32_ref>,\n",
       " <tf.Variable 'rnn/multi_rnn_cell/cell_0/basic_lstm_cell/bias:0' shape=(800,) dtype=float32_ref>,\n",
       " <tf.Variable 'softmax_w:0' shape=(200, 10000) dtype=float32_ref>,\n",
       " <tf.Variable 'softmax_b:0' shape=(10000,) dtype=float32_ref>]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get all TensorFlow variables marked as \"trainable\" (i.e. all of them except lr, which we just created).\n",
    "tvars = tf.trainable_variables()\n",
    "tvars"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can find the name and scope of all variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "tvars = tvars[3:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['softmax_w:0', 'softmax_b:0']"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[v.name for v in tvars]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Calculate the gradients based on the loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'truediv:0' shape=() dtype=float32>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Variable 'softmax_w:0' shape=(200, 10000) dtype=float32_ref>,\n",
       " <tf.Variable 'softmax_b:0' shape=(10000,) dtype=float32_ref>]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tvars"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Gradient:\n",
    "The gradient of a function is the slope of the line, or the rate of change of a function. It's a vector (a direction to move) that points in the direction of greatest increase of the function, and calculated by __derivative__ operation.\n",
    "\n",
    "First lets recall the gradient function using an toy example:\n",
    "$$ z=\\left(2x^2+3xy\\right)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8.0"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "var_x = tf.placeholder(tf.float32)\n",
    "var_y = tf.placeholder(tf.float32) \n",
    "func_test = 2.0 * var_x * var_x + 3.0 * var_x * var_y\n",
    "session.run(tf.global_variables_initializer())\n",
    "feed = {var_x: 1.0, var_y: 2.0}\n",
    "session.run(func_test, feed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `tf.gradients()` function allows you to compute the symbolic gradient of one tensor with respect to one or more other tensors - including variables. `tf.gradients(func, xs)` constructs symbolic partial derivatives of sum of `func` w.r.t. x in `xs`. \n",
    "\n",
    "Now, lets look at the derivitive w.r.t. `var_x`:\n",
    "$$ \\frac{\\partial \\:}{\\partial \\:x}\\left(2x^2+3xy\\right)=4x+3y $$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[10.0]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "var_grad = tf.gradients(func_test, [var_x])\n",
    "session.run(var_grad, feed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The derivitive w.r.t. `var_y`:\n",
    "$$ \\frac{\\partial \\:}{\\partial \\:y}\\left(2x^2+3xy\\right)=3x $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[3.0]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "var_grad = tf.gradients(func_test, [var_y])\n",
    "session.run(var_grad, feed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can look at gradients w.r.t all variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor 'gradients_2/MatMul_grad/MatMul_1:0' shape=(200, 10000) dtype=float32>,\n",
       " <tf.Tensor 'gradients_2/add_grad/Reshape_1:0' shape=(10000,) dtype=float32>]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.gradients(cost, tvars)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([[-8.33650629e-05, -3.26001864e-05, -1.04524006e-04, ...,\n",
       "          2.46637796e-07,  2.41907458e-07,  2.43596673e-07],\n",
       "        [-5.81361091e-05, -9.65743893e-05, -7.65056029e-05, ...,\n",
       "          1.70635587e-07,  1.67399520e-07,  1.68530349e-07],\n",
       "        [-6.67524073e-05, -1.40866468e-04, -2.93421210e-04, ...,\n",
       "          2.45405346e-07,  2.40726251e-07,  2.42392417e-07],\n",
       "        ...,\n",
       "        [-1.30252956e-04, -2.94189464e-04, -1.86949241e-04, ...,\n",
       "          3.69254451e-07,  3.62220675e-07,  3.64666676e-07],\n",
       "        [ 2.63405673e-05, -1.14045048e-04, -1.10268797e-04, ...,\n",
       "          2.12287162e-07,  2.08215198e-07,  2.09683805e-07],\n",
       "        [ 1.53462388e-05, -1.35425507e-05, -2.78567495e-06, ...,\n",
       "          3.43967343e-09,  3.38012640e-09,  3.40703110e-09]], dtype=float32),\n",
       " array([-0.79802614, -1.0313637 , -1.031363  , ...,  0.00202788,\n",
       "         0.00198919,  0.00200279], dtype=float32)]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grad_t_list = tf.gradients(cost, tvars)\n",
    "session.run(grad_t_list, feed_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we have a list of tensors, t-list. We can use it to find clipped tensors. `clip_by_global_norm` clips values of multiple tensors by the ratio of the sum of their norms.\n",
    "\n",
    "`clip_by_global_norm` get _t-list_ as input and returns 2 things:\n",
    " - a list of clipped tensors, so called __list_clipped__ \n",
    " - the global norm (global_norm) of all tensors in t_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_grad_norm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_0:0' shape=(200, 10000) dtype=float32>,\n",
       " <tf.Tensor 'clip_by_global_norm/clip_by_global_norm/_1:0' shape=(10000,) dtype=float32>]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define the gradient clipping threshold.\n",
    "grads, _ = tf.clip_by_global_norm(grad_t_list, max_grad_norm)\n",
    "grads"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([[-8.33650629e-05, -3.26001864e-05, -1.04524006e-04, ...,\n",
       "          2.46637796e-07,  2.41907458e-07,  2.43596673e-07],\n",
       "        [-5.81361091e-05, -9.65743893e-05, -7.65056029e-05, ...,\n",
       "          1.70635587e-07,  1.67399520e-07,  1.68530349e-07],\n",
       "        [-6.67524073e-05, -1.40866468e-04, -2.93421210e-04, ...,\n",
       "          2.45405346e-07,  2.40726251e-07,  2.42392417e-07],\n",
       "        ...,\n",
       "        [-1.30252956e-04, -2.94189464e-04, -1.86949241e-04, ...,\n",
       "          3.69254451e-07,  3.62220675e-07,  3.64666676e-07],\n",
       "        [ 2.63405673e-05, -1.14045048e-04, -1.10268797e-04, ...,\n",
       "          2.12287162e-07,  2.08215198e-07,  2.09683805e-07],\n",
       "        [ 1.53462388e-05, -1.35425507e-05, -2.78567495e-06, ...,\n",
       "          3.43967343e-09,  3.38012640e-09,  3.40703110e-09]], dtype=float32),\n",
       " array([-0.79802614, -1.0313637 , -1.031363  , ...,  0.00202788,\n",
       "         0.00198919,  0.00200279], dtype=float32)]"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session.run(grads, feed_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Apply the optimizer to the variables / gradients tuple."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the training TensorFlow Operation through the optimizer.\n",
    "train_op = optimizer.apply_gradients(zip(grads, tvars))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "session.run(tf.global_variables_initializer())\n",
    "session.run(train_op, feed_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We learned how the model is build step by step. Now, lets create a Class that represents our model. This class needs a few things:\n",
    "- We have to create the model in accordance with our defined hyperparameters\n",
    "- We have to create the placeholders for our input data and expected outputs (the real data)\n",
    "- We have to create the LSTM cell structure and connect them with our RNN structure\n",
    "- We have to create the word embeddings and point them to the input data\n",
    "- We have to create the input structure for our RNN\n",
    "- We have to instanciate our RNN model and retrieve the variable in which we should expect our outputs to appear\n",
    "- We need to create a logistic structure to return the probability of our words\n",
    "- We need to create the loss and cost functions for our optimizer to work and then create the optimizer\n",
    "- And finally, we need to create a training operation that can be run to actually train our model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PTBModel(object):\n",
    "    def __init__(self, is_training):\n",
    "        # Setting parameters for ease of use.\n",
    "        self.batch_size = batch_size\n",
    "        self.num_steps = num_steps\n",
    "        size = hidden_size\n",
    "        self.vocab_size = vocab_size\n",
    "        \n",
    "        # Creating placeholders for our input data and expected outputs (target data).\n",
    "        self._input_data = tf.placeholder(tf.int32, [batch_size, num_steps])  # [30#20]\n",
    "        self._targets = tf.placeholder(tf.int32, [batch_size, num_steps])  # [30#20]\n",
    "\n",
    "        # Creating the LSTM cell structure and connect it with the RNN structure.\n",
    "        # Create the LSTM unit. \n",
    "        # This creates only the structure for the LSTM and has to be associated with a RNN unit still.\n",
    "        # The argument n_hidden (size=200) of BasicLSTMCell is size of hidden layer, that is,\n",
    "        # the number of hidden units of the LSTM.\n",
    "        # Size is the same as the size of our hidden layer, and no bias is added to the Forget Gate. \n",
    "        # LSTM cell processes one word at a time and computes probabilities of the possible continuations of the sentence.\n",
    "        lstm_cell = tf.contrib.rnn.BasicLSTMCell(size, forget_bias=0.0)\n",
    "        \n",
    "        # Unless you changed keep_prob, this won't actually execute - this is a dropout wrapper for our LSTM unit.\n",
    "        # This is an optimization of the LSTM output, but is not needed at all.\n",
    "        if is_training and keep_prob < 1:\n",
    "            lstm_cell = tf.contrib.rnn.DropoutWrapper(lstm_cell, output_keep_prob=keep_prob)\n",
    "        \n",
    "        # By taking in the LSTM cells as parameters, the MultiRNNCell function junctions the LSTM units to the RNN units.\n",
    "        # RNN cell composed sequentially of multiple simple cells.\n",
    "        stacked_lstm = tf.contrib.rnn.MultiRNNCell([lstm_cell] * num_layers)\n",
    "\n",
    "        # Define the initial state, i.e. the model state for the very first data point.\n",
    "        # It initialize the state of the LSTM memory. The memory state of the network is initialized\n",
    "        # with a vector of zeros and gets updated after reading each word.\n",
    "        self._initial_state = stacked_lstm.zero_state(batch_size, tf.float32)\n",
    "\n",
    "        # Creating the word embeddings and pointing them to the input data.\n",
    "        with tf.device(\"/cpu:0\"):\n",
    "            # Create the embeddings for our input data. Size is hidden size.\n",
    "            embedding = tf.get_variable(\"embedding\", [vocab_size, size])  # [10000x200]\n",
    "            # Define where to get the data for our embeddings from.\n",
    "            inputs = tf.nn.embedding_lookup(embedding, self._input_data)\n",
    "\n",
    "        # Unless you changed keep_prob, this won't actually execute - this is a dropout addition for our inputs.\n",
    "        # This is an optimization of the input processing and is not needed at all.\n",
    "        if is_training and keep_prob < 1:\n",
    "            inputs = tf.nn.dropout(inputs, keep_prob)\n",
    "\n",
    "        # Creating the input structure for our RNN.\n",
    "        # Input structure is 20x[30x200].\n",
    "        # Considering each word is represended by a 200 dimentional vector, and we have 30 batches,\n",
    "        # we create 30 word-vectors of size [30x2000].\n",
    "        #inputs = [tf.squeeze(input_, [1]) for input_ in tf.split(1, num_steps, inputs)]\n",
    "        # The input structure is fed from the embeddings, which are filled in by the input data.\n",
    "        # Feeding a batch of b sentences to a RNN:\n",
    "        # In step 1, first word of each of the b sentences (in a batch) is input in parallel.  \n",
    "        # In step 2, second word of each of the b sentences is input in parallel. \n",
    "        # The parallelism is only for efficiency.  \n",
    "        # Each sentence in a batch is handled in parallel, but the network sees one word of a sentence\n",
    "        # at a time and does the computations accordingly. \n",
    "        # All the computations involving the words of all sentences in a batch at a given time step\n",
    "        # are done in parallel. \n",
    "\n",
    "        # Instanciating our RNN model and retrieving the structure for returning the outputs and the state.\n",
    "        outputs, state = tf.nn.dynamic_rnn(stacked_lstm, inputs, initial_state=self._initial_state)\n",
    "\n",
    "        # Creating a logistic unit to return the probability of the output word.\n",
    "        output = tf.reshape(outputs, [-1, size])\n",
    "        softmax_w = tf.get_variable(\"softmax_w\", [size, vocab_size])  # [200x1000]\n",
    "        softmax_b = tf.get_variable(\"softmax_b\", [vocab_size])  # [1x1000]\n",
    "        logits = tf.matmul(output, softmax_w) + softmax_b\n",
    "\n",
    "        # Defining the loss and cost functions for the model's learning to work.\n",
    "        loss = tf.contrib.legacy_seq2seq.sequence_loss_by_example([logits], [tf.reshape(self._targets, [-1])],\n",
    "                                                      [tf.ones([batch_size * num_steps])])\n",
    "        self._cost = cost = tf.reduce_sum(loss) / batch_size\n",
    "\n",
    "        # Store the final state.\n",
    "        self._final_state = state\n",
    "\n",
    "        # Everything after this point is relevant only for training.\n",
    "        if not is_training:\n",
    "            return\n",
    "\n",
    "        # Creating the Training Operation for the Model.\n",
    "        # Create a variable for the learning rate.\n",
    "        self._lr = tf.Variable(0.0, trainable=False)\n",
    "        # Get all TensorFlow variables marked as \"trainable\" (i.e. all of them except _lr).\n",
    "        tvars = tf.trainable_variables()\n",
    "        # Define the gradient clipping threshold.\n",
    "        grads, _ = tf.clip_by_global_norm(tf.gradients(cost, tvars), max_grad_norm)\n",
    "        # Create the gradient descent optimizer with the learning rate.\n",
    "        optimizer = tf.train.GradientDescentOptimizer(self._lr)\n",
    "        # Create the training TensorFlow Operation through the optimizer.\n",
    "        self._train_op = optimizer.apply_gradients(zip(grads, tvars))\n",
    "\n",
    "    # Helper functions for the LSTM RNN class\n",
    "\n",
    "    \"\"\"Assign the learning rate for the model.\"\"\"\n",
    "    def assign_lr(self, session, lr_value):\n",
    "        session.run(tf.assign(self._lr, lr_value))\n",
    "\n",
    "    \"\"\"Returns the input data for the model.\"\"\"\n",
    "    @property\n",
    "    def input_data(self):\n",
    "        return self._input_data\n",
    "\n",
    "    \"\"\"Returns the targets for the model.\"\"\"\n",
    "    @property\n",
    "    def targets(self):\n",
    "        return self._targets\n",
    "\n",
    "    \"\"\"Returns the initial state for the model.\"\"\"\n",
    "    @property\n",
    "    def initial_state(self):\n",
    "        return self._initial_state\n",
    "\n",
    "    \"\"\"Returns the defined cost.\"\"\"\n",
    "    @property\n",
    "    def cost(self):\n",
    "        return self._cost\n",
    "\n",
    "    \"\"\"Returns the final state for the model.\"\"\"\n",
    "    @property\n",
    "    def final_state(self):\n",
    "        return self._final_state\n",
    "\n",
    "    \"\"\"Returns the current learning rate for the model.\"\"\"\n",
    "    @property\n",
    "    def lr(self):\n",
    "        return self._lr\n",
    "\n",
    "    \"\"\"Returns the training operation defined for the model.\"\"\"\n",
    "    @property\n",
    "    def train_op(self):\n",
    "        return self._train_op"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With that, the actual structure of the Recurrent Neural Network with Long Short-Term Memory is finished. What remains is to actually create the methods to run through time - that is, the `run_epoch` method to be run at each epoch and a `main` script which ties all of this together.\n",
    "\n",
    "What the `run_epoch` method should do is take the input data and feed it to the relevant operations. This will return at the very least the current result for the cost function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"run_epoch takes as parameters the current session, the model instance, the data to be\n",
    "fed, and the operation to be run.\"\"\"\n",
    "def run_epoch(session, m, data, eval_op, verbose=False):\n",
    "    # Define the epoch size based on the length of the data, batch size and the number of steps.\n",
    "    epoch_size = ((len(data) // m.batch_size) - 1) // m.num_steps\n",
    "    start_time = time.time()\n",
    "    costs = 0.0\n",
    "    iters = 0\n",
    "    #state = m.initial_state.eval()\n",
    "    #m.initial_state = tf.convert_to_tensor(m.initial_state) \n",
    "    #state = m.initial_state.eval()\n",
    "    state = session.run(m.initial_state)\n",
    "    \n",
    "    # For each step and data point.\n",
    "    for step, (x, y) in enumerate(reader.ptb_iterator(data, m.batch_size, m.num_steps)):\n",
    "        # Evaluate and return cost and state by running cost, final_state and the function passed as parameter.\n",
    "        cost, state, _ = session.run([m.cost, m.final_state, eval_op],\n",
    "                                     {m.input_data: x,\n",
    "                                      m.targets: y,\n",
    "                                      m.initial_state: state})\n",
    "        \n",
    "        # Add returned cost to costs (which keeps track of the total costs for this epoch).\n",
    "        costs += cost\n",
    "        \n",
    "        # Add number of steps to iteration counter.\n",
    "        iters += m.num_steps\n",
    "\n",
    "        if verbose and step % (epoch_size // 10) == 10:\n",
    "            print(\"%.3f perplexity: %.3f speed: %.0f wps\" % (step * 1.0 / epoch_size, np.exp(costs / iters),\n",
    "              iters * m.batch_size / (time.time() - start_time)))\n",
    "\n",
    "    # Returns the Perplexity rating to keep track of how the model is evolving.\n",
    "    return np.exp(costs / iters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we create the `main` method to tie everything together. The code here reads the data from the directory, using the `reader` helper module, and then trains and evaluates the model on both a testing and a validating subset of data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reads the data and separates it into training data, validation data and testing data.\n",
    "raw_data = reader.ptb_raw_data(data_dir)\n",
    "train_data, valid_data, test_data, _, _ = raw_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 : Learning rate: 1.000\n",
      "0.006 perplexity: 5891.393 speed: 1667 wps\n",
      "0.106 perplexity: 1084.938 speed: 1933 wps\n",
      "0.205 perplexity: 816.106 speed: 1924 wps\n",
      "0.305 perplexity: 661.687 speed: 1914 wps\n",
      "0.404 perplexity: 555.329 speed: 1920 wps\n",
      "0.504 perplexity: 490.278 speed: 1926 wps\n",
      "0.603 perplexity: 441.656 speed: 1929 wps\n",
      "0.702 perplexity: 405.473 speed: 1930 wps\n",
      "0.802 perplexity: 377.890 speed: 1932 wps\n",
      "0.901 perplexity: 352.301 speed: 1933 wps\n",
      "Epoch 1 : Train Perplexity: 332.073\n",
      "Epoch 1 : Valid Perplexity: 201.592\n",
      "Epoch 2 : Learning rate: 1.000\n",
      "0.006 perplexity: 222.417 speed: 1904 wps\n",
      "0.106 perplexity: 197.852 speed: 1933 wps\n",
      "0.205 perplexity: 189.711 speed: 1938 wps\n",
      "0.305 perplexity: 181.570 speed: 1937 wps\n",
      "0.404 perplexity: 173.281 speed: 1911 wps\n",
      "0.504 perplexity: 169.721 speed: 1890 wps\n",
      "0.603 perplexity: 166.098 speed: 1889 wps\n",
      "0.702 perplexity: 162.890 speed: 1892 wps\n",
      "0.802 perplexity: 160.616 speed: 1868 wps\n",
      "0.901 perplexity: 156.890 speed: 1846 wps\n",
      "Epoch 2 : Train Perplexity: 154.250\n",
      "Epoch 2 : Valid Perplexity: 156.702\n",
      "Epoch 3 : Learning rate: 1.000\n",
      "0.006 perplexity: 148.527 speed: 1799 wps\n"
     ]
    }
   ],
   "source": [
    "# Initializes the Execution Graph and the Session.\n",
    "with tf.Graph().as_default(), tf.Session() as session:\n",
    "    initializer = tf.random_uniform_initializer(-init_scale, init_scale)\n",
    "    \n",
    "    # Instantiates the model for training.\n",
    "    # tf.variable_scope add a prefix to the variables created with tf.get_variable\n",
    "    with tf.variable_scope(\"model\", reuse=None, initializer=initializer):\n",
    "        m = PTBModel(is_training=True)\n",
    "        \n",
    "    # Reuses the trained parameters for the validation and testing models.\n",
    "    # They are different instances but use the same variables for weights and biases,\n",
    "    # they just don't change when data is input.\n",
    "    with tf.variable_scope(\"model\", reuse=True, initializer=initializer):\n",
    "        mvalid = PTBModel(is_training=False)\n",
    "        mtest = PTBModel(is_training=False)\n",
    "\n",
    "    # Initialize all variables.\n",
    "    tf.global_variables_initializer().run()\n",
    "\n",
    "    for i in range(max_max_epoch):\n",
    "        # Define the decay for this epoch.\n",
    "        lr_decay = decay ** max(i - max_epoch, 0.0)\n",
    "        \n",
    "        # Set the decayed learning rate as the learning rate for this epoch.\n",
    "        m.assign_lr(session, learning_rate * lr_decay)\n",
    "\n",
    "        print(\"Epoch %d : Learning rate: %.3f\" % (i + 1, session.run(m.lr)))\n",
    "        \n",
    "        # Run the loop for this epoch in the training model.\n",
    "        train_perplexity = run_epoch(session, m, train_data, m.train_op,\n",
    "                                   verbose=True)\n",
    "        print(\"Epoch %d : Train Perplexity: %.3f\" % (i + 1, train_perplexity))\n",
    "        \n",
    "        # Run the loop for this epoch in the validation model.\n",
    "        valid_perplexity = run_epoch(session, mvalid, valid_data, tf.no_op())\n",
    "        print(\"Epoch %d : Valid Perplexity: %.3f\" % (i + 1, valid_perplexity))\n",
    "    \n",
    "    # Run the loop in the testing model to see how effective was the training.\n",
    "    test_perplexity = run_epoch(session, mtest, test_data, tf.no_op())\n",
    "    \n",
    "    print(\"Test Perplexity: %.3f\" % test_perplexity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the model's perplexity rating drops very quickly after a few iterations. As was elaborated before, **lower perplexity means that the model is more certain about its prediction**."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}