data-visualise.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "95f0a171",
   "metadata": {},
   "source": [
    "(data-visualise)=\n",
    "# Data Visualisation\n",
    "\n",
    "## Introduction\n",
    "\n",
    "> \"The simple graph has brought more information to the data analyst's mind than any other device.\" --- John Tukey\n",
    "\n",
    "This chapter will teach you how to visualise your data using using **[letsplot](https://lets-plot.org/)**.\n",
    "\n",
    "There are broadly two categories of approach to using code to create data visualisations: imperative, where you build what you want, and declarative, where you say what you want. Choosing which to use involves a trade-off: imperative libraries offer you flexibility but at the cost of some verbosity; declarative libraries offer you a quick way to plot your data, but only if it’s in the right format to begin with, and customisation to special chart types is more difficult. Python has many excellent plotting packages, including perhaps the most powerful imperative plotting package around, **matplotlib**.\n",
    "\n",
    "However, we'll get further faster by learning one system and applying it in many places—and the beauty of declarative plotting is that it covers lots of standard charts simply and well. **letsplot** implements the so-called **grammar of graphics**, a coherent declarative system for describing and building graphs.\n",
    "\n",
    "We will start by creating a simple scatterplot and use that to introduce aesthetic mappings and geometric objects—the fundamental building blocks of **letsplot**. We will then walk you through visualising distributions of single variables as well as visualising relationships between two or more variables. We’ll finish off with saving your plots and troubleshooting tips. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17575f3a",
   "metadata": {},
   "source": [
    "### Prerequisites\n",
    "\n",
    "You will need to install the **letsplot** package for this chapter. To do this, open up the command line of your computer, type in `pip install lets-plot`, and hit enter."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "792902c7",
   "metadata": {},
   "source": [
    "```{note}\n",
    "The command line can be opened within Visual Studio Code and Codespaces by going to View -> Terminal.\n",
    "```\n",
    "\n",
    "Note that you only need to install a package once in each Python environment."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0ad70c8",
   "metadata": {},
   "source": [
    "We'll also need to have the **pandas** package installed—this package, which we'll be seeing a lot of, is for data. You can similarly install it by running `pip install pandas` on the command line.\n",
    "\n",
    "Finally, we'll also need some data (you can't science without data). We'll be using the Palmer penguins dataset. Unusually, this can also be installed as a package—normally you would load data from a file, but these data are so popular for tutorials they've found their way into an installable package. Run `pip install palmerpenguins` to get these data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8852373a",
   "metadata": {},
   "source": [
    "Our next task is to load these into our Python session, either in a Python notebook cell within a Jupyter Notebook, by writing it in a script that we then send to the interactive window, or by typing it directly into the interactive window and hitting shift and enter. Here's the code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a86fb211",
   "metadata": {},
   "outputs": [],
   "source": [
    "from lets_plot import *\n",
    "from palmerpenguins import load_penguins\n",
    "\n",
    "LetsPlot.setup_html()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4443f4dd",
   "metadata": {},
   "source": [
    "These lines import parts of the **pandas** and **palmerpenguins** packages, then import all (`*`) of the functions of the **letsplot** package. The final line allows charts to display in HTML."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bc87ab8",
   "metadata": {},
   "source": [
    "## First Steps\n",
    "\n",
    "Do penguins with longer flippers weigh more or less than penguins with shorter flippers? You probably already have an answer, but try to make your answer precise. What does the relationship between flipper length and body mass look like? Is it positive? Negative? Linear? Nonlinear? Does the relationship vary by the species of the penguin? How about by the island where the penguin lives? Let’s create visualisations that we can use to answer these questions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4eb9c4f",
   "metadata": {},
   "source": [
    "### The `penguins` data frame\n",
    "\n",
    "You can test your answers to those questions with the penguins data frame found in palmerpenguins (a.k.a. `from palmerpenguins import load_penguins`). A data frame is a rectangular collection of variables (in the columns) and observations (in the rows). `penguins` contains 344 observations collected and made available by Dr. Kristen Gorman and the Palmer Station, Antarctica LTER.{cite:p}`horst2020palmerpenguins`.\n",
    "\n",
    "To make the discussion easier, let's define some terms:\n",
    "\n",
    "-   A **variable** is a quantity, quality, or property that you can measure.\n",
    "\n",
    "-   A **value** is the state of a variable when you measure it.\n",
    "    The value of a variable may change from measurement to measurement.\n",
    "\n",
    "-   An **observation** is a set of measurements made under similar conditions (you usually make all of the measurements in an observation at the same time and on the same object).\n",
    "    An observation will contain several values, each associated with a different variable.\n",
    "    We'll sometimes refer to an observation as a data point.\n",
    "\n",
    "-   **Tabular data** is a set of values, each associated with a variable and an observation.\n",
    "    Tabular data is *tidy* if each value is placed in its own \"cell\", each variable in its own column, and each observation in its own row.\n",
    "\n",
    "In this context, a variable refers to an attribute of all the penguins, and an observation refers to all the attributes of a single penguin.\n",
    "\n",
    "Type the name of the data frame in the interactive window and Python will print a preview of its contents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0cf986aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "penguins = load_penguins()\n",
    "penguins"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc310b4f",
   "metadata": {},
   "source": [
    "For an alternative view, where you can see the first few observations of each variable, use `penguins.head()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23c75ba7",
   "metadata": {},
   "outputs": [],
   "source": [
    "penguins.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3eb1881",
   "metadata": {},
   "source": [
    "Among the variables in `penguins` are:\n",
    "\n",
    "1.  `species`: a penguin's species (Adelie, Chinstrap, or Gentoo).\n",
    "\n",
    "2.  `flipper_length_mm`: length of a penguin's flipper, in millimeters.\n",
    "\n",
    "3.  `body_mass_g`: body mass of a penguin, in grams.\n",
    "\n",
    "To learn more about `penguins`, open the help page of its data-loading function by running `help(load_penguins)`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "caf04bde",
   "metadata": {},
   "source": [
    "### Ultimate Goal\n",
    "\n",
    "Our ultimate goal in this chapter is to recreate the following visualisation displaying the relationship between flipper lengths and body masses of these penguins, taking into consideration the species of the penguin."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "574fe39f",
   "metadata": {
    "tags": [
     "remove-input"
    ]
   },
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(penguins, aes(x=\"flipper_length_mm\", y=\"body_mass_g\"))\n",
    "    + geom_point(aes(color=\"species\", shape=\"species\"))\n",
    "    + geom_smooth(method=\"lm\")\n",
    "    + labs(\n",
    "        title=\"Body mass and flipper length\",\n",
    "        subtitle=\"Dimensions for Adelie, Chinstrap, and Gentoo Penguins\",\n",
    "        x=\"Flipper length (mm)\",\n",
    "        y=\"Body mass (g)\",\n",
    "        color=\"Species\",\n",
    "        shape=\"Species\",\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "339966d7",
   "metadata": {},
   "source": [
    "### Creating a Plot\n",
    "\n",
    "Let's recreate this plot step-by-step.\n",
    "\n",
    "With **letsplot**, you begin a plot with the function `ggplot()`, defining a plot object that you then add **layers** to.\n",
    "\n",
    "The first argument of `ggplot()` is the dataset to use in the graph and so `ggplot(data = penguins)` creates an empty graph that is primed to display the `penguins` data, but since we haven't told it how to visualise it yet, for now it's empty. Because it's empty, running this alone would raise an error message: it's an empty canvas that you'll paint the remaining layers of your plot onto.\n",
    "\n",
    "```python\n",
    "ggplot(data = penguins)\n",
    "```\n",
    "\n",
    "Next, we need to tell `ggplot()` how the information from our data will be visually represented.\n",
    "\n",
    "The `mapping` argument of the `ggplot()` function defines how variables in your dataset are mapped to visual properties (**aesthetics**) of your plot.\n",
    "The `mapping` argument is always defined in the `aes()` function, and the `x` and `y` arguments of `aes()` specify which variables to map to the x and y axes.\n",
    "For now, we will only map flipper length to the `x` aesthetic and body mass to the `y` aesthetic. **letsplot** looks for the mapped variables in the `data` argument, in this case, `penguins`.\n",
    "\n",
    "Again, we haven't actually specified anything to plot, so running\n",
    "\n",
    "```python\n",
    "ggplot(\n",
    "  data = penguins,\n",
    "  mapping = aes(x = \"flipper_length_mm\", y = \"body_mass_g\")\n",
    ")\n",
    "```\n",
    "\n",
    "would raise an error. This is because we have not yet articulated, in our code, how to represent the observations from our data frame on our plot.\n",
    "\n",
    "To do so, we need to define a **geom**: the geometrical object that a plot uses to represent data.\n",
    "These geometric objects are made available in **letsplot** with functions that start with `geom_`.\n",
    "\n",
    "People often describe plots by the type of geom that the plot uses.\n",
    "For example, bar charts use bar geoms (`geom_bar()`), line charts use line geoms (`geom_line()`), boxplots use boxplot geoms (`geom_boxplot()`), scatterplots use point geoms (`geom_point()`), and so on.\n",
    "\n",
    "The function `geom_point()` adds a layer of points to your plot, which creates a scatterplot.\n",
    "**letsplot** comes with many geom functions that each adds a different type of layer to a plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15c3848b",
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(data=penguins, mapping=aes(x=\"flipper_length_mm\", y=\"body_mass_g\"))\n",
    "    + geom_point()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8d8bfff",
   "metadata": {},
   "source": [
    "Now we have something that looks like what we might think of as a \"scatterplot\".\n",
    "It doesn't yet match our \"ultimate goal\" plot, but using this plot we can start answering the question that motivated our exploration: \"What does the relationship between flipper length and body mass look like?\" The relationship appears to be positive (as flipper length increases, so does body mass), fairly linear (the points are clustered around a line instead of a curve), and moderately strong (there isn't too much scatter around such a line).\n",
    "Penguins with longer flippers are generally larger in terms of their body mass.\n",
    "\n",
    "It's a good point to flag that although we have plotted everything in the `penguins` data frame, there were a couple of rows with undefined values—and of course these cannot be plotted."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40818ed6",
   "metadata": {},
   "source": [
    "### Adding aesthetics and layers\n",
    "\n",
    "Scatterplots are useful for displaying the relationship between two numerical variables, but it's always a good idea to be skeptical of any apparent relationship between two variables and ask if there may be other variables that explain or change the nature of this apparent relationship. For example, does the relationship between flipper length and body mass differ by species?\n",
    "\n",
    "Let's incorporate species into our plot and see if this reveals any additional insights into the apparent relationship between these variables.\n",
    "We will do this by representing species with different colored points.\n",
    "\n",
    "To achieve this, will we need to modify the aesthetic or the geom?\n",
    "If you guessed \"in the aesthetic mapping, inside of `aes()`\", you're already getting the hang of creating data visualisations with **letsplot**!\n",
    "And if not, don't worry.\n",
    "\n",
    "Throughout the book you will make many more plots and have many more opportunities to check your intuition as you make them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b0e1c38",
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(\n",
    "        data=penguins,\n",
    "        mapping=aes(x=\"flipper_length_mm\", y=\"body_mass_g\", color=\"species\"),\n",
    "    )\n",
    "    + geom_point()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7272e621",
   "metadata": {},
   "source": [
    "When a categorical variable is mapped to an aesthetic, **letsplot** will automatically assign a unique value of the aesthetic (here a unique color) to each unique level of the variable (each of the three species), a process known as **scaling**.\n",
    "\n",
    "**letsplot** will also add a legend that explains which values correspond to which levels.\n",
    "\n",
    "Now let's add one more layer: a smooth curve displaying the relationship between body mass and flipper length.\n",
    "\n",
    "Before you proceed, refer back to the code above, and think about how we can add this to our existing plot.\n",
    "\n",
    "Since this is a new geometric object representing our data, we will add a new geom as a layer on top of our point geom: `geom_smooth()`.\n",
    "\n",
    "And we will specify that we want to draw the line of best fit based on a `l`inear `m`odel with `method = \"lm\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "943efd36",
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(\n",
    "        data=penguins,\n",
    "        mapping=aes(x=\"flipper_length_mm\", y=\"body_mass_g\", color=\"species\"),\n",
    "    )\n",
    "    + geom_point()\n",
    "    + geom_smooth(method=\"lm\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c1cd3c7",
   "metadata": {},
   "source": [
    "We have successfully added lines, but this plot doesn't look like the plot from earlier as that only had one line for the entire dataset as opposed to separate lines for each of the penguin species.\n",
    "\n",
    "When aesthetic mappings are defined in `ggplot()`, at the *global* level, they're passed down to each of the subsequent geom layers of the plot.\n",
    "\n",
    "However, each geom function in **letplot** can also take a `mapping` argument, which allows for aesthetic mappings at the *local* level that are added to those inherited from the global level.\n",
    "\n",
    "Since we want points to be colored based on species but don't want the lines to be separated out for them, we should specify `color = species` for `geom_point()` only: therefore we take it out of the global `aes()` and just add it to `geom_point()`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e12b3bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(data=penguins, mapping=aes(x=\"flipper_length_mm\", y=\"body_mass_g\"))\n",
    "    + geom_point(mapping=aes(color=\"species\"))\n",
    "    + geom_smooth(method=\"lm\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "928898ef",
   "metadata": {},
   "source": [
    "Voila! We have something that looks very much like our ultimate goal, though it's not yet perfect.\n",
    "\n",
    "We still need to use different shapes for each species of penguins and improve labels.\n",
    "\n",
    "It's generally not a good idea to represent information using only colors on a plot, as people perceive colors differently due to color blindness or other color vision differences. Therefore, in addition to color, we can also map `species` to the `shape` aesthetic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17d5803b",
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(data=penguins, mapping=aes(x=\"flipper_length_mm\", y=\"body_mass_g\"))\n",
    "    + geom_point(mapping=aes(color=\"species\", shape=\"species\"))\n",
    "    + geom_smooth(method=\"lm\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cfae7fc",
   "metadata": {},
   "source": [
    "Note that the legend is automatically updated to reflect the different shapes of the points as well.\n",
    "\n",
    "And finally, we can improve the labels of our plot using the `labs()` function in a new layer. Some of the arguments to `labs()` might be self explanatory: `title` adds a title and `subtitle` adds a subtitle to the plot. Other arguments match the aesthetic mappings, `x` is the x-axis label, `y` is the y-axis label, and `color` and `shape` define the label for the legend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9b98ec4",
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(data=penguins, mapping=aes(x=\"flipper_length_mm\", y=\"body_mass_g\"))\n",
    "    + geom_point(aes(color=\"species\", shape=\"species\"))\n",
    "    + geom_smooth(method=\"lm\")\n",
    "    + labs(\n",
    "        title=\"Body mass and flipper length\",\n",
    "        subtitle=\"Dimensions for Adelie, Chinstrap, and Gentoo Penguins\",\n",
    "        x=\"Flipper length (mm)\",\n",
    "        y=\"Body mass (g)\",\n",
    "        color=\"Species\",\n",
    "        shape=\"Species\",\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdc33b33",
   "metadata": {},
   "source": [
    "We finally have a plot that perfectly matches our \"ultimate goal\"!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81863a95",
   "metadata": {},
   "source": [
    "### Exercises\n",
    "\n",
    "1.  How many rows are in `penguins`?\n",
    "    How many columns?\n",
    "\n",
    "2.  What does the `bill_depth_mm` variable in the `penguins` data frame describe?\n",
    "    Read the help for `load_penguins()` to find out, eg run `help(load_penguins)`.\n",
    "\n",
    "3.  Make a scatterplot of `bill_depth_mm` vs. `bill_length_mm`.\n",
    "    That is, make a scatterplot with `bill_depth_mm` on the y-axis and `bill_length_mm` on the x-axis.\n",
    "    Describe the relationship between these two variables.\n",
    "\n",
    "4.  What happens if you make a scatterplot of `species` vs. `bill_depth_mm`?\n",
    "    What might be a better choice of geom?\n",
    "\n",
    "5.  Why does the following give an error and how would you fix it?\n",
    "\n",
    "    ```python\n",
    "    (ggplot(data = penguins) + \n",
    "      geom_point())\n",
    "    ```\n",
    "\n",
    "6.  Add the following caption to the plot you made in the previous exercise: \"Data come from the palmerpenguins package.\" Hint: Take a look at the documentation for `labs()`.\n",
    "\n",
    "7.  Recreate the following visualisation.\n",
    "    What aesthetic should `bill_depth_mm` be mapped to?\n",
    "    And should it be mapped at the global level or at the geom level?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c76be4b",
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(data=penguins, mapping=aes(x=\"flipper_length_mm\", y=\"body_mass_g\"))\n",
    "    + geom_point(aes(color=\"bill_depth_mm\"))\n",
    "    + geom_smooth()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "986fdc29",
   "metadata": {},
   "source": [
    "\n",
    "8.  Run this code in your head and predict what the output will look like.\n",
    "    Then, run the code in Python and check your predictions.\n",
    "\n",
    "    ```python\n",
    "\n",
    "    (ggplot(\n",
    "      data = penguins,\n",
    "      mapping = aes(x = \"flipper_length_mm\", y = \"body_mass_g\", color = \"island\")\n",
    "    ) +\n",
    "      geom_point() +\n",
    "      geom_smooth(se = False)\n",
    "    )\n",
    "    ```\n",
    "\n",
    "9.  Will these two graphs look different?\n",
    "    Why/why not?\n",
    "\n",
    "    ```python\n",
    "\n",
    "    (ggplot(\n",
    "      data = penguins,\n",
    "      mapping = aes(x = \"flipper_length_mm\", y = \"body_mass_g\")\n",
    "    ) +\n",
    "      geom_point() +\n",
    "      geom_smooth()\n",
    "    )\n",
    "    ```\n",
    "    ```python\n",
    "    (ggplot() +\n",
    "      geom_point(\n",
    "        data = penguins,\n",
    "        mapping = aes(x = \"flipper_length_mm\", y = \"body_mass_g\")\n",
    "      ) +\n",
    "      geom_smooth(\n",
    "        data = penguins,\n",
    "        mapping = aes(x = \"flipper_length_mm\", y = \"body_mass_g\")\n",
    "      )\n",
    "    )\n",
    "    ```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10806a67",
   "metadata": {},
   "source": [
    "## **letsplot** calls\n",
    "\n",
    "As we move on from these introductory sections, we'll transition to a more concise expression of **letsplot** code.\n",
    "\n",
    "So far we've been very explicit, which is helpful when you are learning:\n",
    "\n",
    "```python\n",
    "(ggplot(\n",
    "  data = penguins,\n",
    "  mapping = aes(x = \"flipper_length_mm\", y = \"body_mass_g\")\n",
    ") +\n",
    "  geom_point())\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4f403fb",
   "metadata": {},
   "source": [
    "Typically, the first one or two arguments to a function are so important that you should know them by heart.\n",
    "The first two arguments to `ggplot()` are `data` and `mapping`, in the remainder of the book, we won't supply those names—the way the function is written, Python knows to expect these variables because of their position. Not writing them in saves typing, and, by reducing the amount of extra text, makes it easier to see what's different between plots.\n",
    "That's a really important programming concern that we'll come back to later.\n",
    "\n",
    "Rewriting the previous plot more concisely yields:\n",
    "\n",
    "```python\n",
    "(\n",
    "    ggplot(penguins, aes(x = \"flipper_length_mm\", y = \"body_mass_g\")) + \n",
    "  geom_point()\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d219ea2",
   "metadata": {},
   "source": [
    "## visualising distributions\n",
    "\n",
    "How you visualise the distribution of a variable depends on the type of variable: categorical or numerical.\n",
    "\n",
    "### A categorical variable\n",
    "\n",
    "A variable is **categorical** if it can only take one of a small set of values.\n",
    "To examine the distribution of a categorical variable, you can use a bar chart.\n",
    "The height of the bars displays how many observations occurred with each `x` value.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21b45061",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(penguins, aes(x=\"species\")) + geom_bar())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "699f42eb",
   "metadata": {},
   "source": [
    "You may have seen earlier that the *data type* of the `\"species\"` column is string. Ideally, we want it to be categorical, so that there's no confusion about the fact that we're dealing with a finite number of mutually exclusive groups here. Another advantage is that it allows plotting tools to realise what kind of data it is working with.\n",
    "\n",
    "We can transform the variable to a categorical variable using **pandas** like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e046bb2",
   "metadata": {},
   "outputs": [],
   "source": [
    "penguins[\"species\"] = penguins[\"species\"].astype(\"category\")\n",
    "penguins.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06d834a5",
   "metadata": {},
   "source": [
    "You will learn more about categorical variables later in the book."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9ca3124",
   "metadata": {},
   "source": [
    "\n",
    "### A numerical variable\n",
    "\n",
    "A variable is **numerical** (or quantitative) if it can take on a wide range of numerical values, and it is sensible to add, subtract, or take averages with those values. Numerical variables can be continuous or discrete.\n",
    "\n",
    "One commonly used visualisation for distributions of continuous variables is a histogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93675336",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(penguins, aes(x=\"body_mass_g\")) + geom_histogram(binwidth=200))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdac30fd",
   "metadata": {},
   "source": [
    "A histogram divides the x-axis into equally spaced bins and then uses the height of a bar to display the number of observations that fall in each bin.\n",
    "In the graph above, the tallest bar shows that 39 observations have a `body_mass_g` value between 3,500 and 3,700 grams, which are the left and right edges of the bar.\n",
    "\n",
    "You can set the width of the intervals in a histogram with the binwidth argument, which is measured in the units of the `x` variable.\n",
    "You should always explore a variety of binwidths when working with histograms, as different binwidths can reveal different patterns.\n",
    "In the plots below a binwidth of 20 is too narrow, resulting in too many bars, making it difficult to determine the shape of the distribution.\n",
    "Similarly, a binwidth of 2,000 is too high, resulting in all data being binned into only three bars, and also making it difficult to determine the shape of the distribution.\n",
    "A binwidth of 200 provides a sensible balance, but you should always look at your data a few different ways, especially with histograms as they can be misleading."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5f1980f",
   "metadata": {},
   "source": [
    "An alternative visualisation for distributions of numerical variables is a density plot.\n",
    "A density plot is a smoothed-out version of a histogram and a practical alternative, particularly for continuous data that comes from an underlying smooth distribution.\n",
    "We won't go into how `geom_density()` estimates the density (you can read more about that in the function documentation), but let's explain how the density curve is drawn with an analogy.\n",
    "Imagine a histogram made out of wooden blocks.\n",
    "Then, imagine that you drop a cooked spaghetti string over it.\n",
    "The shape the spaghetti will take draped over blocks can be thought of as the shape of the density curve.\n",
    "It shows fewer details than a histogram but can make it easier to quickly glean the shape of the distribution, particularly with respect to modes and skewness.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a58021f",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(penguins, aes(x=\"body_mass_g\")) + geom_density())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0086086",
   "metadata": {},
   "source": [
    "### Exercises\n",
    "\n",
    "1.  Make a bar plot of `\"species\"` of `penguins`, where you assign `\"species\"` to the `y` aesthetic.\n",
    "    How is this plot different?\n",
    "\n",
    "2.  How are the following two plots different?\n",
    "    Which aesthetic, `color` or `fill`, is more useful for changing the color of bars?\n",
    "\n",
    "    ```Python\n",
    "\n",
    "    (ggplot(penguins, aes(x = species)) +\n",
    "      geom_bar(color = \"red\"))\n",
    "\n",
    "    (ggplot(penguins, aes(x = species)) +\n",
    "      geom_bar(fill = \"red\"))\n",
    "    ```\n",
    "\n",
    "3.  What does the `bins` argument in `geom_histogram()` do?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "015b31a0",
   "metadata": {},
   "source": [
    "## Visualising Relationships\n",
    "\n",
    "To visualise a relationship we need to have at least two variables mapped to aesthetics of a plot—though you should remember that correlation is not causation, and causation is not correlation!\n",
    "\n",
    "In the following sections you will learn about commonly used plots for visualising relationships between two or more variables and the geoms used for creating them."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85458170",
   "metadata": {},
   "source": [
    "### A numerical and a categorical variable\n",
    "\n",
    "To visualise the relationship between a numerical and a categorical variable we can use side-by-side box plots.\n",
    "\n",
    "A **boxplot** is a type of visual shorthand for measures of position within a distribution (percentiles).\n",
    "\n",
    "It is also useful for identifying potential outliers. Each boxplot consists of:\n",
    "\n",
    "-   A box that indicates the range of the middle half of the data, a distance known as the interquartile range (IQR), stretching from the 25th percentile of the distribution to the 75th percentile.\n",
    "    In the middle of the box is a line that displays the median, i.e. 50th percentile, of the distribution.\n",
    "    These three lines give you a sense of the spread of the distribution and whether or not the distribution is symmetric about the median or skewed to one side.\n",
    "\n",
    "-   Visual points that display observations that fall more than 1.5 times the IQR from either edge of the box.\n",
    "    These outlying points are unusual so are plotted individually.\n",
    "\n",
    "-   A line (or whisker) that extends from each end of the box and goes to the farthest non-outlier point in the distribution.\n",
    "\n",
    "\n",
    "Let's take a look at the distribution of body mass by species using `geom_boxplot()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a636947a",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(penguins, aes(x=\"species\", y=\"body_mass_g\")) + geom_boxplot())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97b24caa",
   "metadata": {},
   "source": [
    "Alternatively, we can make probability density plots with `geom_density()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b85a2df",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(penguins, aes(x=\"body_mass_g\", color=\"species\")) + geom_density(size=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0c10ae4",
   "metadata": {},
   "source": [
    "We've also customized the thickness of the lines using the `size` argument in order to make them stand out a bit more against the background.\n",
    "\n",
    "Additionally, we can map `species` to both `color` and `fill` aesthetics and use the `alpha` aesthetic to add transparency to the filled density curves.\n",
    "This aesthetic takes values between 0 (completely transparent) and 1 (completely opaque).\n",
    "In the following plot it's *set* to 0.5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "353189e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(penguins, aes(x=\"body_mass_g\", color=\"species\", fill=\"species\"))\n",
    "    + geom_density(alpha=0.5)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a2c7d59",
   "metadata": {},
   "source": [
    "Note the terminology we have used here:\n",
    "\n",
    "-   We *map* variables to aesthetics if we want the visual attribute represented by that aesthetic to vary based on the values of that variable.\n",
    "-   Otherwise, we *set* the value of an aesthetic.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63de3309",
   "metadata": {},
   "source": [
    "### Two categorical variables\n",
    "\n",
    "We can use stacked bar plots to visualise the relationship between two categorical variables.\n",
    "\n",
    "For example, the following two stacked bar plots both display the relationship between `island` and `species`, or specifically, visualising the distribution of `species` within each island.\n",
    "\n",
    "The first plot shows the frequencies of each species of penguins on each island.\n",
    "The plot of frequencies show that there are equal numbers of Adelies on each island.\n",
    "\n",
    "But we don't have a good sense of the percentage balance within each island."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e091e211",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(penguins, aes(x=\"island\", fill=\"species\")) + geom_bar())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e34c211",
   "metadata": {},
   "source": [
    "The second plot is a relative frequency plot, created by setting `position = \"fill\"` in the geom is more useful for comparing species distributions across islands since it's not affected by the unequal numbers of penguins across the islands.\n",
    "\n",
    "Using this plot we can see that Gentoo penguins all live on Biscoe island and make up roughly 75% of the penguins on that island, Chinstrap all live on Dream island and make up roughly 50% of the penguins on that island, and Adelie live on all three islands and make up all of the penguins on Torgersen.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7df8fb7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(penguins, aes(x=\"island\", fill=\"species\")) + geom_bar(position=\"fill\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc83c3db",
   "metadata": {},
   "source": [
    "In creating these bar charts, we map the variable that will be separated into bars to the `x` aesthetic, and the variable that will change the colors inside the bars to the `fill` aesthetic."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f77e5c39",
   "metadata": {},
   "source": [
    "### Two numerical variables\n",
    "\n",
    "So far you've learned about scatterplots (created with `geom_point()`) and smooth curves (created with `geom_smooth()`) for visualising the relationship between two numerical variables.\n",
    "A scatterplot is probably the most commonly used plot for visualising the relationship between two numerical variables.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5066527d",
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(penguins, aes(x=\"flipper_length_mm\", y=\"body_mass_g\")) + geom_point())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "427f22c9",
   "metadata": {},
   "source": [
    "### Three or more variables\n",
    "\n",
    "As we saw already, we can incorporate more variables into a plot by mapping them to additional aesthetics.\n",
    "\n",
    "For example, in the following scatterplot the colors of points represent species and the shapes of points represent islands.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ca23d34",
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(penguins, aes(x=\"flipper_length_mm\", y=\"body_mass_g\"))\n",
    "    + geom_point(aes(color=\"species\", shape=\"island\"))\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14d0aacb",
   "metadata": {},
   "source": [
    "However adding too many aesthetic mappings to a plot makes it cluttered and difficult to make sense of.\n",
    "\n",
    "Another way, which is particularly useful for categorical variables, is to split your plot into **facets** (also known as **small multiples**), subplots that each display one subset of the data.\n",
    "\n",
    "To facet your plot by a single variable, use `facet_wrap()`.\n",
    "\n",
    "The first argument of `facet_wrap()` tells the function what variable to have in successive charts. The variable that you pass to `facet_wrap()` should be categorical."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00dd36e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(penguins, aes(x=\"flipper_length_mm\", y=\"body_mass_g\"))\n",
    "    + geom_point(aes(color=\"species\", shape=\"species\"))\n",
    "    + facet_wrap(facets=\"island\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee5a3eed",
   "metadata": {},
   "source": [
    "You will learn about many other geoms for visualising distributions of variables and relationships between them in later chapters."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4566786",
   "metadata": {},
   "source": [
    "### Exercises\n",
    "\n",
    "1.  Make a scatterplot of `bill_depth_mm` vs. `bill_length_mm` and color the points by `species`.\n",
    "    What does adding coloring by species reveal about the relationship between these two variables?\n",
    "    What about faceting by `species`?\n",
    "\n",
    "2.  Why does the following yield two separate legends?\n",
    "    How would you fix it to combine the two legends?\n",
    "\n",
    "    ```python\n",
    "    (\n",
    "        ggplot(\n",
    "          data = penguins,\n",
    "          mapping = aes(\n",
    "            x = \"bill_length_mm\", y = \"bill_depth_mm\", \n",
    "            color = \"species\", shape = \"species\"\n",
    "          )\n",
    "        ) +\n",
    "          geom_point() +\n",
    "          labs(color = \"Species\")\n",
    "    )\n",
    "    ```\n",
    "\n",
    "3.  Create the two following stacked bar plots.\n",
    "    Which question can you answer with the first one?\n",
    "    Which question can you answer with the second one?\n",
    "\n",
    "    ```python\n",
    "    ggplot(penguins, aes(x = \"island\", fill = \"species\")) +\n",
    "      geom_bar(position = \"fill\")\n",
    "    ggplot(penguins, aes(x = \"species\", fill = \"island\")) +\n",
    "      geom_bar(position = \"fill\")\n",
    "    ```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "150fbd1f",
   "metadata": {},
   "source": [
    "## Saving your plots\n",
    "\n",
    "Once you've made a plot, you might want to save it as an image that you can use elsewhere.\n",
    "That's the job of `ggsave()`, which will save the plot most recently created to disk:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3410634b",
   "metadata": {},
   "outputs": [],
   "source": [
    "plotted_data = (\n",
    "    ggplot(penguins, aes(x=\"flipper_length_mm\", y=\"body_mass_g\")) + geom_point()\n",
    ")\n",
    "ggsave(plotted_data, filename=\"penguin-plot.svg\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee6fa32c",
   "metadata": {},
   "source": [
    "This saved the figure to disk at the location shown—by default it's in a subdirectory called \"lets-plot-images\".\n",
    "\n",
    "We used the file format \"svg\". There are lots of output options to choose from to save your file to. Remember that, for graphics, *vector formats* are generally better than *raster formats*. In practice, this means saving plots in svg or pdf formats over jpg or png file formats. The svg format works in a lot of contexts (including Microsoft Word) and is a good default. To choose between formats, just supply the file extension and the file type will change automatically, eg \"chart.svg\" for svg or \"chart.png\" for png. You can also save figures in HTML format.\n",
    "\n",
    "If you're using a raster format then you'll need to specify how big the figure is via the *scale* keyword argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "852afe51",
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "import shutil\n",
    "\n",
    "shutil.rmtree(\"lets-plot-images\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2987bf18",
   "metadata": {},
   "source": [
    "### Exercises\n",
    "\n",
    "1. Save the figure above as a PNG. Try varying the scale."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1390edc2",
   "metadata": {},
   "source": [
    "## Common Problems\n",
    "\n",
    "As you start to run code, you're likely to run into problems.\n",
    "Don't worry—it happens to everyone.\n",
    "We have all been writing Python code for years, but every day we still write code that doesn't work on the first try!\n",
    "\n",
    "Start by carefully comparing the code that you're running to the code in the book: A misplaced character can make all the difference!\n",
    "Make sure that every `(` is matched with a `)` and every `\"` is paired with another `\"`. In  Visual Studio Code, you can get extensions that colour match brackets so you can easily see if you closed them or not.\n",
    "\n",
    "Sometimes you'll run the code and nothing happens.\n",
    "\n",
    "For those coming from the R statistical programming language, you may be concerned about getting your `+` in the wrong place. Have no fear, however, as in the syntax for **letsplot** the `+` can go at the start or the end of the line.\n",
    "\n",
    "\n",
    "If you're still stuck, try the help.\n",
    "You can get help about any Python function by running `help(function_name)` in the interactive window.\n",
    "Don't worry if the help doesn't seem that helpful - instead skip down to the examples and look for code that matches what you're trying to do.\n",
    "\n",
    "If you're still stuck, check out the **letsplot** [documentation](https://lets-plot.org/) or doing a Google search (especially helpful for error messages).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f33dc022",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "In this chapter, you've learned the basics of data visualisation with **letsplot**.\n",
    "We started with the basic idea that underpins **letsplot**: a visualisation is a mapping from variables in your data to aesthetic properties like position, colour, size and shape.\n",
    "You then learned about increasing the complexity and improving the presentation of your plots layer-by-layer.\n",
    "You also learned about commonly used plots for visualising the distribution of a single variable as well as for visualising relationships between two or more variables, by leveraging additional aesthetic mappings and/or splitting your plot into small multiples using faceting.\n",
    "\n",
    "We'll use visualisations again and again throughout this book, introducing new techniques as we need them as well as do a deeper dive into creating visualisations with **letsplot** in subsequent chapters.\n",
    "\n",
    "With the basics of visualisation under your belt, in the next chapter we're going to switch gears a little and give you some practical workflow advice.\n",
    "We intersperse workflow advice with data science tools throughout this part of the book because it'll help you stay organised as you write more Python code."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "encoding": "# -*- coding: utf-8 -*-",
   "formats": "md:myst",
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  },
  "toc-showtags": true
 },
 "nbformat": 4,
 "nbformat_minor": 5
}