working-with-foreign-functions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Working with Foreign Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Suppose your Python library needs to load some sensitive binary data from a file into a contiguous block of memory (*e.g.*, in order to use it for some application-specific operation). Furthermore, you have some additional requirements that must be met for security and auditing purposes:\n",
    "\n",
    "* you need to ensure that your code does not inadvertently cause the interpreter to copy any part of the data loaded from the file into some other region of memory,\n",
    "* you need to log the memory address at which the data was stored, and\n",
    "* you need to clear the memory region that held the data by overwriting it with random bytes.\n",
    "\n",
    "One strategy you might employ in order maintain tight control over what your code is doing is to use C functions found in a compiled shared library to read the data from disk, to load that data into a region of memory, and at the end to clear that region. What minimal collection of built-in Python features will you need to invoke functions that are found in a shared library? How can you transform Python values (such as strings representing the location of the file) into an appropriate form on which the function can operate? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Python offers a rich set of capabilities via the built-in [ctypes](https://docs.python.org/3/library/ctypes.html) library that make it possible to invoke (or wrap in a Python function) foreign functions that have been implemented using another language (such as C/C++) and compiled into shared libraries. This article reviews the basics of employing foreign functions by demonstrating how to load and apply to the above use case the instance of the [GNU C Library](https://www.gnu.org/software/libc/) available on most operating systems. The same techniques can be used for any shared library. An alternative approach used by some popular Python packages is briefly reviewed, as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Loading a Shared Library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "To load a shared library file for which you know the relative or absolute path, you can normally use the [`LoadLibrary`](https://docs.python.org/3/library/ctypes.html#loading-dynamic-link-libraries) method of either the `cdll` or the `windll` instance (depending on your operating system) of the [`LibraryLoader`](https://docs.python.org/3/library/ctypes.html#ctypes.LibraryLoader) class found in [ctypes](https://docs.python.org/3/library/ctypes.html). For the purposes of the use case in this article, it is sufficient to load the GNU C Library. In the example below, the [`system`](https://docs.python.org/3/library/platform.html#platform.system) function from the [platform](https://docs.python.org/3/library/platform.html) library is used to distinguish between Windows and Linux/macOS environments. In the Linux/macOS case, the [`find_library`](https://docs.python.org/3/library/ctypes.html#finding-shared-libraries) function is used to determine the absolute path of the shared library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import ctypes\n",
    "import platform\n",
    "\n",
    "if platform.system() == \"Windows\":\n",
    "    libc = ctypes.windll.msvcrt\n",
    "else:\n",
    "    libc = ctypes.cdll.LoadLibrary(ctypes.util.find_library(\"c\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Invoking Foreign Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The first portion of your workflow involves loading a file into memory. The Python code below writes a file to disk that contains a sequence of 32 random bytes. The file can be used to test the workflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from secrets import token_bytes\n",
    "with open(\"data.txt\", \"wb\") as file:\n",
    "    file.write(token_bytes(32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The C function [`fopen`](https://www.gnu.org/software/libc/manual/html_node/Opening-Streams.html) expects two arguments: a pointer to the first character of a string that represents the path of the file, and a pointer to the first character of the string that represents the mode (*i.e.*, reading or writing) in which the file is opened. You can use the [`c_char_p`](https://docs.python.org/3/library/ctypes.html#ctypes.c_char_p) function to turn Python strings into a representation in memory that can be handled by the C function. Note the use of the [`encode`](https://docs.python.org/3/library/stdtypes.html#str.encode) string method to provide an explicit encoding for the string as a byte sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from ctypes import c_char_p\n",
    "file = c_char_p(\"data.txt\".encode(\"ascii\"))\n",
    "mode = c_char_p(\"rb\".encode(\"ascii\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Unfortunately, it is not possible within Python to examine the `libc` object that was created by the [`LibraryLoader`](https://docs.python.org/3/library/ctypes.html#ctypes.LibraryLoader) instance in order to determine what symbols are defined within it. However, in this case we know that the functions `fopen`, `fread`, and `fclose` must exist. For each of these functions, an instance of the [`FuncPtr`](https://docs.python.org/3/library/ctypes.html#ctypes._FuncPtr) class can be found in `libc`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "fopen = libc.fopen\n",
    "fread = libc.fread\n",
    "fclose = libc.fclose"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Before you can safely invoke these functions, you need to [specify their argument types](https://docs.python.org/3/library/ctypes.html#specifying-the-required-argument-types-function-prototypes) and their [return types](https://docs.python.org/3/library/ctypes.html#return-types). This can be accomplished by first consulting the [GNU C Library](https://www.gnu.org/software/libc/) documentation to find the signature for each of the C functions you would like to use. Then, the appropriate [data type classes](https://docs.python.org/3/library/ctypes.html#fundamental-data-types) can be used to assign the correct sequence of argument types and the correct return type to the `argtypes` and `restype` attributes, respectively, of each `FuncPtr` class instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from ctypes import c_int, c_size_t, c_void_p\n",
    "\n",
    "fopen.argtypes = [c_char_p, c_char_p]\n",
    "fopen.restype = c_void_p\n",
    "\n",
    "fread.argtypes = [c_void_p, c_size_t, c_size_t, c_void_p]\n",
    "fread.restype = c_size_t\n",
    "\n",
    "fclose.argtypes = [c_void_p]\n",
    "fclose.restype = c_int"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "It is now possible to invoke these functions on some inputs. You can allocate a memory buffer for the 32 bytes of data that you will be loading from the file using the [`create_string_buffer`](https://docs.python.org/3/library/ctypes.html#ctypes.create_string_buffer) function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from ctypes import create_string_buffer\n",
    "data = ctypes.create_string_buffer(32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "You can now open the file, load the data, and close the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'fa2f53dd629ffad90033c0d531ef33d408270dc59fdb95d0df0e10f6aae976e2'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fp = fopen(file, mode)\n",
    "fread(data, 32, 1, fp)\n",
    "fclose(fp)\n",
    "bytes(data).hex()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "You can determine the memory address corresponding to the memory buffer `data` using the [`addressof`](https://docs.python.org/3/library/ctypes.html#ctypes.addressof) function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0xe8e740'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from ctypes import addressof\n",
    "hex(addressof(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "You can now clear the memory region. The example below uses the `memset` C function for this purpose. An example that uses a random sequence generator that is appropriate for cryptographic applications appears in the next section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0000000000000000000000000000000000000000000000000000000000000000'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "libc.memset.argtypes = [c_void_p, c_int, c_size_t]\n",
    "libc.memset(data, 0, 32)\n",
    "bytes(data).hex()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Alternative Approaches"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The [C Foreign Function Interface](https://cffi.readthedocs.io/) library is similar to the built-in ctypes module and is used by some popular packages, including the [PyNaCl](https://pypi.org/project/PyNaCl/) library that acts as a Python interface for the cryptographic library [libsodium](https://doc.libsodium.org/). In the example below, the C implementation of the [`randombytes`](https://libsodium.gitbook.io/doc/generating_random_data) function is invoked on a character buffer `bs` and then the contents of that buffer are displayed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'c16f995e12206cb6'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nacl import _sodium\n",
    "lib = _sodium.lib\n",
    "\n",
    "from cffi import FFI\n",
    "ffi = FFI()\n",
    "\n",
    "bs = ffi.new(\"unsigned char[]\", 8)\n",
    "lib.randombytes(bs, 8)\n",
    "bytes(bs).hex()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "You might choose to call the C implementation directly for a variety of reasons, including to improve performance. This may be useful to do when a more high-level library method allocates new memory for a byte sequence during every invocation, while your own solution can reuse the same memory over and over to store each new batch of bytes. In the example below, the time to invoke the C function over one million iterations is measured."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.4877797999652103"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import time\n",
    "start = time.perf_counter()\n",
    "for _ in range(10**6):\n",
    "    lib.randombytes(bs, 8)\n",
    "time.perf_counter() - start"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The below example measures the amount of time it takes to invoke the Python wrapper in PyNaCl over the same number of iterations. The longer running time may be the result of a number of factors; regardless of the underlying reason that may apply for any particular function, the example demonstrates that direct access to the C method gives you more control over those factors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6.109586499980651\n"
     ]
    }
   ],
   "source": [
    "from nacl.bindings import randombytes\n",
    "start = time.perf_counter()\n",
    "for _ in range(10**6):\n",
    "    bs = randombytes(8)\n",
    "print(time.perf_counter() - start)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Further Reading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "If you are interested in learning what other C functions can be used in the manner described in this article, you can review [The GNU C Library Reference Manual](https://www.gnu.org/software/libc/manual/html_node/index.html). In addition to [ctypes](https://docs.python.org/3/library/ctypes.html) and [CFFI](https:/eli/cffi.readthedocs.io/), there exist specialized variants such as the [NumPy](https://numpy.org/)-specific [`numpy.ctypeslib`](https://numpy.org/doc/stable/reference/routines.ctypeslib.html) library (which comes with features that make it easier to package and deliver NumPy data structures to C functions). It is also possible to implement extension modules for Python in C/C++. Useful [definitions and guidelines](https://docs.python.org/3/extending/extending.html) are provided that make it possible to write C/C++ code that interacts in appropriate ways with the Python interpreter. If you would like to leverage even more interoperation between Python and C/C++ code, you can investigate the [Cython](https://cython.org/) compiler."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  },
  "livereveal": {
   "scroll": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}