{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Abstract\n",
    "\n",
    "[**hdf5plugin**](https://github.com/silx-kit/hdf5plugin) is a *Python* package (1) providing a set of [**HDF5**](https://portal.hdfgroup.org/display/HDF5/) compression filters (namely: blosc, bitshuffle, lz4, FCIDECOMP, ZFP, Zstandard) and (2) enabling their use from the *Python* programming language with [**h5py**](https://docs.h5py.org/) a thin, pythonic wrapper around `libHDF5`.\n",
    "\n",
    "This presentation illustrates how to use **hdf5plugin** for reading and writing compressed datasets from *Python* and gives an overview of the different HDF5 compression filters it provides.\n",
    "\n",
    "It also illustrates how the provided compression filters can be enabled to read compressed datasets from other (non-*Python*) application.\n",
    "\n",
    "Finally, it discusses how **hdf5plugin** manages to distribute the HDF5 plugins for reuse with different `libHDF5`.\n",
    "\n",
    "License: [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbsphinx": "hidden",
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# Notebook requirements\n",
    "# A recent version of Pillow is required!\n",
    "%pip install numpy h5py hdf5plugin h5glance rise jupyterlab matplotlib ipympl Pillow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbsphinx": "hidden",
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "# Creates data.h5 used for demos\n",
    "from matplotlib import pyplot as plt\n",
    "import h5py\n",
    "import hdf5plugin\n",
    "import numpy\n",
    "from PIL import Image\n",
    "import urllib.request\n",
    "\n",
    "url = \"https://www.eiroforum.org/wp-content/uploads/pg-esrf-01.jpg\"\n",
    "filename = urllib.request.urlretrieve(url)[0]\n",
    "image = numpy.array(Image.open(filename).convert(\"L\"))\n",
    "plt.imshow(image)\n",
    "\n",
    "h5file = h5py.File(\"data.h5\", mode=\"w\")\n",
    "h5file[\"copyright\"] = \"P.Ginter/ESRF\"\n",
    "h5file.attrs[\"url\"] = url\n",
    "h5file.create_dataset(\"/data\", data=image)\n",
    "h5file.create_dataset(\"/compressed_data_bitshuffle_lz4\", data=image, **hdf5plugin.Bitshuffle())\n",
    "h5file.close()\n",
    "\n",
    "# Restart kernel once the file is created!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "nbsphinx": "hidden",
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbsphinx": "hidden",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![ESRF logo](data:image/png;base64,[encoded data, 462192 characters] \"ESRF logo\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# hdf5plugin\n",
    "\n",
    "[**hdf5plugin**](https://github.com/silx-kit/hdf5plugin) packages a set of [HDF5](https://portal.hdfgroup.org/display/HDF5/) compression filters (namely: blosc, bitshuffle, lz4, FCIDECOMP, ZFP, Zstandard) and makes them usable from the Python programming language through [h5py](https://docs.h5py.org/).\n",
    "\n",
    "[**h5py**](https://docs.h5py.org/) is a thin, pythonic wrapper around [HDF5](https://portal.hdfgroup.org/display/HDF5/).\n",
    "\n",
    "&nbsp;\n",
    "\n",
    "&nbsp;\n",
    "\n",
    "Presenter: Thomas VINCENT\n",
    "\n",
    "European HDF Users Group Summer 2021, July 7-8, 2021"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>/* Based on https://css-tricks.com/snippets/css/nested-expandable-folders/\n",
       "   Reused with permission from https://css-tricks.com/license/\n",
       "*/\n",
       "\n",
       ".h5glance-css-treeview ul,\n",
       ".h5glance-css-treeview li\n",
       "{\n",
       "\tpadding: 0;\n",
       "\tmargin: 0;\n",
       "    margin-left: 12px;\n",
       "\tlist-style: none;\n",
       "}\n",
       "\n",
       "/* Override padding from Jupyter CSS */\n",
       ".rendered_html .h5glance-css-treeview ul {\n",
       "    padding: 0;\n",
       "}\n",
       "\n",
       "/* Make the checkbox itself invisible */\n",
       ".h5glance-css-treeview input\n",
       "{\n",
       "\tposition: absolute;\n",
       "\topacity: 0;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview\n",
       "{\n",
       "    font-family: sans-serif;\n",
       "\t-moz-user-select: none;\n",
       "\t-webkit-user-select: none;\n",
       "\tuser-select: none;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview code {\n",
       "    font-size: 12pt;\n",
       "    font-family: \"courier\", monospace;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview a.h5glance-dataset-copylink\n",
       "{\n",
       "    text-decoration: none;\n",
       "}\n",
       "\n",
       "/* These next two bits drive the expanding/collapsing behaviour */\n",
       ".h5glance-css-treeview input + label + ul\n",
       "{\n",
       "\tdisplay: none;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview input:checked:not(:disabled) + label + ul\n",
       "{\n",
       "\tdisplay: block;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview label,\n",
       ".h5glance-css-treeview label::before\n",
       "{\n",
       "\tdisplay: inline-block;\n",
       "\theight: 16px;\n",
       "\tline-height: 16px;,\n",
       "\tvertical-align: middle;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview label\n",
       "{\n",
       "    margin-left: -1.5em;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview label::before\n",
       "{\n",
       "\tcontent: \"\\229e\"; /* squared plus */\n",
       "\twidth: 1.5em;\n",
       "\tvertical-align: middle;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview input:checked + label::before\n",
       "{\n",
       "    content: \"\\229f\";  /* squared minus */\n",
       "}\n",
       "\n",
       "/* webkit adjacent element selector bugfix */\n",
       "@media screen and (-webkit-min-device-pixel-ratio:0)\n",
       "{\n",
       "\t.h5glance-css-treeview\n",
       "\t{\n",
       "\t\t-webkit-animation: webkit-adjacent-element-selector-bugfix infinite 1s;\n",
       "\t}\n",
       "\n",
       "\t@-webkit-keyframes webkit-adjacent-element-selector-bugfix\n",
       "\t{\n",
       "\t\tfrom\n",
       "\t\t{\n",
       "\t\t\tpadding: 0;\n",
       "\t\t}\n",
       "\t\tto\n",
       "\t\t{\n",
       "\t\t\tpadding: 0;\n",
       "\t\t}\n",
       "\t}\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview li.h5glance-dataset {\n",
       "    margin-bottom: 3px;\n",
       "}\n",
       "\n",
       ".h5glance-dataset-name {\n",
       "    font-weight: bold;\n",
       "}\n",
       "</style><div id=\"h5glance-container-0\" class=\"h5glance-css-treeview\"><ul><li><input checked=\"checked\" id=\"h5glance-expand-switch-0\" type=\"checkbox\"/><label for=\"h5glance-expand-switch-0\">data.h5</label><ul><li class=\"h5glance-dataset\"><span class=\"h5glance-dataset-name\">compressed_data_bitshuffle_lz4</span> <a data-hdf5-path=\"/compressed_data_bitshuffle_lz4\" href=\"#\" class=\"h5glance-dataset-copylink\">[📋]</a>: 1969 × 2961 entries, dtype: <abbr title=\"8-bit unsigned integer\">uint8</abbr></li><li class=\"h5glance-dataset\"><span class=\"h5glance-dataset-name\">copyright</span> <a data-hdf5-path=\"/copyright\" href=\"#\" class=\"h5glance-dataset-copylink\">[📋]</a>: scalar entries, dtype: UTF-8 string</li><li class=\"h5glance-dataset\"><span class=\"h5glance-dataset-name\">data</span> <a data-hdf5-path=\"/data\" href=\"#\" class=\"h5glance-dataset-copylink\">[📋]</a>: 1969 × 2961 entries, dtype: <abbr title=\"8-bit unsigned integer\">uint8</abbr></li></ul></li></ul></div><script>// Based on code by Stackoverflow user Dean Taylor\n",
       "// https://stackoverflow.com/questions/400212/how-do-i-copy-to-the-clipboard-in-javascript/30810322\n",
       "// Used under Stackoverflow's CC-BY-SA 3.0 license\n",
       "\n",
       "(function() {\n",
       "    function copyTextToClipboard(text) {\n",
       "        let textArea = document.createElement(\"textarea\");\n",
       "\n",
       "        //\n",
       "        // *** This styling is an extra step which is likely not required. ***\n",
       "        //\n",
       "        // Why is it here? To ensure:\n",
       "        // 1. the element is able to have focus and selection.\n",
       "        // 2. if element was to flash render it has minimal visual impact.\n",
       "        // 3. less flakyness with selection and copying which **might** occur if\n",
       "        //    the textarea element is not visible.\n",
       "        //\n",
       "        // The likelihood is the element won't even render, not even a flash,\n",
       "        // so some of these are just precautions. However in IE the element\n",
       "        // is visible whilst the popup box asking the user for permission for\n",
       "        // the web page to copy to the clipboard.\n",
       "        //\n",
       "\n",
       "        // Place in top-left corner of screen regardless of scroll position.\n",
       "        textArea.style.position = 'fixed';\n",
       "        textArea.style.top = 0;\n",
       "        textArea.style.left = 0;\n",
       "\n",
       "        // Ensure it has a small width and height. Setting to 1px / 1em\n",
       "        // doesn't work as this gives a negative w/h on some browsers.\n",
       "        textArea.style.width = '2em';\n",
       "        textArea.style.height = '2em';\n",
       "\n",
       "        // We don't need padding, reducing the size if it does flash render.\n",
       "        textArea.style.padding = 0;\n",
       "\n",
       "        // Clean up any borders.\n",
       "        textArea.style.border = 'none';\n",
       "        textArea.style.outline = 'none';\n",
       "        textArea.style.boxShadow = 'none';\n",
       "\n",
       "        // Avoid flash of white box if rendered for any reason.\n",
       "        textArea.style.background = 'transparent';\n",
       "\n",
       "\n",
       "        textArea.value = text;\n",
       "\n",
       "        document.body.appendChild(textArea);\n",
       "        textArea.focus();\n",
       "        textArea.select();\n",
       "\n",
       "        try {\n",
       "            if (!document.execCommand('copy')) {\n",
       "                console.log(\"Unable to copy text with document.execCommand()\");\n",
       "            }\n",
       "        } finally {\n",
       "            document.body.removeChild(textArea);\n",
       "        }\n",
       "    }\n",
       "\n",
       "    function copy_event_handler(event) {\n",
       "        copyTextToClipboard(event.target.dataset.hdf5Path);\n",
       "        event.preventDefault();\n",
       "    }\n",
       "\n",
       "    function enable_copylinks(parent) {\n",
       "        let links = parent.querySelectorAll(\".h5glance-dataset-copylink\");\n",
       "        links.forEach(function (link) {\n",
       "            link.addEventListener(\"click\", copy_event_handler);\n",
       "        });\n",
       "    }\n",
       "\n",
       "    // The code to actually trigger this is substituted below.\n",
       "    \n",
       "enable_copylinks(document.getElementById(\"h5glance-container-0\"));\n",
       "\n",
       "})();\n",
       "</script></div>"
      ],
      "text/plain": [
       "<h5glance.generate.H5Glance at 0x115641c88>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from h5glance import H5Glance  # Browsing HDF5 files\n",
    "H5Glance(\"data.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.colorbar.Colorbar at 0x119479358>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 112616 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import h5py  # Pythonic HDF5 wrapper: https://docs.h5py.org/\n",
    "\n",
    "h5file = h5py.File(\"data.h5\", mode=\"r\")  # Open HDF5 file in read mode\n",
    "data = h5file[\"/data\"][()]               # Access HDF5 dataset \"/data\"\n",
    "plt.imshow(data); plt.colorbar()         # Display data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "ename": "OSError",
     "evalue": "Can't read data (can't open directory: /usr/local/hdf5/lib/plugin)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mOSError\u001b[0m                                   Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-4-4bb532391a0f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mh5file\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"/compressed_data_bitshuffle_lz4\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m  \u001b[0;31m# Access compressed dataset\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32mh5py/_objects.pyx\u001b[0m in \u001b[0;36mh5py._objects.with_phil.wrapper\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mh5py/_objects.pyx\u001b[0m in \u001b[0;36mh5py._objects.with_phil.wrapper\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32m~/venv/py37env/lib/python3.7/site-packages/h5py/_hl/dataset.py\u001b[0m in \u001b[0;36m__getitem__\u001b[0;34m(self, args, new_dtype)\u001b[0m\n\u001b[1;32m    760\u001b[0m         \u001b[0mmspace\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mh5s\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcreate_simple\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mselection\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmshape\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    761\u001b[0m         \u001b[0mfspace\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mselection\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mid\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 762\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mid\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmspace\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfspace\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0marr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdxpl\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_dxpl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    763\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    764\u001b[0m         \u001b[0;31m# Patch up the output for NumPy\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mh5py/_objects.pyx\u001b[0m in \u001b[0;36mh5py._objects.with_phil.wrapper\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mh5py/_objects.pyx\u001b[0m in \u001b[0;36mh5py._objects.with_phil.wrapper\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mh5py/h5d.pyx\u001b[0m in \u001b[0;36mh5py.h5d.DatasetID.read\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mh5py/_proxy.pyx\u001b[0m in \u001b[0;36mh5py._proxy.dset_rw\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mOSError\u001b[0m: Can't read data (can't open directory: /usr/local/hdf5/lib/plugin)"
     ]
    }
   ],
   "source": [
    "data = h5file[\"/compressed_data_bitshuffle_lz4\"][()]  # Access compressed dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `hdf5plugin` usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Reading compressed datasets\n",
    "\n",
    "To enable reading compressed datasets not supported by `libHDF5` and `h5py`:\n",
    "Install **hdf5plugin** & import it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "pip3 install hdf5plugin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Or: `conda install -c conda-forge hdf5plugin`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import hdf5plugin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.colorbar.Colorbar at 0x11bc666d8>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 112616 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = h5file[\"/compressed_data_bitshuffle_lz4\"][()]  # Access datset\n",
    "plt.imshow(data); plt.colorbar()                      # Display data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "h5file.close()  # Close the HDF5 file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Writing compressed datasets\n",
    "\n",
    "When writing datasets with `h5py`, compression can be specified with: [h5py.Group.create_dataset](https://docs.h5py.org/en/stable/high/group.html?highlight=create_dataset#h5py.Group.create_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Create a dataset with h5py without compression\n",
    "h5file = h5py.File(\"new_file_uncompressed.h5\", mode=\"w\")\n",
    "h5file.create_dataset(\"/data\", data=data)\n",
    "h5file.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Create a compressed dataset\n",
    "h5file = h5py.File(\"new_file_bitshuffle_lz4.h5\", mode=\"w\")\n",
    "h5file.create_dataset(\n",
    "    \"/compressed_data_bitshuffle_lz4\",\n",
    "    data=data,\n",
    "    compression=32008,  # bitshuffle/lz4 HDF5 filter identifier\n",
    "    compression_opts=(0, 2)  # options: default number of elements/block, enable LZ4\n",
    ")\n",
    "h5file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "`hdf5plugin` provides some [helpers](http://www.silx.org/doc/hdf5plugin/latest/usage.html#write-compressed-datasets) to ease dealing with compression filter and options:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "h5file = h5py.File(\"new_file_bitshuffle_lz4.h5\", mode=\"w\")\n",
    "h5file.create_dataset(\n",
    "    \"/compressed_data_bitshuffle_lz4\",\n",
    "    data=data,\n",
    "    **hdf5plugin.Bitshuffle()  # Or: **hdf5plugin.BitShuffle(lz4=True)\n",
    ")\n",
    "h5file.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "hdf5plugin.Bitshuffle?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>/* Based on https://css-tricks.com/snippets/css/nested-expandable-folders/\n",
       "   Reused with permission from https://css-tricks.com/license/\n",
       "*/\n",
       "\n",
       ".h5glance-css-treeview ul,\n",
       ".h5glance-css-treeview li\n",
       "{\n",
       "\tpadding: 0;\n",
       "\tmargin: 0;\n",
       "    margin-left: 12px;\n",
       "\tlist-style: none;\n",
       "}\n",
       "\n",
       "/* Override padding from Jupyter CSS */\n",
       ".rendered_html .h5glance-css-treeview ul {\n",
       "    padding: 0;\n",
       "}\n",
       "\n",
       "/* Make the checkbox itself invisible */\n",
       ".h5glance-css-treeview input\n",
       "{\n",
       "\tposition: absolute;\n",
       "\topacity: 0;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview\n",
       "{\n",
       "    font-family: sans-serif;\n",
       "\t-moz-user-select: none;\n",
       "\t-webkit-user-select: none;\n",
       "\tuser-select: none;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview code {\n",
       "    font-size: 12pt;\n",
       "    font-family: \"courier\", monospace;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview a.h5glance-dataset-copylink\n",
       "{\n",
       "    text-decoration: none;\n",
       "}\n",
       "\n",
       "/* These next two bits drive the expanding/collapsing behaviour */\n",
       ".h5glance-css-treeview input + label + ul\n",
       "{\n",
       "\tdisplay: none;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview input:checked:not(:disabled) + label + ul\n",
       "{\n",
       "\tdisplay: block;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview label,\n",
       ".h5glance-css-treeview label::before\n",
       "{\n",
       "\tdisplay: inline-block;\n",
       "\theight: 16px;\n",
       "\tline-height: 16px;,\n",
       "\tvertical-align: middle;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview label\n",
       "{\n",
       "    margin-left: -1.5em;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview label::before\n",
       "{\n",
       "\tcontent: \"\\229e\"; /* squared plus */\n",
       "\twidth: 1.5em;\n",
       "\tvertical-align: middle;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview input:checked + label::before\n",
       "{\n",
       "    content: \"\\229f\";  /* squared minus */\n",
       "}\n",
       "\n",
       "/* webkit adjacent element selector bugfix */\n",
       "@media screen and (-webkit-min-device-pixel-ratio:0)\n",
       "{\n",
       "\t.h5glance-css-treeview\n",
       "\t{\n",
       "\t\t-webkit-animation: webkit-adjacent-element-selector-bugfix infinite 1s;\n",
       "\t}\n",
       "\n",
       "\t@-webkit-keyframes webkit-adjacent-element-selector-bugfix\n",
       "\t{\n",
       "\t\tfrom\n",
       "\t\t{\n",
       "\t\t\tpadding: 0;\n",
       "\t\t}\n",
       "\t\tto\n",
       "\t\t{\n",
       "\t\t\tpadding: 0;\n",
       "\t\t}\n",
       "\t}\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview li.h5glance-dataset {\n",
       "    margin-bottom: 3px;\n",
       "}\n",
       "\n",
       ".h5glance-dataset-name {\n",
       "    font-weight: bold;\n",
       "}\n",
       "</style><div id=\"h5glance-container-1\" class=\"h5glance-css-treeview\"><ul><li><input checked=\"checked\" id=\"h5glance-expand-switch-1\" type=\"checkbox\"/><label for=\"h5glance-expand-switch-1\">new_file_bitshuffle_lz4.h5</label><ul><li class=\"h5glance-dataset\"><span class=\"h5glance-dataset-name\">compressed_data_bitshuffle_lz4</span> <a data-hdf5-path=\"/compressed_data_bitshuffle_lz4\" href=\"#\" class=\"h5glance-dataset-copylink\">[📋]</a>: 1969 × 2961 entries, dtype: <abbr title=\"8-bit unsigned integer\">uint8</abbr></li></ul></li></ul></div><script>// Based on code by Stackoverflow user Dean Taylor\n",
       "// https://stackoverflow.com/questions/400212/how-do-i-copy-to-the-clipboard-in-javascript/30810322\n",
       "// Used under Stackoverflow's CC-BY-SA 3.0 license\n",
       "\n",
       "(function() {\n",
       "    function copyTextToClipboard(text) {\n",
       "        let textArea = document.createElement(\"textarea\");\n",
       "\n",
       "        //\n",
       "        // *** This styling is an extra step which is likely not required. ***\n",
       "        //\n",
       "        // Why is it here? To ensure:\n",
       "        // 1. the element is able to have focus and selection.\n",
       "        // 2. if element was to flash render it has minimal visual impact.\n",
       "        // 3. less flakyness with selection and copying which **might** occur if\n",
       "        //    the textarea element is not visible.\n",
       "        //\n",
       "        // The likelihood is the element won't even render, not even a flash,\n",
       "        // so some of these are just precautions. However in IE the element\n",
       "        // is visible whilst the popup box asking the user for permission for\n",
       "        // the web page to copy to the clipboard.\n",
       "        //\n",
       "\n",
       "        // Place in top-left corner of screen regardless of scroll position.\n",
       "        textArea.style.position = 'fixed';\n",
       "        textArea.style.top = 0;\n",
       "        textArea.style.left = 0;\n",
       "\n",
       "        // Ensure it has a small width and height. Setting to 1px / 1em\n",
       "        // doesn't work as this gives a negative w/h on some browsers.\n",
       "        textArea.style.width = '2em';\n",
       "        textArea.style.height = '2em';\n",
       "\n",
       "        // We don't need padding, reducing the size if it does flash render.\n",
       "        textArea.style.padding = 0;\n",
       "\n",
       "        // Clean up any borders.\n",
       "        textArea.style.border = 'none';\n",
       "        textArea.style.outline = 'none';\n",
       "        textArea.style.boxShadow = 'none';\n",
       "\n",
       "        // Avoid flash of white box if rendered for any reason.\n",
       "        textArea.style.background = 'transparent';\n",
       "\n",
       "\n",
       "        textArea.value = text;\n",
       "\n",
       "        document.body.appendChild(textArea);\n",
       "        textArea.focus();\n",
       "        textArea.select();\n",
       "\n",
       "        try {\n",
       "            if (!document.execCommand('copy')) {\n",
       "                console.log(\"Unable to copy text with document.execCommand()\");\n",
       "            }\n",
       "        } finally {\n",
       "            document.body.removeChild(textArea);\n",
       "        }\n",
       "    }\n",
       "\n",
       "    function copy_event_handler(event) {\n",
       "        copyTextToClipboard(event.target.dataset.hdf5Path);\n",
       "        event.preventDefault();\n",
       "    }\n",
       "\n",
       "    function enable_copylinks(parent) {\n",
       "        let links = parent.querySelectorAll(\".h5glance-dataset-copylink\");\n",
       "        links.forEach(function (link) {\n",
       "            link.addEventListener(\"click\", copy_event_handler);\n",
       "        });\n",
       "    }\n",
       "\n",
       "    // The code to actually trigger this is substituted below.\n",
       "    \n",
       "enable_copylinks(document.getElementById(\"h5glance-container-1\"));\n",
       "\n",
       "})();\n",
       "</script></div>"
      ],
      "text/plain": [
       "<h5glance.generate.H5Glance at 0x11bd34c88>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "H5Glance(\"new_file_bitshuffle_lz4.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 112616 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "h5file = h5py.File(\"new_file_bitshuffle_lz4.h5\", mode=\"r\")\n",
    "plt.imshow(h5file[\"/compressed_data_bitshuffle_lz4\"][()]); plt.colorbar()\n",
    "h5file.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r--  1 tvincent  staff  4278852 Jul  8 14:25 new_file_bitshuffle_lz4.h5\r\n",
      "-rw-r--r--  1 tvincent  staff  5832257 Jul  8 14:24 new_file_uncompressed.h5\r\n"
     ]
    }
   ],
   "source": [
    "!ls -l new_file*.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## HDF5 compression filters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Available through `h5py`\n",
    "\n",
    "[Compression filters provided by h5py](https://docs.h5py.org/en/stable/high/dataset.html#lossless-compression-filters):\n",
    "\n",
    "- Provided by `libhdf5`: **\"gzip\"** and eventually **\"szip\"** (optional)\n",
    "- Bundled with `h5py`: [**\"lzf\"**](https://confluence.hdfgroup.org/display/support/Filters#Filters-32000)\n",
    "\n",
    "Pre-compression filter: [Byte-Shuffle](https://docs.h5py.org/en/stable/high/dataset.html#shuffle-filter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "h5file = h5py.File(\"new_file_shuffle_gzip.h5\", mode=\"w\")\n",
    "h5file.create_dataset(\n",
    "    \"/compressed_data_shuffle_gzip\", data=data, shuffle=True, compression=\"gzip\")\n",
    "h5file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Provided by `hdf5plugin`\n",
    "\n",
    "Additional compression filters provided by `hdf5plugin`: Bitshuffle, Blosc, FciDecomp, LZ4, ZFP, Zstandard.\n",
    "\n",
    "6 out of the 25 [HDF5 registered filter plugins](https://confluence.hdfgroup.org/display/support/Registered+Filter+Plugins) as of June 2021."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "h5file = h5py.File(\"new_file_blosc.h5\", mode=\"w\")\n",
    "h5file.create_dataset(\n",
    "    \"/compressed_data_blosc\",\n",
    "    data=data,\n",
    "    **hdf5plugin.Blosc(cname='zlib', clevel=5, shuffle=hdf5plugin.Blosc.SHUFFLE)\n",
    ")\n",
    "h5file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### General purpose lossless compression\n",
    "\n",
    "- [Bitshuffle(nelems=0, lz4=True)](http://www.silx.org/doc/hdf5plugin/latest/usage.html#bitshuffle) (Filter ID [32008](https://confluence.hdfgroup.org/display/support/Filters#Filters-32008)): **Bit**-Shuffle + LZ4\n",
    "- [LZ4(nbytes=0)](http://www.silx.org/doc/hdf5plugin/latest/usage.html#lz4) (Filter ID [32004](https://confluence.hdfgroup.org/display/support/Filters#Filters-32004))\n",
    "- [Zstd()](http://www.silx.org/doc/hdf5plugin/latest/usage.html#zstd) (Filter ID [32015](https://confluence.hdfgroup.org/display/support/Filters#Filters-32015))\n",
    "- [Blosc(cname='lz4', clevel=5, shuffle=1)](http://www.silx.org/doc/hdf5plugin/latest/usage.html#blosc) (Filter ID [32001](https://confluence.hdfgroup.org/display/support/Filters#Filters-32001)):\n",
    "  Based on c-blosc: A blocking, shuffling and lossless compression library.\n",
    "  - Pre-compression shuffle: None, **Byte**-Shuffle, **Bit**-Shuffle\n",
    "  - Compression: `blosclz`, `lz4`, `lz4hc`, `snappy` (**optional**, requires C++11), `zlib`, `zstd`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Specific compression\n",
    "\n",
    "- [FciDecomp()](http://www.silx.org/doc/hdf5plugin/latest/usage.html#fcidecomp) (Filter ID [32018](https://confluence.hdfgroup.org/display/support/Filters#Filters-32018)): Based on JPEG-LS:\n",
    "  - **Optional**: requires C++11\n",
    "  - Data type: `(u)int8` or `(u)int16`\n",
    "  - Chunk shape: \"Image-like\"; 2 or 3 dimensions with at least 16 pixels and at most 65535 rows and columns and at most 4 planes for 3D datasets.\n",
    "- [ZFP(rate=None, precision=None, accuracy=None, reversible=False, minbits=None, maxbits=None, maxprec=None, minexp=None)](http://www.silx.org/doc/hdf5plugin/latest/usage.html#zfp) (Filter ID [32013](https://confluence.hdfgroup.org/display/support/Filters#Filters-32013)): **Lossy**\n",
    "  - Data type: `float32`, `float64`, `(u)int32`, `(u)int64`\n",
    "  - Chunk shape: must have at most 4 non-unity dimensions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Benchmark\n",
    "\n",
    "- Machine: 6 cores+hyperthreading (Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz)\n",
    "- Filesystem: RAM disk\n",
    "- HDF5 chunk: 1 frame\n",
    "- `hdf5plugin` built from source\n",
    "- Diffraction tomography dataset: 100 frames from http://www.silx.org/pub/pyFAI/pyFAI_UM_2020/data_ID13/kevlar.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![Benchmark](data:image/png;base64,[encoded data, 197420 characters] \"Benchmark\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Equivalent filters\n",
    "\n",
    "`Blosc` includes pre-compression filters and algorithms provided by other HDF5 compression filters:\n",
    "\n",
    "- `LZ4()` => `Blosc(\"lz4\", 9)`\n",
    "- `Zstd()` => `Blosc(\"zstd\", 2)`\n",
    "- HDF5 shuffle  => `Blosc` with `shuffle=hdf5plugin.Blosc.SHUFFLE`\n",
    "- `Bitshuffle()` => `Blosc(\"lz4\", 5, hdf5plugin.Blosc.BITSHUFFLE)`...\n",
    "  \n",
    "  **Except for  OpenMP support** with `Bitshuffle`!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Summary\n",
    "\n",
    "Having different pre-compression filters and compression algorithms at hand offer different read/write speed versus compression rate (and eventually error rate) trade-offs.\n",
    "\n",
    "Also to keep in mind availability/compatibility: `\"gzip\"` as included in `libHDF5` is the most compatible one (and also `\"lzf\"` as included in `h5py`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Using `hdf5plugin` filters with other applications\n",
    "\n",
    "Note: With notebook, using **!** enables running shell commands"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HDF5 \"data.h5\" {\r\n",
      "DATASET \"/compressed_data_bitshuffle_lz4\" {\r\n",
      "   DATATYPE  H5T_STD_U8LE\r\n",
      "   DATASPACE  SIMPLE { ( 1969, 2961 ) / ( 1969, 2961 ) }\r\n",
      "   SUBSET {\r\n",
      "      START ( 0, 0 );\r\n",
      "      STRIDE ( 1, 1 );\r\n",
      "      COUNT ( 5, 10 );\r\n",
      "      BLOCK ( 1, 1 );\r\n",
      "      DATA {\r\n",
      "      }\r\n",
      "   }\r\n",
      "}\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!h5dump -d /compressed_data_bitshuffle_lz4 -s \"0,0\" -c \"5,10\" data.h5 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "A solution: Set `HDF5_PLUGIN_PATH` environment variable to: `hdf5plugin.PLUGINS_PATH`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# Directory where HDF5 compression filters are stored\n",
    "hdf5plugin.PLUGINS_PATH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Retrieve hdf5plugin.PLUGINS_PATH from the command line\n",
    "!python3 -c \"import hdf5plugin; print(hdf5plugin.PLUGINS_PATH)\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31mlibh5blosc.dylib\u001b[m\u001b[m     \u001b[31mlibh5fcidecomp.dylib\u001b[m\u001b[m \u001b[31mlibh5zfp.dylib\u001b[m\u001b[m\r\n",
      "\u001b[31mlibh5bshuf.dylib\u001b[m\u001b[m     \u001b[31mlibh5lz4.dylib\u001b[m\u001b[m       \u001b[31mlibh5zstd.dylib\u001b[m\u001b[m\r\n"
     ]
    }
   ],
   "source": [
    "!ls `python3 -c \"import hdf5plugin; print(hdf5plugin.PLUGINS_PATH)\"`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HDF5 \"data.h5\" {\r\n",
      "DATASET \"/compressed_data_bitshuffle_lz4\" {\r\n",
      "   DATATYPE  H5T_STD_U8LE\r\n",
      "   DATASPACE  SIMPLE { ( 1969, 2961 ) / ( 1969, 2961 ) }\r\n",
      "   SUBSET {\r\n",
      "      START ( 0, 0 );\r\n",
      "      STRIDE ( 1, 1 );\r\n",
      "      COUNT ( 5, 10 );\r\n",
      "      BLOCK ( 1, 1 );\r\n",
      "      DATA {\r\n",
      "      (0,0): 53, 52, 53, 54, 54, 55, 55, 56, 56, 57,\r\n",
      "      (1,0): 49, 50, 54, 55, 53, 54, 55, 56, 56, 58,\r\n",
      "      (2,0): 50, 51, 54, 54, 53, 55, 56, 57, 58, 57,\r\n",
      "      (3,0): 51, 54, 55, 54, 54, 55, 56, 57, 58, 59,\r\n",
      "      (4,0): 53, 55, 54, 54, 56, 56, 58, 57, 57, 58\r\n",
      "      }\r\n",
      "   }\r\n",
      "}\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "# Set HDF5_PLUGIN_PATH environment variable to hdf5plugin.PLUGINS_PATH\n",
    "!HDF5_PLUGIN_PATH=`python3 -c \"import hdf5plugin; print(hdf5plugin.PLUGINS_PATH)\"` h5dump -d /compressed_data_bitshuffle_lz4 -s \"0,0\" -c \"5,10\" data.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "**Note**: Only works for reading compressed datasets, **not for writing!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Insights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### The Problem\n",
    "\n",
    "For **reading** compressed datasets, compression filters do **NOT** need information from `libHDF5`. They work with the compressed stream.\n",
    "\n",
    "For **writing** compressed datasets, some information about the dataset (e.g., data type size) can be needed by the filter (e.g., to shuffle the data). This information is retrieve through `libHDF5` C-API (e.g., [**H5Tget_size**](https://support.hdfgroup.org/HDF5/doc/RM/RM_H5T.html#Datatype-GetSize)).\n",
    "\n",
    "Access to `libHDF5` C-API is needed, but linking compression filters with `libHDF5` is cumbersome in a dynamic environment like *Python*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### On Windows\n",
    "\n",
    "Symbols from dynamically loaded *Python* modules and libraries are accessible to others.\n",
    "\n",
    "Register compression filter at C-level with `H5Zregister` (see [src/register_win32.c](https://github.com/silx-kit/hdf5plugin/blob/main/src/register_win32.c))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### On Linux, macos\n",
    "\n",
    "In *Python*, symbols from dynamically loaded modules and libraries are **NOT** visible to others.\n",
    "\n",
    "- Do not link filters with `libHDF5`.\n",
    "- Instead, provide some function wrappers to replace `libHDF5` C-API and link the compression filter with those.\n",
    "\n",
    "  - Those functions call `libHDF5` corresponding functions that are dynamically loaded at runtime.\n",
    "  \n",
    "- At runtime, we need to initialize the compression filter to load symbols dynamically from `libHDF5` used by `h5py` and use them from the function wrappers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "[src/hdf5_dl.c](https://github.com/silx-kit/hdf5plugin/blob/main/src/hdf5_dl.c):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "```c\n",
    "typedef size_t (* DL_func_H5Tget_size)(hid_t type_id);\n",
    "static struct { /* Structure storing HDF5 function pointers */\n",
    "    DL_func_H5Tget_size H5Tget_size;\n",
    "} DL_H5Functions = {NULL};\n",
    "\n",
    "/* Init wrapper by loading symbols from `libHDF5` */\n",
    "int init_filter(const char* libname) {\n",
    "    void * handle = dlopen(libname, RTLD_LAZY | RTLD_LOCAL);  /*Load libHDF5*/\n",
    "    DL_H5Functions.H5Tget_size = (DL_func_H5Tget_size)dlsym(handle, \"H5Tget_size\");\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "```c\n",
    "/* H5Tget_size libHDF5 C-API wrapper*/\n",
    "size_t H5Tget_size(hid_t type_id) {\n",
    "    if(DL_H5Functions.H5Tget_size != NULL) {\n",
    "        return DL_H5Functions.H5Tget_size(type_id);\n",
    "    } else {\n",
    "        return 0;\n",
    "    }\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Concluding remark\n",
    "\n",
    "In the event the HDF5 compression filter API evolves, it would be great to take this into account to ease distribution of compression filters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## A word about `hdf5plugin` license\n",
    "\n",
    "The source code of `hdf5plugin` itself is licensed under the [MIT](https://github.com/silx-kit/hdf5plugin/blob/main/LICENSE) license...\n",
    "\n",
    "It also embeds the source code of the provided compression filters and libraries which are licensed under [different open-source licenses](http://www.silx.org/doc/hdf5plugin/latest/information.html#license) (Apache, BSD-2, BSD-3, MIT, Zlib...) and copyrights."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Conlusion\n",
    "\n",
    "`hdf5plugin` provides additional HDF5 compression filters (namely: `Bitshuffle`, `Blosc`, `FciDecomp`, `LZ4`, `ZFP`, `Zstandard`) mainly for use with [h5py](https://docs.h5py.org/) but not only.\n",
    "\n",
    "- Packaged for [**pip**](https://pypi.org/project/hdf5plugin/) and [**conda**](https://anaconda.org/conda-forge/hdf5plugin)\n",
    "- Documentation: http://www.silx.org/doc/hdf5plugin/latest/\n",
    "- Source code repository: https://github.com/silx-kit/hdf5plugin\n",
    "\n",
    "**Credits to the** [**contributors**](https://github.com/silx-kit/hdf5plugin/graphs/contributors): [Thomas Vincent](https://github.com/t20100), [Armando Sole](https://github.com/vasole), [\\@Florian-toll](https://github.com/Florian-toll), [\\@fpwg](https://github.com/fpwg), [Jerome Kieffer](https://github.com/kif), [\\@Anthchirp](https://github.com/Anthchirp), [\\@mobiusklein](https://github.com/mobiusklein), [\\@junyuewang](https://github.com/junyuewang)\n",
    "\n",
    "Partially funded by the [PaNOSC](https://www.panosc.eu/) EU-project.\n",
    "\n",
    "<img align=\"left\" vertical-align=\"top\" src=\"data:image/svg+xml,%3Csvg%20width%3D%2240.5%22%20height%3D%2227%22%20version%3D%221.1%22%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20xmlns%3Axlink%3D%22http%3A%2F%2Fwww.w3.org%2F1999%2Fxlink%22%3E%3Cdesc%3EEuropean%20flag%3C%2Fdesc%3E%3Cdefs%3E%3Cg%20id%3D%22s%22%3E%3Cg%20id%3D%22c%22%3E%3Cpath%20id%3D%22t%22%20transform%3D%22rotate%2818%203.1569%20-.5%29%22%20d%3D%22m0%200v1h0.5z%22%2F%3E%3Cuse%20transform%3D%22scale%28-1%2C1%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23t%22%2F%3E%3C%2Fg%3E%3Cg%20id%3D%22a%22%3E%3Cuse%20transform%3D%22rotate%2872%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23c%22%2F%3E%3Cuse%20transform%3D%22rotate%28144%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23c%22%2F%3E%3C%2Fg%3E%3Cuse%20transform%3D%22scale%28-1%2C1%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23a%22%2F%3E%3C%2Fg%3E%3C%2Fdefs%3E%3Crect%20width%3D%2240.5%22%20height%3D%2227%22%20fill%3D%22%23039%22%20stroke-width%3D%22.05%22%2F%3E%3Cg%20transform%3D%22matrix%281.5%2C0%2C0%2C1.5%2C20.25%2C13.5%29%22%20fill%3D%22%23fc0%22%3E%3Cuse%20y%3D%22-6%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23s%22%2F%3E%3Cuse%20y%3D%226%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23s%22%2F%3E%3Cg%20id%3D%22l%22%3E%3Cuse%20x%3D%22-6%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23s%22%2F%3E%3Cuse%20transform%3D%22rotate%28216%20-2.3442%20-2.1107%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23s%22%2F%3E%3Cuse%20transform%3D%22rotate%28144%2C-2.1107%2C-2.3442%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23s%22%2F%3E%3Cuse%20transform%3D%22rotate%2872%2C-4.6626%2C-2.0759%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23s%22%2F%3E%3Cuse%20transform%3D%22rotate%2872%20-5.0759%20.5335%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23s%22%2F%3E%3C%2Fg%3E%3Cuse%20transform%3D%22scale%28-1%2C1%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23l%22%2F%3E%3C%2Fg%3E%3C%2Fsvg%3E\"/>&nbsp;This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 823852."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbsphinx": "hidden",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Thanks for your attention! Questions?"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Diaporama",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "nbsphinx": {
   "orphan": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
