{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Abstract\n",
    "\n",
    "[**hdf5plugin**](https://github.com/silx-kit/hdf5plugin) is a *Python* package (1) providing a set of [**HDF5**](https://portal.hdfgroup.org/display/HDF5/) compression filters (namely: Blosc, Blosc2, BitShuffle, BZip2, FciDecomp, LZ4, SZ, SZ3, ZFP, ZStandard) and (2) enabling their use from the *Python* programming language with [**h5py**](https://docs.h5py.org/) a thin, pythonic wrapper around `libHDF5`.\n",
    "\n",
    "This presentation illustrates how to use **hdf5plugin** for reading and writing compressed datasets from *Python* and gives an overview of the different HDF5 compression filters it provides.\n",
    "\n",
    "License: [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbsphinx": "hidden",
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# Notebook requirements\n",
    "# A recent version of Pillow is required!\n",
    "#%pip install numpy h5py hdf5plugin h5glance rise jupyterlab matplotlib ipympl Pillow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbsphinx": "hidden",
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "# Creates data.h5 used for demos\n",
    "from matplotlib import pyplot as plt\n",
    "import h5py\n",
    "import hdf5plugin\n",
    "import numpy\n",
    "from PIL import Image\n",
    "import urllib.request\n",
    "\n",
    "url = \"https://www.desy.de/e409/e116959/e119238/media/7795/1963_Luftbild_DESY-Gelaende_1284_gf_01sw_a3.jpg\"\n",
    "filename = urllib.request.urlretrieve(url)[0]\n",
    "image = numpy.array(Image.open(filename).convert(\"L\"))\n",
    "plt.imshow(image, cmap=\"gray\")\n",
    "\n",
    "h5file = h5py.File(\"data.h5\", mode=\"w\")\n",
    "h5file[\"copyright\"] = \"DESY\"\n",
    "h5file.attrs[\"url\"] = url\n",
    "h5file.create_dataset(\"/data\", data=image)\n",
    "h5file.create_dataset(\n",
    "    \"/compressed_data\",\n",
    "    data=image,\n",
    "    chunks=image.shape,\n",
    "    **hdf5plugin.Blosc2('lz4', filters=hdf5plugin.Blosc.BITSHUFFLE)\n",
    ")\n",
    "h5file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "**Restart kernel once the file is created!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os._exit(0)  # Makes the kernel restart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbsphinx": "hidden",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![ESRF logo](data:image/png;base64,[encoded data, 462192 characters] \"ESRF logo\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&nbsp;\n",
    "\n",
    "<table align=\"right\" vertical-align=\"bottom\" width=\"15%\"><tr>\n",
    "    <td><img src=\"data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iMjE0cHgiIGhlaWdodD0iODBweCIgZW5hYmxlLWJhY2tncm91bmQ9Im5ldyAwIDAgMjE0IDgwIiB2ZXJzaW9uPSIxLjEiIHZpZXdCb3g9IjAgMCAyMTQgODAiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+PHN0eWxlIHR5cGU9InRleHQvY3NzIj4uc3Qwe2ZpbGw6IzAwMDAwMDt9Cgkuc3Qxe2ZpbGw6IzgxQjdFNDt9Cgkuc3Qye2ZpbGw6IzY1NkRCMTt9Cgkuc3Qze2ZpbGw6I0JBNDU3Njt9PC9zdHlsZT48Zz48cGF0aCBjbGFzcz0ic3QwIiBkPSJtLTAuMSA3MS43di0yNS41YzAtMi4zIDAuNC0zLjMgNS4yLTQgMi43LTAuNCA3LTAuNyAxMS41LTAuNyAxNC40IDAgMTcuNiAzLjggMTcuNiAxMi4ycy0yLjcgMTIuMi0xNi45IDEyLjJjLTIuNyAwLTQuOC0wLjItNi41LTAuMy0wLjgtMC4xLTEuNCAwLjEtMS40IDEuMnY0LjhjMCAyLjgtMC42IDMuMy00LjIgMy4zaC0xYy0zLjcgMC4yLTQuMy0wLjQtNC4zLTMuMm0yNC4zLTE3LjljMC03LjQtMi04LjktNy43LTguOS0yLjkgMC01LjQgMC40LTUuOSAwLjUtMC42IDAuMS0xLjIgMC4zLTEuMiAxLjN2MTQuM2MwIDEgMC42IDEuMiAxLjIgMS40IDAuNSAwLjEgMy4xIDAuNCA1LjkgMC40IDUuNy0wLjEgNy43LTEuNiA3LjctOSIvPjxwYXRoIGNsYXNzPSJzdDAiIGQ9Im0zNyA1OS41YzAtNS4yIDUuNS01LjYgMTMtNS42aDUuOGMwLjYgMCAxLjItMC4xIDEuMi0xLjF2LTAuOGMwLTMuMi0yLjItMy4zLTYuNi0zLjMtNC42IDAtOC43IDAuNi05LjcgMC43LTEuMSAwLjEtMS41LTAuMS0xLjUtMS41IDAtMS4zIDAuMi0xLjYgMS41LTEuOCAwLjgtMC4yIDUuOS0wLjcgMTEuNy0wLjcgNC40IDAgNy43IDAuNCA5LjggMS40IDIuMiAxIDIuOCAyLjcgMi44IDV2MTAuMmMwIDEuOS0wLjQgMi44LTQuNSAzLjQtMi41IDAuMy02LjkgMC42LTEwLjMgMC42LTExLjUgMC0xMy4yLTItMTMuMi02LjVtMTguNiAzLjNjMC44LTAuMiAxLjQtMC40IDEuNC0xLjZ2LTMuM2MwLTAuOS0wLjctMS4xLTEuMi0xLjFoLTUuM2MtMy42IDAtNS4yIDAuMy01LjIgMi44IDAgMyAxLjIgMy41IDUuNiAzLjUgMiAwLjEgNC0wLjEgNC43LTAuMyIvPjxwYXRoIGNsYXNzPSJzdDAiIGQ9Im04MC41IDYyLjJjMCAyLjgtMC42IDMuMy00LjIgMy4zaC0xYy0zLjggMC00LjMtMC42LTQuMy0zLjN2LTE1LjljMC0yLjMgMC40LTMuMSA1LjQtMy45IDMuMi0wLjUgOC40LTAuOSAxMy0wLjkgMTAuNiAwIDE0LjcgMi42IDE0LjcgOXYxMS43YzAgMi44LTAuNiAzLjMtNC4yIDMuM2gtMWMtMy45IDAtNC40LTAuNi00LjQtMy4zdi0xMS44YzAtNC42LTIuNC01LjQtNi43LTUuNC0yLjQgMC00LjMgMC4yLTUuOSAwLjYtMC45IDAuMi0xLjQgMC42LTEuNCAxLjd2MTQuOXoiLz48cGF0aCBjbGFzcz0ic3QwIiBkPSJtMTA4LjcgNTMuOGMwLTYuOCAxLjYtMTIuMyAxNy44LTEyLjNzMTcuOCA1LjUgMTcuOCAxMi4zLTEuNyAxMi4yLTE3LjggMTIuMmMtMTYuMiAwLjEtMTcuOC01LjQtMTcuOC0xMi4ybTI1LjcgMGMwLTUuOC0wLjYtOS03LjgtOS03LjMgMC03LjggMy4yLTcuOCA5czAuNiA5IDcuOCA5IDcuOC0zLjEgNy44LTkiLz48cGF0aCBjbGFzcz0ic3QwIiBkPSJtMTQ5LjkgNjUuMWMtMS41LTAuMy0xLjctMC41LTEuNy0yLjEgMC0xLjcgMC41LTIgMS44LTEuOCAxLjUgMC4yIDcuMiAxLjEgMTMuOCAxLjEgNS4zIDAgNy0wLjQgNy0yLjQgMC0yLjItMC44LTIuNS03LjgtMi45LTExLjMtMC41LTE1LTEuNC0xNS03LjcgMC03LjcgNi4yLTcuOCAxNS44LTcuOCA2LjMgMCAxMS4xIDAuNyAxMi4xIDAuOSAxLjUgMC4yIDEuNyAwLjQgMS43IDIgMCAxLjctMC41IDItMS44IDEuOC0xLjEtMC4xLTUuOS0wLjgtMTEuNS0wLjhzLTcuMyAwLjEtNy4zIDIuMmMwIDIgMC42IDIuNCA3LjEgMi42IDExLjUgMC41IDE1LjQgMSAxNS40IDggMCA3LjUtNy43IDcuOS0xNi4zIDcuOS03IDAtMTIuMi0wLjgtMTMuMy0xIi8+PHBhdGggY2xhc3M9InN0MCIgZD0ibTE4MyA1My43YzAtOCAzLjYtMTIuMyAxOC0xMi4yIDYuNyAwLjEgMTAuNSAwLjcgMTEuNCAwLjggMS41IDAuMyAxLjggMC41IDEuOCAyLjEgMCAxLjctMC41IDEuOC0xLjggMS43LTEuNS0wLjItNS44LTAuNy05LjMtMC43LTguMiAwLTEwIDIuMi0xMCA4LjNzMS44IDguMyAxMCA4LjNjMy42IDAgNy44LTAuNSA5LjMtMC43IDEuNC0wLjIgMS44IDAgMS44IDEuNyAwIDEuNi0wLjIgMS44LTEuOCAyLjEtMC45IDAuMi00LjcgMC44LTExLjQgMC44LTE0LjQgMC4yLTE4LTQuMS0xOC0xMi4yIi8+PC9nPjxwYXRoIGNsYXNzPSJzdDEiIGQ9Im0xNTEuMyAxNy40cy0xOC42LTM5LjEtNDguNSAwLTQ4LjUgMC00OC41IDBoLTkuNHMxOC42IDM5LjEgNDguNSAwIDQ4LjUgMCA0OC41IDBoOS40eiIvPjxwYXRoIGNsYXNzPSJzdDIiIGQ9Im0xMzYuNCAxNy40cy0xOC42LTM5LjEtNDguNSAwLTQ4LjUgMC00OC41IDBoLTkuNHMxOC42IDM5LjEgNDguNSAwIDQ4LjUgMCA0OC41IDBoOS40eiIvPjxwYXRoIGNsYXNzPSJzdDMiIGQ9Im0xMjAuNSAxNy40cy0xOC42LTM5LjEtNDguNSAwLTQ4LjUgMC00OC41IDBoLTkuNHMxOC42IDM5LjEgNDguNSAwIDQ4LjUgMCA0OC41IDBoOS40eiIvPjwvc3ZnPg==\" /></td>\n",
    "    <td><img src=\"data:image/png;base64,[encoded data, 30592 characters]\" /></td>\n",
    "</tr></table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# hdf5plugin\n",
    "\n",
    "[**hdf5plugin**](https://github.com/silx-kit/hdf5plugin) packages a set of [HDF5](https://portal.hdfgroup.org/display/HDF5/) compression filters and makes them usable from the Python programming language through [h5py](https://docs.h5py.org/).\n",
    "\n",
    "[**h5py**](https://docs.h5py.org/) is a thin, pythonic wrapper around [HDF5](https://portal.hdfgroup.org/display/HDF5/).\n",
    "\n",
    "&nbsp;\n",
    "\n",
    "&nbsp;\n",
    "\n",
    "Presenter: Thomas VINCENT\n",
    "\n",
    "European HDF5 User Group Meeting 2023, September 19, 2023"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>/* Based on https://css-tricks.com/snippets/css/nested-expandable-folders/\n",
       "   Reused with permission from https://css-tricks.com/license/\n",
       "*/\n",
       "\n",
       ".h5glance-css-treeview ul,\n",
       ".h5glance-css-treeview li\n",
       "{\n",
       "\tpadding: 0;\n",
       "\tmargin: 0;\n",
       "    margin-left: 12px;\n",
       "\tlist-style: none;\n",
       "}\n",
       "\n",
       "/* Override padding from Jupyter CSS */\n",
       ".rendered_html .h5glance-css-treeview ul {\n",
       "    padding: 0;\n",
       "}\n",
       "\n",
       "/* Make the checkbox itself invisible */\n",
       ".h5glance-css-treeview input\n",
       "{\n",
       "\tposition: absolute;\n",
       "\topacity: 0;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview\n",
       "{\n",
       "    font-family: sans-serif;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview code {\n",
       "    font-size: 12pt;\n",
       "    font-family: \"courier\", monospace;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview a.h5glance-dataset-copylink\n",
       "{\n",
       "    text-decoration: none;\n",
       "}\n",
       "\n",
       "/* These next two bits drive the expanding/collapsing behaviour */\n",
       ".h5glance-css-treeview input + label + ul\n",
       "{\n",
       "\tdisplay: none;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview input:checked:not(:disabled) + label + ul\n",
       "{\n",
       "\tdisplay: block;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview label,\n",
       ".h5glance-css-treeview label::before\n",
       "{\n",
       "\tdisplay: inline-block;\n",
       "\theight: 16px;\n",
       "\tline-height: 16px;,\n",
       "\tvertical-align: middle;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview label\n",
       "{\n",
       "    margin-left: -1.5em;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview label::before\n",
       "{\n",
       "\tcontent: \"\\229e\"; /* squared plus */\n",
       "\twidth: 1.5em;\n",
       "\tvertical-align: middle;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview input:checked + label::before\n",
       "{\n",
       "    content: \"\\229f\";  /* squared minus */\n",
       "}\n",
       "\n",
       "/* webkit adjacent element selector bugfix */\n",
       "@media screen and (-webkit-min-device-pixel-ratio:0)\n",
       "{\n",
       "\t.h5glance-css-treeview\n",
       "\t{\n",
       "\t\t-webkit-animation: webkit-adjacent-element-selector-bugfix infinite 1s;\n",
       "\t}\n",
       "\n",
       "\t@-webkit-keyframes webkit-adjacent-element-selector-bugfix\n",
       "\t{\n",
       "\t\tfrom\n",
       "\t\t{\n",
       "\t\t\tpadding: 0;\n",
       "\t\t}\n",
       "\t\tto\n",
       "\t\t{\n",
       "\t\t\tpadding: 0;\n",
       "\t\t}\n",
       "\t}\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview li.h5glance-dataset {\n",
       "    margin-bottom: 3px;\n",
       "}\n",
       "\n",
       ".h5glance-dataset-name {\n",
       "    font-weight: bold;\n",
       "}\n",
       "</style><div id=\"h5glance-container-0\" class=\"h5glance-css-treeview\"><ul><li><input checked=\"checked\" id=\"h5glance-expand-switch-0\" type=\"checkbox\"/><label for=\"h5glance-expand-switch-0\">data.h5</label><ul><li class=\"h5glance-dataset\"><span class=\"h5glance-dataset-name\">compressed_data</span> <a data-hdf5-path=\"/compressed_data\" href=\"#\" class=\"h5glance-dataset-copylink\">[📋]</a>: 3744 × 5286 entries, dtype: <abbr title=\"8-bit unsigned integer\">uint8</abbr></li><li class=\"h5glance-dataset\"><span class=\"h5glance-dataset-name\">copyright</span> <a data-hdf5-path=\"/copyright\" href=\"#\" class=\"h5glance-dataset-copylink\">[📋]</a>: scalar entries, dtype: UTF-8 string</li><li class=\"h5glance-dataset\"><span class=\"h5glance-dataset-name\">data</span> <a data-hdf5-path=\"/data\" href=\"#\" class=\"h5glance-dataset-copylink\">[📋]</a>: 3744 × 5286 entries, dtype: <abbr title=\"8-bit unsigned integer\">uint8</abbr></li></ul></li></ul></div><script>// Based on code by Stackoverflow user Dean Taylor\n",
       "// https://stackoverflow.com/questions/400212/how-do-i-copy-to-the-clipboard-in-javascript/30810322\n",
       "// Used under Stackoverflow's CC-BY-SA 3.0 license\n",
       "\n",
       "(function() {\n",
       "    function copyTextToClipboard(text) {\n",
       "        let textArea = document.createElement(\"textarea\");\n",
       "\n",
       "        //\n",
       "        // *** This styling is an extra step which is likely not required. ***\n",
       "        //\n",
       "        // Why is it here? To ensure:\n",
       "        // 1. the element is able to have focus and selection.\n",
       "        // 2. if element was to flash render it has minimal visual impact.\n",
       "        // 3. less flakyness with selection and copying which **might** occur if\n",
       "        //    the textarea element is not visible.\n",
       "        //\n",
       "        // The likelihood is the element won't even render, not even a flash,\n",
       "        // so some of these are just precautions. However in IE the element\n",
       "        // is visible whilst the popup box asking the user for permission for\n",
       "        // the web page to copy to the clipboard.\n",
       "        //\n",
       "\n",
       "        // Place in top-left corner of screen regardless of scroll position.\n",
       "        textArea.style.position = 'fixed';\n",
       "        textArea.style.top = 0;\n",
       "        textArea.style.left = 0;\n",
       "\n",
       "        // Ensure it has a small width and height. Setting to 1px / 1em\n",
       "        // doesn't work as this gives a negative w/h on some browsers.\n",
       "        textArea.style.width = '2em';\n",
       "        textArea.style.height = '2em';\n",
       "\n",
       "        // We don't need padding, reducing the size if it does flash render.\n",
       "        textArea.style.padding = 0;\n",
       "\n",
       "        // Clean up any borders.\n",
       "        textArea.style.border = 'none';\n",
       "        textArea.style.outline = 'none';\n",
       "        textArea.style.boxShadow = 'none';\n",
       "\n",
       "        // Avoid flash of white box if rendered for any reason.\n",
       "        textArea.style.background = 'transparent';\n",
       "\n",
       "\n",
       "        textArea.value = text;\n",
       "\n",
       "        document.body.appendChild(textArea);\n",
       "        textArea.focus();\n",
       "        textArea.select();\n",
       "\n",
       "        try {\n",
       "            if (!document.execCommand('copy')) {\n",
       "                console.log(\"Unable to copy text with document.execCommand()\");\n",
       "            }\n",
       "        } finally {\n",
       "            document.body.removeChild(textArea);\n",
       "        }\n",
       "    }\n",
       "\n",
       "    function copy_event_handler(event) {\n",
       "        copyTextToClipboard(event.target.dataset.hdf5Path);\n",
       "        event.preventDefault();\n",
       "    }\n",
       "\n",
       "    function enable_copylinks(parent) {\n",
       "        let links = parent.querySelectorAll(\".h5glance-dataset-copylink\");\n",
       "        links.forEach(function (link) {\n",
       "            link.addEventListener(\"click\", copy_event_handler);\n",
       "        });\n",
       "    }\n",
       "\n",
       "    // The code to actually trigger this is substituted below.\n",
       "    \n",
       "enable_copylinks(document.getElementById(\"h5glance-container-0\"));\n",
       "\n",
       "})();\n",
       "</script></div>"
      ],
      "text/plain": [
       "data.h5/ (1 attributes)\n",
       "├compressed_data\t[uint8: 3744 × 5286]\n",
       "├copyright\t[UTF-8 string: scalar]\n",
       "└data\t[uint8: 3744 × 5286]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from h5glance import H5Glance  # Browsing HDF5 files\n",
    "H5Glance(\"data.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import h5py  # Pythonic HDF5 wrapper: https://docs.h5py.org/\n",
    "\n",
    "h5file = h5py.File(\"data.h5\", mode=\"r\")  # Open HDF5 file in read mode\n",
    "data = h5file[\"/data\"][()]               # Access HDF5 dataset \"/data\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f0896e60760>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 135580 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "plt.imshow(data, cmap=\"gray\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "ename": "OSError",
     "evalue": "Can't read data (can't open directory: /usr/local/hdf5/lib/plugin)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mOSError\u001b[0m                                   Traceback (most recent call last)",
      "Input \u001b[0;32mIn [4]\u001b[0m, in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0m data \u001b[38;5;241m=\u001b[39m \u001b[43mh5file\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43m/compressed_data\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m]\u001b[49m\u001b[43m[\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m]\u001b[49m\n",
      "File \u001b[0;32mh5py/_objects.pyx:54\u001b[0m, in \u001b[0;36mh5py._objects.with_phil.wrapper\u001b[0;34m()\u001b[0m\n",
      "File \u001b[0;32mh5py/_objects.pyx:55\u001b[0m, in \u001b[0;36mh5py._objects.with_phil.wrapper\u001b[0;34m()\u001b[0m\n",
      "File \u001b[0;32m~/venvs/py310/lib/python3.10/site-packages/h5py/_hl/dataset.py:758\u001b[0m, in \u001b[0;36mDataset.__getitem__\u001b[0;34m(self, args, new_dtype)\u001b[0m\n\u001b[1;32m    756\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_fast_read_ok \u001b[38;5;129;01mand\u001b[39;00m (new_dtype \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m):\n\u001b[1;32m    757\u001b[0m     \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 758\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_fast_reader\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mread\u001b[49m\u001b[43m(\u001b[49m\u001b[43margs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    759\u001b[0m     \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mTypeError\u001b[39;00m:\n\u001b[1;32m    760\u001b[0m         \u001b[38;5;28;01mpass\u001b[39;00m  \u001b[38;5;66;03m# Fall back to Python read pathway below\u001b[39;00m\n",
      "File \u001b[0;32mh5py/_selector.pyx:376\u001b[0m, in \u001b[0;36mh5py._selector.Reader.read\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mOSError\u001b[0m: Can't read data (can't open directory: /usr/local/hdf5/lib/plugin)"
     ]
    }
   ],
   "source": [
    "data = h5file[\"/compressed_data\"][()]  # Access compressed dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# Check dataset's filters\n",
    "plist = h5file[\"/compressed_data\"].id.get_create_plist()\n",
    "plist.get_filter(0)[0::3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `hdf5plugin` usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Reading compressed datasets\n",
    "\n",
    "To enable reading compressed datasets not supported by `libHDF5` and `h5py`:\n",
    "Install **hdf5plugin** & import it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "pip3 install hdf5plugin\n",
    "# Or:\n",
    "conda install -c conda-forge hdf5plugin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Or on [Debian12](https://packages.debian.org/bookworm/python3-hdf5plugin) and [Ubuntu23.04](https://packages.ubuntu.com/lunar/python3-hdf5plugin):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "apt-get install python3-hdf5plugin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import hdf5plugin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f089182d480>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 135580 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = h5file[\"/compressed_data\"][()]  # Access datset\n",
    "plt.imshow(data, cmap=\"gray\")          # Display data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "h5file.close()  # Close the HDF5 file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Writing compressed datasets\n",
    "\n",
    "When writing datasets with `h5py`, compression can be specified with: [h5py.Group.create_dataset](https://docs.h5py.org/en/stable/high/group.html?highlight=create_dataset#h5py.Group.create_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Create a dataset with h5py without compression\n",
    "h5file = h5py.File(\"new_file_uncompressed.h5\", mode=\"w\")\n",
    "h5file.create_dataset(\"/data\", data=data)\n",
    "h5file.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Create a compressed dataset\n",
    "h5file = h5py.File(\"new_file_blosc2_bitshuffle_lz4.h5\", mode=\"w\")\n",
    "h5file.create_dataset(\n",
    "    \"/compressed_data\",\n",
    "    data=data,\n",
    "    compression=32026,  # Blosc2 HDF5 filter identifier\n",
    "    # options: 0, 0, 0, 0, level, filter, compression\n",
    "    compression_opts=(0, 0, 0, 0, 5, 2, 1)  \n",
    ")\n",
    "h5file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "`hdf5plugin` provides some [helpers](http://www.silx.org/doc/hdf5plugin/latest/usage.html#write-compressed-datasets) to ease dealing with compression filter and options:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "h5file = h5py.File(\"new_file_blosc2_bitshuffle_lz4.h5\", mode=\"w\")\n",
    "h5file.create_dataset(\n",
    "    \"/compressed_data\",\n",
    "    data=data,\n",
    "    compression=hdf5plugin.Blosc2(\n",
    "        cname='lz4',\n",
    "        clevel=5,\n",
    "        filters=hdf5plugin.Blosc2.BITSHUFFLE),\n",
    ")\n",
    "h5file.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "help(hdf5plugin.Blosc2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>/* Based on https://css-tricks.com/snippets/css/nested-expandable-folders/\n",
       "   Reused with permission from https://css-tricks.com/license/\n",
       "*/\n",
       "\n",
       ".h5glance-css-treeview ul,\n",
       ".h5glance-css-treeview li\n",
       "{\n",
       "\tpadding: 0;\n",
       "\tmargin: 0;\n",
       "    margin-left: 12px;\n",
       "\tlist-style: none;\n",
       "}\n",
       "\n",
       "/* Override padding from Jupyter CSS */\n",
       ".rendered_html .h5glance-css-treeview ul {\n",
       "    padding: 0;\n",
       "}\n",
       "\n",
       "/* Make the checkbox itself invisible */\n",
       ".h5glance-css-treeview input\n",
       "{\n",
       "\tposition: absolute;\n",
       "\topacity: 0;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview\n",
       "{\n",
       "    font-family: sans-serif;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview code {\n",
       "    font-size: 12pt;\n",
       "    font-family: \"courier\", monospace;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview a.h5glance-dataset-copylink\n",
       "{\n",
       "    text-decoration: none;\n",
       "}\n",
       "\n",
       "/* These next two bits drive the expanding/collapsing behaviour */\n",
       ".h5glance-css-treeview input + label + ul\n",
       "{\n",
       "\tdisplay: none;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview input:checked:not(:disabled) + label + ul\n",
       "{\n",
       "\tdisplay: block;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview label,\n",
       ".h5glance-css-treeview label::before\n",
       "{\n",
       "\tdisplay: inline-block;\n",
       "\theight: 16px;\n",
       "\tline-height: 16px;,\n",
       "\tvertical-align: middle;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview label\n",
       "{\n",
       "    margin-left: -1.5em;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview label::before\n",
       "{\n",
       "\tcontent: \"\\229e\"; /* squared plus */\n",
       "\twidth: 1.5em;\n",
       "\tvertical-align: middle;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview input:checked + label::before\n",
       "{\n",
       "    content: \"\\229f\";  /* squared minus */\n",
       "}\n",
       "\n",
       "/* webkit adjacent element selector bugfix */\n",
       "@media screen and (-webkit-min-device-pixel-ratio:0)\n",
       "{\n",
       "\t.h5glance-css-treeview\n",
       "\t{\n",
       "\t\t-webkit-animation: webkit-adjacent-element-selector-bugfix infinite 1s;\n",
       "\t}\n",
       "\n",
       "\t@-webkit-keyframes webkit-adjacent-element-selector-bugfix\n",
       "\t{\n",
       "\t\tfrom\n",
       "\t\t{\n",
       "\t\t\tpadding: 0;\n",
       "\t\t}\n",
       "\t\tto\n",
       "\t\t{\n",
       "\t\t\tpadding: 0;\n",
       "\t\t}\n",
       "\t}\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview li.h5glance-dataset {\n",
       "    margin-bottom: 3px;\n",
       "}\n",
       "\n",
       ".h5glance-dataset-name {\n",
       "    font-weight: bold;\n",
       "}\n",
       "</style><div id=\"h5glance-container-1\" class=\"h5glance-css-treeview\"><ul><li><input checked=\"checked\" id=\"h5glance-expand-switch-1\" type=\"checkbox\"/><label for=\"h5glance-expand-switch-1\">new_file_blosc2_bitshuffle_lz4.h5</label><ul><li class=\"h5glance-dataset\"><span class=\"h5glance-dataset-name\">compressed_data</span> <a data-hdf5-path=\"/compressed_data\" href=\"#\" class=\"h5glance-dataset-copylink\">[📋]</a>: 3744 × 5286 entries, dtype: <abbr title=\"8-bit unsigned integer\">uint8</abbr></li></ul></li></ul></div><script>// Based on code by Stackoverflow user Dean Taylor\n",
       "// https://stackoverflow.com/questions/400212/how-do-i-copy-to-the-clipboard-in-javascript/30810322\n",
       "// Used under Stackoverflow's CC-BY-SA 3.0 license\n",
       "\n",
       "(function() {\n",
       "    function copyTextToClipboard(text) {\n",
       "        let textArea = document.createElement(\"textarea\");\n",
       "\n",
       "        //\n",
       "        // *** This styling is an extra step which is likely not required. ***\n",
       "        //\n",
       "        // Why is it here? To ensure:\n",
       "        // 1. the element is able to have focus and selection.\n",
       "        // 2. if element was to flash render it has minimal visual impact.\n",
       "        // 3. less flakyness with selection and copying which **might** occur if\n",
       "        //    the textarea element is not visible.\n",
       "        //\n",
       "        // The likelihood is the element won't even render, not even a flash,\n",
       "        // so some of these are just precautions. However in IE the element\n",
       "        // is visible whilst the popup box asking the user for permission for\n",
       "        // the web page to copy to the clipboard.\n",
       "        //\n",
       "\n",
       "        // Place in top-left corner of screen regardless of scroll position.\n",
       "        textArea.style.position = 'fixed';\n",
       "        textArea.style.top = 0;\n",
       "        textArea.style.left = 0;\n",
       "\n",
       "        // Ensure it has a small width and height. Setting to 1px / 1em\n",
       "        // doesn't work as this gives a negative w/h on some browsers.\n",
       "        textArea.style.width = '2em';\n",
       "        textArea.style.height = '2em';\n",
       "\n",
       "        // We don't need padding, reducing the size if it does flash render.\n",
       "        textArea.style.padding = 0;\n",
       "\n",
       "        // Clean up any borders.\n",
       "        textArea.style.border = 'none';\n",
       "        textArea.style.outline = 'none';\n",
       "        textArea.style.boxShadow = 'none';\n",
       "\n",
       "        // Avoid flash of white box if rendered for any reason.\n",
       "        textArea.style.background = 'transparent';\n",
       "\n",
       "\n",
       "        textArea.value = text;\n",
       "\n",
       "        document.body.appendChild(textArea);\n",
       "        textArea.focus();\n",
       "        textArea.select();\n",
       "\n",
       "        try {\n",
       "            if (!document.execCommand('copy')) {\n",
       "                console.log(\"Unable to copy text with document.execCommand()\");\n",
       "            }\n",
       "        } finally {\n",
       "            document.body.removeChild(textArea);\n",
       "        }\n",
       "    }\n",
       "\n",
       "    function copy_event_handler(event) {\n",
       "        copyTextToClipboard(event.target.dataset.hdf5Path);\n",
       "        event.preventDefault();\n",
       "    }\n",
       "\n",
       "    function enable_copylinks(parent) {\n",
       "        let links = parent.querySelectorAll(\".h5glance-dataset-copylink\");\n",
       "        links.forEach(function (link) {\n",
       "            link.addEventListener(\"click\", copy_event_handler);\n",
       "        });\n",
       "    }\n",
       "\n",
       "    // The code to actually trigger this is substituted below.\n",
       "    \n",
       "enable_copylinks(document.getElementById(\"h5glance-container-1\"));\n",
       "\n",
       "})();\n",
       "</script></div>"
      ],
      "text/plain": [
       "new_file_blosc2_bitshuffle_lz4.h5/\n",
       "└compressed_data\t[uint8: 3744 × 5286]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "H5Glance(\"new_file_blosc2_bitshuffle_lz4.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 135580 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "h5file = h5py.File(\"new_file_blosc2_bitshuffle_lz4.h5\", mode=\"r\")\n",
    "plt.imshow(h5file[\"/compressed_data\"][()], cmap=\"gray\")\n",
    "h5file.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "18M new_file_blosc2_bitshuffle_lz4.h5  19M new_file_uncompressed.h5\r\n"
     ]
    }
   ],
   "source": [
    "!ls -sh new_file*.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## HDF5 compression filters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Available through `h5py`\n",
    "\n",
    "- Pre-compression filter: [Byte-Shuffle](https://docs.h5py.org/en/stable/high/dataset.html#shuffle-filter) provided by `libhdf5`\n",
    "- [Compression filters provided by h5py](https://docs.h5py.org/en/stable/high/dataset.html#lossless-compression-filters):\n",
    "\n",
    "  - Provided by `libhdf5`: **\"gzip\"** and eventually **\"szip\"** (optional)\n",
    "  - Bundled with `h5py`: [**\"lzf\"**](https://confluence.hdfgroup.org/display/support/Filters#Filters-32000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "h5file = h5py.File(\"new_file_shuffle_gzip.h5\", mode=\"w\")\n",
    "h5file.create_dataset(\n",
    "    \"/compressed_data_shuffle_gzip\", data=data, shuffle=True, compression=\"gzip\")\n",
    "h5file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Provided by `hdf5plugin`\n",
    "\n",
    "Additional compression filters provided by `hdf5plugin`:\n",
    "\n",
    "BitShuffle, Blosc, Blosc2, BZip2, FciDecomp, LZ4, SZ, SZ3, ZFP, Zstandard\n",
    "\n",
    "10 out of the 29 [HDF5 registered filter plugins](https://confluence.hdfgroup.org/display/support/Registered+Filter+Plugins) as of September 2023"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "h5file = h5py.File(\"new_file_bitshuffle_lz4.h5\", mode=\"w\")\n",
    "h5file.create_dataset(\n",
    "    \"/compressed_data_bitshuffle_lz4\",\n",
    "    data=data,\n",
    "    compression=hdf5plugin.Bitshuffle()\n",
    ")\n",
    "h5file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### General purpose lossless compression\n",
    "\n",
    "- [Bitshuffle(nelems=0, cname='lz4', clevel=3)](http://www.silx.org/doc/hdf5plugin/latest/usage.html#bitshuffle) - ID [32008](https://confluence.hdfgroup.org/display/support/Filters#Filters-32008)\n",
    "  - **Bit**-Shuffle + LZ4, ZStd or no compression\n",
    "- [BZip2(blocksize=9)](http://www.silx.org/doc/hdf5plugin/latest/usage.html#bzip2) - ID [307](https://confluence.hdfgroup.org/display/support/Filters#Filters-307)\n",
    "- [LZ4(nbytes=0)](http://www.silx.org/doc/hdf5plugin/latest/usage.html#lz4) - ID [32004](https://confluence.hdfgroup.org/display/support/Filters#Filters-32004)\n",
    "- [Zstd(clevel=3)](http://www.silx.org/doc/hdf5plugin/latest/usage.html#zstd) - ID [32015](https://confluence.hdfgroup.org/display/support/Filters#Filters-32015)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Specific compression\n",
    "\n",
    "- [FciDecomp()](http://www.silx.org/doc/hdf5plugin/latest/usage.html#fcidecomp) - ID [32018](https://confluence.hdfgroup.org/display/support/Filters#Filters-32018): Based on JPEG-LS:\n",
    "  - **Optional**: requires C++11\n",
    "  - Data type: `(u)int8` or `(u)int16`\n",
    "  - Chunk shape: \"Image-like\"; 2 or 3 dimensions with at least 16 pixels and at most 65535 rows and columns and at most 4 planes for 3D datasets.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Lossy compression 1/2\n",
    "\n",
    "[SZcompressor](https://szcompressor.org/) family: error-bounded lossy compression\n",
    "\n",
    "- [SZ(absolute=None, relative=None, pointwise_relative=None)](http://www.silx.org/doc/hdf5plugin/latest/usage.html#sz) - ID [32017](https://confluence.hdfgroup.org/display/support/Filters#Filters-32017)\n",
    "- [SZ3(absolute=None, relative=None, norm2=None, peak_signal_to_noise_ratio=None)](http://www.silx.org/doc/hdf5plugin/latest/usage.html#sz3) - ID [32024](https://confluence.hdfgroup.org/display/support/Filters#Filters-32024)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Lossy compression 2/2\n",
    "\n",
    "- [ZFP(rate=None, precision=None, accuracy=None, reversible=False, minbits=None, maxbits=None, maxprec=None, minexp=None)](http://www.silx.org/doc/hdf5plugin/latest/usage.html#zfp) - ID [32013](https://confluence.hdfgroup.org/display/support/Filters#Filters-32013):\n",
    "  - Data type: `float32`, `float64`, `(u)int32`, `(u)int64`\n",
    "  - Chunk shape: must have at most 4 non-unity dimensions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Meta-compressor: Blosc family\n",
    "\n",
    "- [Blosc(cname='lz4', clevel=5, shuffle=1)](http://www.silx.org/doc/hdf5plugin/latest/usage.html#blosc) - ID [32001](https://confluence.hdfgroup.org/display/support/Filters#Filters-32001):\n",
    "  - Based on [c-blosc](https://github.com/blosc/c-blosc): A blocking, shuffling and lossless compression library\n",
    "  - Pre-compression shuffle: None, **Byte**-Shuffle, **Bit**-Shuffle\n",
    "  - Compression: `blosclz`, `lz4`, `lz4hc`, `snappy` (**optional**, requires C++11), `zlib`, `zstd`\n",
    "- [Blosc2(cname='blosclz', clevel=5, filters=1)](http://www.silx.org/doc/hdf5plugin/latest/usage.html#blosc2) - ID [32026](https://confluence.hdfgroup.org/display/support/Filters#Filters-32026):\n",
    "  - Based on [c-blosc2](https://github.com/blosc/c-blosc2): A high performance compressor optimized for binary data\n",
    "  - Pre-compression filters: None, **Byte**-Shuffle, **Bit**-Shuffle, and more\n",
    "  - Compression: `blosclz`, `lz4`, `lz4hc`, `zlib`, `zstd`\n",
    "  - More filters and compressions can be supported: `zfp`, `htj2k`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Equivalent filters\n",
    "\n",
    "`Blosc` and `Blosc2` includes some pre-compression filters and algorithms provided by other HDF5 compression filters:\n",
    "\n",
    "- HDF5 shuffle  => `Blosc2(..., filters=Blosc2.SHUFFLE)`\n",
    "- `Bitshuffle()` => `Blosc2(\"lz4\" or \"zstd\", 5, Blosc2.BITSHUFFLE)`\n",
    "- `LZ4()` => `Blosc2(\"lz4\", 9)`\n",
    "- `Zstd()` => `Blosc2(\"zstd\", 2)`\n",
    "\n",
    "`Blosc2` filter could also provide `ZFP`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### A look at performances on a single use case\n",
    "\n",
    "- Machine: Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz (40 cores)\n",
    "- Filesystem: /dev/shm\n",
    "- `hdf5plugin` built from source\n",
    "\n",
    "\n",
    "- Running on a single core\n",
    "\n",
    "\n",
    "- Diffraction tomography dataset: 100 frames from http://www.silx.org/pub/pyFAI/pyFAI_UM_2020/data_ID13/kevlar.h5\n",
    "- Dataset: 100x2167x2070, uint16, chunk: 2167x2070"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img alt=\"\" src=\"data:image/png;base64,[encoded data, 295640 characters]\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Multithreaded filter execution\n",
    "\n",
    "Some filters can use multithreading:\n",
    "\n",
    "- Blosc/Blosc2:\n",
    "  - Using a pool of threads\n",
    "  - Disabled by default for Blosc1\n",
    "  - Configurable with `BLOSC_NTHREADS` environment variable\n",
    "- Bitshuffle, Fcidecomp, SZ, SZ3, ZFP:\n",
    "  - Using OpenMP\n",
    "  - Enabled at compilation time\n",
    "  - If enabled, configurable with `OMP_NUM_THREADS` environment variable\n",
    "\n",
    "Performance do not increase linearly with the number of CPU cores used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Summary\n",
    "\n",
    "Having different pre-compression filters and compression algorithms at hand offers different read/write speed versus compression rate (and eventually error rate) trade-offs.\n",
    "\n",
    "Also to keep in mind availability/compatibility: Since `\"gzip\"` is included in `libHDF5` it is the most compatible one (and also `\"lzf\"` as included in `h5py`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Using `hdf5plugin` filters with other applications\n",
    "\n",
    "Set the `HDF5_PLUGIN_PATH` environment variable to: `hdf5plugin.PLUGINS_PATH`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "export HDF5_PLUGIN_PATH=`python3 -c \"\n",
    "import hdf5plugin; print(hdf5plugin.PLUGINS_PATH)\"`\n",
    "echo \"HDF5_PLUGIN_PATH=${HDF5_PLUGIN_PATH}\"\n",
    "ls ${HDF5_PLUGIN_PATH}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "**Note**: Only works for reading compressed datasets, **not for writing!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## A word about `hdf5plugin` license\n",
    "\n",
    "The source code of `hdf5plugin` itself is licensed under the [MIT](https://github.com/silx-kit/hdf5plugin/blob/main/LICENSE) license...\n",
    "\n",
    "It also embeds the source code of the provided compression filters and libraries which are licensed under [different open-source licenses](http://www.silx.org/doc/hdf5plugin/latest/information.html#license) (Apache, BSD-2, BSD-3, MIT, Zlib...) and copyrights."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Limitations\n",
    "\n",
    "- Only \"gzip\" available by default:\n",
    "  - Many compression filters provided by `hdf5plugin` are included in [c-blosc2](https://github.com/blosc/c-blosc2)\n",
    "- A central repository for filters source code: https://github.com/HDFGroup/hdf5_plugins?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Need to link filters with libhdf5:\n",
    "  - `hdf5plugin` relies on a \"hack\" to avoid linking with libhdf5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Compressed data accessed by \"chunks\" even if compressor uses smaller blocks\n",
    "- Multi-threaded access support\n",
    "- When reading compressed data, some memory copy could be spared:\n",
    "  - Direct chunk access offers a way to improve performance/flexibility"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Avoid memory copies\n",
    "\n",
    "Compression filters allocates a memory buffer to store decompressed data = memory copies.\n",
    "\n",
    "Allowing the user to pass a memory buffer through h5py->libhdf5->compression_filter would prevent it.\n",
    "\n",
    "An example with **h5py** and **Blosc2** (bitshuffle+lz4) for a 8.5MB chunk on 1 core (± ~300 µs):\n",
    "\n",
    "- Standard access `dataset[()]`: 8.9 ms\n",
    "- `read_direct()` to existing array: 5.2 ms\n",
    "- `read_direct_chunk()` & decompression with `blosc2`: 3.7 ms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Credits\n",
    "\n",
    "**To hdf5plugin** [**contributors**](https://github.com/silx-kit/hdf5plugin/graphs/contributors): [Armando Sole](https://github.com/vasole), [\\@orioltinto](https://github.com/orioltinto), [\\@mkitti ](https://github.com/mkitti), [\\@Florian-toll](https://github.com/Florian-toll), [Jerome Kieffer](https://github.com/kif), [\\@fpwg](https://github.com/fpwg), [\\@mobiusklein ](https://github.com/mobiusklein), [\\@junyuewang](https://github.com/junyuewang), [\\@Anthchirp](https://github.com/Anthchirp), and\n",
    "\n",
    "**to all contributors of embedded libraries**\n",
    "\n",
    "&nbsp;\n",
    "\n",
    "&nbsp;\n",
    "\n",
    "Partially funded by the [LEAPS-INNOV](https://www.leaps-innov.eu/) and [PaNOSC](https://www.panosc.eu/) EU-project.\n",
    "\n",
    "\n",
    "<img align=\"left\" vertical-align=\"top\" width=\"5%\" src=\"data:image/svg+xml,%3Csvg%20width%3D%2240.5%22%20height%3D%2227%22%20version%3D%221.1%22%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20xmlns%3Axlink%3D%22http%3A%2F%2Fwww.w3.org%2F1999%2Fxlink%22%3E%3Cdesc%3EEuropean%20flag%3C%2Fdesc%3E%3Cdefs%3E%3Cg%20id%3D%22s%22%3E%3Cg%20id%3D%22c%22%3E%3Cpath%20id%3D%22t%22%20transform%3D%22rotate%2818%203.1569%20-.5%29%22%20d%3D%22m0%200v1h0.5z%22%2F%3E%3Cuse%20transform%3D%22scale%28-1%2C1%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23t%22%2F%3E%3C%2Fg%3E%3Cg%20id%3D%22a%22%3E%3Cuse%20transform%3D%22rotate%2872%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23c%22%2F%3E%3Cuse%20transform%3D%22rotate%28144%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23c%22%2F%3E%3C%2Fg%3E%3Cuse%20transform%3D%22scale%28-1%2C1%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23a%22%2F%3E%3C%2Fg%3E%3C%2Fdefs%3E%3Crect%20width%3D%2240.5%22%20height%3D%2227%22%20fill%3D%22%23039%22%20stroke-width%3D%22.05%22%2F%3E%3Cg%20transform%3D%22matrix%281.5%2C0%2C0%2C1.5%2C20.25%2C13.5%29%22%20fill%3D%22%23fc0%22%3E%3Cuse%20y%3D%22-6%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23s%22%2F%3E%3Cuse%20y%3D%226%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23s%22%2F%3E%3Cg%20id%3D%22l%22%3E%3Cuse%20x%3D%22-6%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23s%22%2F%3E%3Cuse%20transform%3D%22rotate%28216%20-2.3442%20-2.1107%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23s%22%2F%3E%3Cuse%20transform%3D%22rotate%28144%2C-2.1107%2C-2.3442%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23s%22%2F%3E%3Cuse%20transform%3D%22rotate%2872%2C-4.6626%2C-2.0759%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23s%22%2F%3E%3Cuse%20transform%3D%22rotate%2872%20-5.0759%20.5335%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23s%22%2F%3E%3C%2Fg%3E%3Cuse%20transform%3D%22scale%28-1%2C1%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23l%22%2F%3E%3C%2Fg%3E%3C%2Fsvg%3E\"/>&nbsp;<small>This project has received funding from the European Union´s Horizon 2020 research and innovation programme under grant agreement no. 101004728 and 823852.</small>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Conlusion\n",
    "\n",
    "`hdf5plugin` provides additional HDF5 compression filters (namely: BitShuffle, Blosc, Blosc2, BZip2, FciDecomp, LZ4, SZ, SZ3, ZFP, Zstandard) mainly for use with [h5py](https://docs.h5py.org/).\n",
    "\n",
    "- Packaged for [**pip**](https://pypi.org/project/hdf5plugin/) and [**conda**](https://anaconda.org/conda-forge/hdf5plugin)\n",
    "- Documentation: http://www.silx.org/doc/hdf5plugin/latest/\n",
    "- Source code repository: https://github.com/silx-kit/hdf5plugin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbsphinx": "hidden",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Thanks for your attention! Questions?"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  },
  "nbsphinx": {
   "orphan": true
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
