0-Intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# HDF5 and pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "HDF5 is both a data container and a library that is meant to store and retrieve large amounts of data in a convenient way.  It is used extensively in science, engineering, finance and many other fields.  HDF5 has to major Python packages that wrap it:\n",
    "\n",
    "1. h5py\n",
    "2. PyTables\n",
    "\n",
    "Also, pandas is using one of them (PyTables) so as to efficiently store and retrieve dataframes.\n",
    "\n",
    "During this tutorial you will learn how to create and read HDF5 datasets using both h5py and PyTables, as well as introducing the concept of data chunking and how it can be used to compress data efficiently.  Moreover, a gentle description of the querying capabilities of PyTables will be made.  Finally, we will see how HDF5 and pandas can interact, not only to serialize pandas dataframes, but also to efficiently query them right on-disk (i.e. with no need to load the data in-memory). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Caveats for following the tutorial:\n",
    "\n",
    "1. These notebooks have been created and tested mainly on Jupyter notebook 4.4 and Python 3.6, but Python 2.7 should work equally fine, except for some particularities that will be seldom used.\n",
    "\n",
    "2. You can follow the tutorial by re-playing the [provided notebooks](https://github.com/FrancescAlted/PyData-BCN/releases).  For those of you with problems with the Wifi, there are pendrives available.\n",
    "\n",
    "3. **In case** you cannot reproduce the desired results in your own laptop, do not worry too much; my advice is that you just concentrate in tutor's explanations and ask in case something is not clear enough."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Requisites\n",
    "\n",
    "* Jupyter notebook\n",
    "* numpy\n",
    "* h5py\n",
    "* tables (pytables)\n",
    "* pandas\n",
    "* matplotlib\n",
    "* cartopy\n",
    "\n",
    "These should be all in Anaconda or in the PyPI repo.  The only exception could be `cartopy` that might not exist in the regular conda channel, so in order to install it, try the `conda-forge` channel instead:\n",
    "\n",
    "```\n",
    "$ conda install -c conda-forge cartopy\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "1. [Basic Datatypes](1-Basic-Datatypes.ipynb)\n",
    "\n",
    "1. [Chunking](2-Chunking.ipynb)\n",
    "\n",
    "1. [Using Compression](3-Using-Compression.ipynb)\n",
    "\n",
    "1. [Structuring Datasets](4-Structuring-Datasets.ipynb)\n",
    "\n",
    "1. [Querying with PyTables](5-Querying-With-PyTables.ipynb)\n",
    "\n",
    "1. [Integration with pandas](6-Integration-With-Pandas.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}