Provide information about bytes types in HDF5 DataFrames in Python 3

[pksohn]

Some of the pandas DataFrames within HDF5 files with urbansim-related data (tables of parcels, buildings, etc) have index names (i.e. df.index.name) that are saved as binary data (bytes type) rather than strings. Since bytes and str is handled the same way in Python 2.7, it wasn't a problem. In Python 3, they're treated separately.

We've decided not to add code to orca table registration functions that convert these names. Instead, folks using Python 3 with data that has these quirks should just update their datasets. When we release the version of UrbanSim that has python 3 compatibility, we should include a note about this in the release notes.

We can include a code snippet to do this conversion in HDF5 files, something like:

import numpy
import os
import pandas as pd

new_store = pd.HDFStore('path/to/newstore.h5', mode='w')

with pd.HDFStore('path/to/oldstore.h5') as store:
    for tablename in store.keys():
        table = store[tablename]
        if table.index.name and type(table.index.name) == numpy.bytes_:
            table.index.name = table.index.name.decode()
        new_store.put(tablename, table, format='t')
new_store.close()

[pksohn]

Note: I added a functions in utils.py to:

• Check whether an HDF5 store has dataframes with index name or column names of bytes type
• Convert a DataFrame that has bytes types

No utility to convert a whole store altogether, though I'm not sure that's necessary.