Merge pull request #7 from mwouts/version_0.2

Version 0.2

Author: mwouts

CHANGELOG.md

@@ -0,0 +1,20 @@
+0.2.0 (2019-11-20)
+------------------
+
+Added
+=====
+- Large tables are downsampled (#2)
+
+Changed
+=======
+- Javascript code moved to Javascript files
+
+Fixed
+=====
+- Tables with many columns are now well rendered (#5)
+
+
+0.1.0 (2019-04-23)
+------------------
+
+Initial release

HISTORY.md

@@ -19,7 +19,6 @@ df
 
 You don't see any table above? Please either open the [HTML export](https://mwouts.github.io/itables/) of this notebook, or run this README on [Binder](https://mybinder.org/v2/gh/mwouts/itables/master?filepath=README.md)!
 
-
 # Quick start
 
 Install the package with
@@ -129,16 +128,10 @@ show(
 
 ## Column width
 
-FIXME: This does not appear to be working...
-
-```python
-show(df, columnDefs=[{"width": "200px", "target": 3}])
-```
-
-But in some cases - a table with many column like the one below, we can use the `width` parameter...
+For tables that are larger than the notebook, the `columnDefs` argument allows to specify the desired width. If you wish you can also change the default in `itables.options`.
 
 ```python
-show(x.to_frame().T, columnDefs=[{"width": "80px", "targets": "_all"}])
+show(x.to_frame().T, columnDefs=[{"width": "120px", "targets": "_all"}])
 ```
 
 ## HTML in cells
@@ -169,22 +162,37 @@ Not currently implemented. May be made available at a later stage using the [sel
 Not currently implemented. May be made available at a later stage thanks to the [buttons](https://datatables.net/extensions/buttons/) extension for datatable.
 
 
-## Large table support
+## Downsampling
 
-`itables` will not display dataframes that are larger than `maxBytes`, which is equal to 1MB by default. Truncate the dataframe with `df.head()`, or set the `maxBytes` parameter or option to an other value to display the dataframe. Or deactivate the limit with `maxBytes=0`.
+When the data in a table is larger than `maxBytes`, which is equal to 64KB by default, `itables` will display only a subset of the table - one that fits into `maxBytes`. If you wish, you can deactivate the limit with `maxBytes=0`, change the value of `maxBytes`, or similarly set a limit on the number of rows (`maxRows`, defaults to 0) or columns (`maxColumns`, defaults to `pd.get_option('display.max_columns')`).
 
 Note that datatables support [server-side processing](https://datatables.net/examples/data_sources/server_side). At a later stage we may implement support for larger tables using this feature.
 
 ```python
-df = wb.get_indicators()
+df = wb.get_indicators().head(500)
+opt.maxBytes = 10000
 df.values.nbytes
 ```
 
 ```python
-opt.maxBytes = 1000000
 df
 ```
 
+To show the table in full, we can modify the value of `maxBytes` either locally:
+
+```python
+show(df, maxBytes=0)
+```
+
+or globally:
+
+```python
+opt.maxBytes = 2**20
+df
+```
+
+The `maxRows` and `maxColumns` arguments work similarly.
+
 # References
 
 ## DataTables
@@ -195,7 +203,8 @@ df
 
 ## Alternatives
 
-ITables is not a Jupyter widget, which means that it does not allows you to **edit** the content of the dataframe.
+ITables uses basic Javascript, and because of this it will only work in Jupyter Notebook, not in JupyterLab. It is not a Jupyter widget, which means that it does not allows you to **edit** the content of the dataframe.
+
 If you are looking for Jupyter widgets, have a look at
 - [QGrid](https://github.com/quantopian/qgrid) by Quantopian
 - [IPyaggrid](https://dgothrek.gitlab.io/ipyaggrid/) by Louis Raison and Olivier Borderies

README.md

@@ -0,0 +1,67 @@
+import pandas as pd
+import logging
+
+logging.basicConfig()
+logger = logging.getLogger(__name__)
+
+
+def downsample(df, max_rows=0, max_columns=0, max_bytes=0):
+    """Return a subset of the dataframe that fits the limits"""
+    org_rows, org_columns, org_bytes = len(df.index), len(df.columns), df.values.nbytes
+    df = _downsample(df, max_rows=max_rows, max_columns=max_columns, max_bytes=max_bytes)
+
+    if len(df.index) < org_rows or len(df.columns) < org_columns:
+        reasons = []
+        if org_rows > max_rows > 0:
+            reasons.append('maxRows={}'.format(max_rows))
+        if org_columns > max_columns > 0:
+            reasons.append('maxColumns={}'.format(max_columns))
+        if org_bytes > max_bytes > 0:
+            reasons.append('nbytes={}>{}=maxBytes'.format(org_bytes, max_bytes))
+
+        logger.warning('showing {}x{} of {}x{} as {}. See https://mwouts.github.io/itables/#downsampling'.format(
+            len(df.index), len(df.columns), org_rows, org_columns, ' and '.join(reasons)))
+
+    return df
+
+
+def _downsample(df, max_rows=0, max_columns=0, max_bytes=0):
+    """Implementation of downsample - may be called recursively"""
+    if len(df.index) > max_rows > 0:
+        second_half = max_rows // 2
+        first_half = max_rows - second_half
+        if second_half:
+            df = pd.concat((df.iloc[:first_half], df.iloc[-second_half:]))
+        else:
+            df = df.iloc[:first_half]
+
+    if len(df.columns) > max_columns > 0:
+        second_half = max_columns // 2
+        first_half = max_columns - second_half
+        if second_half:
+            df = pd.concat((df.iloc[:, :first_half], df.iloc[:, -second_half:]), axis=1)
+        else:
+            df = df.iloc[:, :first_half]
+
+    if df.values.nbytes > max_bytes > 0:
+        max_rows = len(df.index)
+        max_columns = len(df.columns)
+
+        # we want to decrease max_rows * max_columns by df.values.nbytes / max_bytes
+        max_product = max_rows * max_columns / (float(df.values.nbytes) / max_bytes)
+
+        while max_product >= 1:
+            max_rows = max(max_rows // 2, 1)
+            if max_rows * max_columns <= max_product:
+                return _downsample(df, max_rows, max_columns, max_bytes)
+
+            max_columns = max(max_columns // 2, 1)
+            if max_rows * max_columns <= max_product:
+                return _downsample(df, max_rows, max_columns, max_bytes)
+
+        # max_product < 1.0:
+        df = df.iloc[:1, :1]
+        df.iloc[0, 0] = '...'
+        return df
+
+    return df

index.html

@@ -1,59 +1,86 @@
 """HTML/js representation of Pandas dataframes"""
 
+import os
+import io
 import re
 import uuid
 import json
-import warnings
+import logging
 import numpy as np
 import pandas as pd
 import pandas.io.formats.format as fmt
 from IPython.core.display import display, Javascript, HTML
 import itables.options as opt
+from .downsample import downsample
+
+logging.basicConfig()
+logger = logging.getLogger(__name__)
 
 try:
     unicode  # Python 2
 except NameError:
     unicode = str  # Python 3
 
 
+def read_package_file(*path):
+    current_path = os.path.dirname(__file__)
+    with io.open(os.path.join(current_path, *path), encoding='utf-8') as fp:
+        return fp.read()
+
+
 def load_datatables():
     """Load the datatables.net library, and the corresponding css"""
-    display(Javascript("""require.config({
-    paths: {
-        datatables: 'https://cdn.datatables.net/1.10.19/js/jquery.dataTables.min',
-    }
-});
+    load_datatables_js = read_package_file('javascript', 'load_datatables_connected.js')
+    eval_functions_js = read_package_file('javascript', 'eval_functions.js')
+    load_datatables_js += "\n$('head').append(`<script>\n" + eval_functions_js + "\n</` + 'script>');"
+
+    display(Javascript(load_datatables_js))
+
+
+def _formatted_values(df):
+    """Return the table content as a list of lists for DataTables"""
+    formatted_df = df.copy()
+    for col in formatted_df:
+        x = formatted_df[col]
+        if x.dtype.kind in ['b', 'i', 's']:
+            continue
+
+        if x.dtype.kind == 'O':
+            formatted_df[col] = formatted_df[col].astype(unicode)
+            continue
 
-$('head').append('<link rel="stylesheet" type="text/css" \
-                href = "https://cdn.datatables.net/1.10.19/css/jquery.dataTables.min.css" > ');
+        formatted_df[col] = np.array(fmt.format_array(x.values, None))
+        if x.dtype.kind == 'f':
+            try:
+                formatted_df[col] = formatted_df[col].astype(np.float)
+            except ValueError:
+                pass
 
-$('head').append('<style> table td { text-overflow: ellipsis; overflow: hidden; } </style>');
-"""))
+    return formatted_df.values.tolist()
 
 
 def _datatables_repr_(df=None, tableId=None, **kwargs):
     """Return the HTML/javascript representation of the table"""
 
     # Default options
     for option in dir(opt):
-        if not option in kwargs and not option.startswith("__"):
+        if option not in kwargs and not option.startswith("__"):
             kwargs[option] = getattr(opt, option)
 
     # These options are used here, not in DataTable
     classes = kwargs.pop('classes')
     showIndex = kwargs.pop('showIndex')
-    maxBytes = kwargs.pop('maxBytes')
+    maxBytes = kwargs.pop('maxBytes', 0)
+    maxRows = kwargs.pop('maxRows', 0)
+    maxColumns = kwargs.pop('maxColumns',  pd.get_option('display.max_columns'))
 
     if isinstance(df, (np.ndarray, np.generic)):
         df = pd.DataFrame(df)
 
     if isinstance(df, pd.Series):
         df = df.to_frame()
 
-    if df.values.nbytes > maxBytes > 0:
-        raise ValueError('The dataframe has size {}, larger than the limit {}\n'.format(df.values.nbytes, maxBytes) +
-                         'Please print a smaller dataframe, or enlarge or remove the limit:\n'
-                         'import itables.options as opt; opt.maxBytes=0')
+    df = downsample(df, max_rows=maxRows, max_columns=maxColumns, max_bytes=maxBytes)
 
     # Do not show the page menu when the table has fewer rows than min length menu
     if 'paging' not in kwargs and len(df.index) <= kwargs.get('lengthMenu', [10])[0]:
@@ -77,87 +104,14 @@ def _datatables_repr_(df=None, tableId=None, **kwargs):
         thead = thead.replace('<th></th>', '', 1)
     html_table = '<table id="' + tableId + '" class="' + classes + '"><thead>' + thead + '</thead></table>'
 
-    # Table content as 'data' for DataTable
-    formatted_df = df.reset_index() if showIndex else df.copy()
-    for col in formatted_df:
-        x = formatted_df[col]
-        if x.dtype.kind in ['b', 'i', 's']:
-            continue
-
-        if x.dtype.kind == 'O':
-            formatted_df[col] = formatted_df[col].astype(unicode)
-            continue
-
-        formatted_df[col] = np.array(fmt.format_array(x.values, None))
-        if x.dtype.kind == 'f':
-            try:
-                formatted_df[col] = formatted_df[col].astype(np.float)
-            except ValueError:
-                pass
-
-    kwargs['data'] = formatted_df.values.tolist()
+    kwargs['data'] = _formatted_values(df.reset_index() if showIndex else df)
 
     try:
         dt_args = json.dumps(kwargs)
         return """<div>""" + html_table + """
 <script type="text/javascript">
 require(["datatables"], function (datatables) {
-    $(document).ready(function () {
-        function eval_functions(map_or_text) {
-            if (typeof map_or_text === "string") {
-                if (map_or_text.startsWith("function")) {
-                    try {
-                        // Note: parenthesis are required around the whole expression for eval to return a value!
-                        // See https://stackoverflow.com/a/7399078/911298.
-                        //
-                        // eval("local_fun = " + map_or_text) would fail because local_fun is not declared
-                        // (using var, let or const would work, but it would only be declared in the local scope
-                        // and therefore the value could not be retrieved).
-                        const func = eval(`(${map_or_text})`);
-                        if (typeof func !== "function") {
-                            // Note: backquotes are super convenient! 
-                            // https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals
-                            console.error(
-                                `Evaluated expression "${map_or_text}" is not a function (type is ${typeof func})`
-                            );
-                            return map_or_text;
-                        }
-                        // Return the function        
-                        return func;
-                    } catch (e) {
-                        // Make sure to print the error with a second argument to console.error().
-                        console.error(`itables was not able to parse "${map_or_text}"`, e);
-                    }
-                }
-            } else if (typeof map_or_text === "object") {
-                if (map_or_text instanceof Array) {
-                    // Note: "var" is now superseded by "let" and "const".
-                    // https://medium.com/javascript-scene/javascript-es6-var-let-or-const-ba58b8dcde75
-                    const result = [];
-                    // Note: "for of" is the best way to iterate through an iterable.
-                    // https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for...of
-                    for (const item of map_or_text) {
-                        result.push(eval_functions(item));
-                    }
-                    return result;
-
-                    // Alternatively, more functional approach in one line:
-                    // return map_or_text.map(eval_functions);
-                } else {
-                    const result = {};
-                    // Object.keys() is safer than "for in" because otherwise you might have keys
-                    // that aren't defined in the object itself.
-                    //
-                    // See https://stackoverflow.com/a/684692/911298.
-                    for (const item of Object.keys(map_or_text)) {
-                        result[item] = eval_functions(map_or_text[item]);
-                    }
-                    return result;
-                }
-            }
-
-            return map_or_text;
-        }
+    $(document).ready(function () {        
         var dt_args = """ + dt_args + """;
         dt_args = eval_functions(dt_args);
         table = $('#""" + tableId + """').DataTable(dt_args);
@@ -167,7 +121,7 @@ def _datatables_repr_(df=None, tableId=None, **kwargs):
 </div>
 """
     except TypeError as error:
-        warnings.warn(str(error))
+        logger.error(str(error))
         return ''