Merge pull request #592 from jupyter-incubator/master

Release 0.14.0

Author: itamarst

CHANGELOG.md

@@ -1,5 +1,13 @@
 # Changelog
 
+## 0.14.0
+
+### Bug fixes
+
+* Enabled heartbeat by default, so long-running tasks don't time out. Thanks to John Pugliesi for the bug report.
+* The PySpark kernel uses Python 3 lexer, instead of Python 2. Python 2 support is going away in the near future.
+* Fixed papermill support; there's a different option now you need to use, `all_errors_are_fatal`. See the README for details. Thanks to Devin Stein for the patch.
+
 ## 0.13.1
 
 ### Other changes

README.md

@@ -21,6 +21,7 @@ The Sparkmagic project includes a set of magics for interactively running Spark
 * Automatic visualization of SQL queries in the PySpark, Spark and SparkR kernels; use an easy visual interface to interactively construct visualizations, no code required
 * Easy access to Spark application information and logs (`%%info` magic)
 * Ability to capture the output of SQL queries as Pandas dataframes to interact with other Python libraries (e.g. matplotlib)
+* Send local files or dataframes to a remote cluster (e.g. sending pretrained local ML model straight to the Spark cluster)
 * Authenticate to Livy via Basic Access authentication or via Kerberos
 
 ## Examples
@@ -36,6 +37,10 @@ The sparkmagic library provides a %%spark magic that you can use to easily run c
 The sparkmagic library also provides a set of Scala and Python kernels that allow you to automatically connect to a remote Spark cluster, run code and SQL queries, manage your Livy server and Spark job configuration, and generate automatic visualizations.
 See [Pyspark](examples/Pyspark%20Kernel.ipynb) and [Spark](examples/Spark%20Kernel.ipynb) sample notebooks.
 
+### 3. Sending data to Spark%20Kernel
+
+See the [sending data to Spark notebook](examples/Send local data to Spark.ipynb).
+
 ## Installation
 
 1. Install the library
@@ -80,8 +85,8 @@ If you want Papermill rendering to stop on a Spark error, edit the `~/.sparkmagi
 
 ```json
 {
-    "spark_statement_errors_are_fatal": True,
-    "shutdown_session_on_spark_statement_errors": True
+    "shutdown_session_on_spark_statement_errors": true,
+    "all_errors_are_fatal": true
 }
 ```
 

autovizwidget/autovizwidget/__init__.py

@@ -1 +1 @@
-__version__ = '0.13.1'
+__version__ = '0.14.0'

examples/Magics in IPython Kernel.ipynb

@@ -2526,7 +2526,7 @@
     "df871ce7b5fa43999cd43ed47e911d32": {
      "views": [
       {
-       "cell_index": 26
+       "cell_index": 26.0
       }
      ]
     },

examples/Pyspark Kernel.ipynb

@@ -136,7 +136,7 @@
       ]
      },
      "metadata": {},
-     "output_type": "display_data"
+     "output_type": "execute_result"
     }
    ],
    "source": [
@@ -167,7 +167,7 @@
       ]
      },
      "metadata": {},
-     "output_type": "display_data"
+     "output_type": "execute_result"
     },
     {
      "data": {
@@ -179,7 +179,7 @@
       ]
      },
      "metadata": {},
-     "output_type": "display_data"
+     "output_type": "execute_result"
     }
    ],
    "source": [
@@ -224,7 +224,7 @@
       ]
      },
      "metadata": {},
-     "output_type": "display_data"
+     "output_type": "execute_result"
     },
     {
      "data": {
@@ -236,7 +236,7 @@
       ]
      },
      "metadata": {},
-     "output_type": "display_data"
+     "output_type": "execute_result"
     }
    ],
    "source": [
@@ -482,6 +482,62 @@
     "print(len(query2))"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Send to spark (%%send_to_spark)\n",
+    "\n",
+    "This command gives you the ability to send local data to remote cluster. Note that passing even a simple string allows you to retain pretrained, saved ML model from local filesystem.\n",
+    "\n",
+    "#### Pass string to a remote cluster"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%local\n",
+    "variable = 'this string will be available in spark cluster'"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%send_to_spark -i variable -n custom_name -t str"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print custom_name"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Send pretrained scikit-learn model to a remote cluster"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%local\n",
+    "## todo"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},