Merge branch 'master' into release

Author: devstein

.github/PULL_REQUEST_TEMPLATE/contributor_checklist.md renamed to .github/pull_request_template.md

@@ -1,5 +1,15 @@
 # Changelog
 
+## 0.17.0
+
+### Features
+
+* Customizable Livy authentication methods [#662](https://github.com/jupyter-incubator/sparkmagic/pull/662). Thanks @alexismacaskilll 
+
+### Bug fixes
+
+* Fix Dockerfile.jupyter build  [#672](https://github.com/jupyter-incubator/sparkmagic/pull/672)
+
 ## 0.16.0
 
 ### Bug fixes

CHANGELOG.md

@@ -1,4 +1,4 @@
-FROM jupyter/base-notebook:ae885c0a6226
+FROM jupyter/base-notebook
 
 ARG dev_mode=false
 

Dockerfile.jupyter

@@ -68,16 +68,20 @@ See the [Sending Local Data to Spark notebook](examples/Send%20local%20data%20to
         jupyter serverextension enable --py sparkmagic
 
 ## Authentication Methods
-
-Sparkmagic supports:
-
-* No auth
+Sparkmagic supports: 
+* No auth 
 * Basic authentication
 * Kerberos
 
+The [Authenticator](sparkmagic/sparkmagic/auth/customauth.py) is the mechanism for authenticating to Livy. The base
+Authenticator used by itself supports no auth, but it can be subclassed to enable authentication via other methods. 
+Two such examples are the [Basic](sparkmagic/sparkmagic/auth/basic.py) and [Kerberos](sparkmagic/sparkmagic/auth/kerberos.py) Authenticators. 
+
+### Kerberos Authenticator
+
 Kerberos support is implemented via the [requests-kerberos](https://github.com/requests/requests-kerberos) package. Sparkmagic expects a kerberos ticket to be available in the system. Requests-kerberos will pick up the kerberos ticket from a cache file. For the ticket to be available, the user needs to have run [kinit](https://web.mit.edu/kerberos/krb5-1.12/doc/user/user_commands/kinit.html) to create the kerberos ticket.
 
-### Kerberos Configuration
+#### Kerberos Configuration
 
 By default the `HTTPKerberosAuth` constructor provided by the `requests-kerberos` package will use the following configuration
 ```python
@@ -97,7 +101,56 @@ but this will not be right configuration for every context, so it is able to pas
         "send_cbt": true
     }
 }
-```
+``` 
+
+### Custom Authenticators
+
+You can write custom Authenticator subclasses to enable authentication via other mechanisms. All Authenticator subclasses 
+should override the `Authenticator.__call__(request)` method that attaches HTTP Authentication to the given Request object. 
+
+Authenticator subclasses that add additional class attributes to be used for the authentication, such as the [Basic] (sparkmagic/sparkmagic/auth/basic.py) authenticator which adds `username` and `password` attributes, should override the `__hash__`, `__eq__`, `update_with_widget_values`, and `get_widgets` methods to work with these new attributes. This is necessary in order for the Authenticator to use these attributes in the authentication process.
+
+#### Using a Custom Authenticator with Sparkmagic
+
+If your repository layout is:  
+
+        .
+        ├── LICENSE
+        ├── README.md
+        ├── customauthenticator
+        │   ├── __init__.py 
+        │   ├── customauthenticator.py 
+        └── setup.py
+
+Then to pip install from this repository, run: `pip install git+https://git_repo_url/#egg=customauthenticator`
+
+After installing, you need to register the custom authenticator with Sparkmagic so it can be dynamically imported. This can be done in two different ways:
+1.  Edit the configuration file at [`~/.sparkmagic/config.json`](config.json) with the following settings:
+
+       ```json
+       {
+           "authenticators": {
+               "Kerberos": "sparkmagic.auth.kerberos.Kerberos",
+               "None": "sparkmagic.auth.customauth.Authenticator",
+               "Basic_Access": "sparkmagic.auth.basic.Basic",
+               "Custom_Auth": "customauthenticator.customauthenticator.CustomAuthenticator"
+         }
+       }
+       ```
+
+       This adds your `CustomAuthenticator` class in `customauthenticator.py` to Sparkmagic. `Custom_Auth` is the authentication type that will be displayed in the `%manage_spark` widget's Auth type dropdown as well as the Auth type passed as an argument to the -t flag in the `%spark add session` magic.   
+
+2. Modify the `authenticators` method in [`sparkmagic/utils/configuration.py`](sparkmagic/sparkmagic/utils/configuration.py) to return your custom authenticator:
+
+      ```python
+      def authenticators():
+              return {
+                      u"Kerberos": u"sparkmagic.auth.kerberos.Kerberos",
+                      u"None": u"sparkmagic.auth.customauth.Authenticator",
+                      u"Basic_Access": u"sparkmagic.auth.basic.Basic", 
+                      u"Custom_Auth": u"customauthenticator.customauthenticator.CustomAuthenticator"
+              }
+      ```
 
 ## Papermill
 

README.md

@@ -1 +1 @@
-__version__ = '0.16.0'
+__version__ = '0.17.0'

autovizwidget/autovizwidget/__init__.py

@@ -1 +1 @@
-__version__ = '0.16.0'
+__version__ = '0.17.0'

hdijupyterutils/hdijupyterutils/__init__.py

@@ -41,6 +41,11 @@
       }
     }
   },
+  "authenticators": {
+    "Kerberos": "sparkmagic.auth.kerberos.Kerberos",
+    "None": "sparkmagic.auth.customauth.Authenticator", 
+    "Basic_Access": "sparkmagic.auth.basic.Basic"
+  },
 
   "wait_for_idle_timeout_seconds": 15,
   "livy_session_startup_timeout_seconds": 60,

sparkmagic/example_config.json

@@ -4,6 +4,7 @@
                        'sparkmagic/controllerwidget',
                        'sparkmagic/kernels',
                        'sparkmagic/livyclientlib',
+                       'sparkmagic/auth',
                        'sparkmagic/magics',
                        'sparkmagic/kernels/pysparkkernel',
                        'sparkmagic/kernels/sparkkernel',

sparkmagic/setup.py

@@ -1,12 +1,11 @@
-__version__ = '0.16.0'
+__version__ = '0.17.0'
 
 from sparkmagic.serverextension.handlers import load_jupyter_server_extension
 
 
 def _jupyter_server_extension_paths():
-    return [{
-        "module": "sparkmagic"
-    }]
+    return [{"module": "sparkmagic"}]
+
 
 def _jupyter_nbextension_paths():
     return []

sparkmagic/sparkmagic/__init__.py

@@ -0,0 +1,75 @@
+"""Class for implementing a basic access authenticator for SparkMagic"""
+
+from sparkmagic.livyclientlib.exceptions import BadUserDataException
+from hdijupyterutils.ipywidgetfactory import IpyWidgetFactory
+from requests.auth import HTTPBasicAuth
+from .customauth import Authenticator
+
+class Basic(HTTPBasicAuth, Authenticator):
+    """Basic Access authenticator for SparkMagic"""
+    def __init__(self, parsed_attributes=None):
+        """Initializes the Authenticator with the attributes in the attributes
+        parsed from a %spark magic command if applicable, or with default values
+        otherwise.
+
+        Args:
+            self,
+            parsed_attributes (IPython.core.magics.namespace): The namespace object that
+            is created from parsing %spark magic command.
+        """
+        if parsed_attributes is not None:
+            if parsed_attributes.user is '' or parsed_attributes.password is '':
+                new_exc = BadUserDataException("Need to supply username and password arguments for "\
+                    "Basic Access Authentication. (e.g. -a username -p password).")
+                raise new_exc
+            self.username = parsed_attributes.user
+            self.password = parsed_attributes.password
+        else:
+            self.username = 'username'
+            self.password = 'password'
+        HTTPBasicAuth.__init__(self, self.username, self.password)
+        Authenticator.__init__(self, parsed_attributes)
+
+    def get_widgets(self, widget_width):
+        """Creates and returns a list with an address, username, and password widget
+
+        Args:
+            widget_width (str): The width of all widgets to be created.
+
+        Returns:
+            Sequence[hdijupyterutils.ipywidgetfactory.IpyWidgetFactory]: list of widgets
+        """
+        ipywidget_factory = IpyWidgetFactory()
+
+        self.user_widget = ipywidget_factory.get_text(
+            description='Username:',
+            value=self.username,
+            width=widget_width
+        )
+
+        self.password_widget = ipywidget_factory.get_text(
+            description='Password:',
+            value=self.password,
+            width=widget_width
+        )
+
+        widgets = [self.user_widget, self.password_widget]
+        return Authenticator.get_widgets(self, widget_width) + widgets
+
+    def update_with_widget_values(self):
+        """Updates url, username, and password to be the value of their respective widgets."""
+        Authenticator.update_with_widget_values(self)
+        self.username = self.user_widget.value
+        self.password = self.password_widget.value
+
+    def __eq__(self, other):
+        if not isinstance(other, Basic):
+            return False
+        return self.url == other.url and self.username == other.username and \
+            self.password == other.password
+
+    def __call__(self, request):
+        return HTTPBasicAuth.__call__(self, request)
+
+    def __hash__(self):
+        return hash((self.username, self.password, self.url, self.__class__.__name__))

sparkmagic/sparkmagic/auth/__init__.py

@@ -0,0 +1,58 @@
+"""Base class for implementing an authentication provider for SparkMagic"""
+
+from hdijupyterutils.ipywidgetfactory import IpyWidgetFactory
+from sparkmagic.utils.constants import WIDGET_WIDTH
+
+class Authenticator(object):
+    """Base Authenticator for all Sparkmagic authentication providers."""
+
+    def __init__(self, parsed_attributes=None):
+        """Initializes the Authenticator with the attributes in the attributes
+        parsed from a %spark magic command if applicable, or with default values
+        otherwise.
+
+        Args:
+            self,
+            parsed_attributes (IPython.core.magics.namespace): The namespace object that
+            is created from parsing %spark magic command.
+        """
+        if parsed_attributes is not None:
+            self.url = parsed_attributes.url
+        else:
+            self.url = 'http://example.com/livy'
+        self.widgets = self.get_widgets(WIDGET_WIDTH)
+
+    def get_widgets(self, widget_width):
+        """Creates and returns an address widget
+
+        Args:
+            widget_width (str): The width of all widgets to be created.
+
+        Returns:
+            Sequence[hdijupyterutils.ipywidgetfactory.IpyWidgetFactory]: list of widgets
+        """
+        ipywidget_factory = IpyWidgetFactory()
+
+        self.address_widget = ipywidget_factory.get_text(
+            description='Address:',
+            value='http://example.com/livy',
+            width=widget_width
+        )
+        widgets = [self.address_widget]
+        return widgets
+
+    def update_with_widget_values(self):
+        """Updates url to be value in address widget."""
+        self.url = self.address_widget.value
+
+    def __call__(self, request):
+        """subclasses should override"""
+        return None
+
+    def __eq__(self, other):
+        if not isinstance(other, Authenticator):
+            return False
+        return self.url == other.url
+
+    def __hash__(self):
+        return hash((self.url, self.__class__.__name__))

sparkmagic/sparkmagic/auth/basic.py

@@ -0,0 +1,28 @@
+"""Class for implementing a Kerberos authenticator for SparkMagic"""
+
+from requests_kerberos import HTTPKerberosAuth
+import sparkmagic.utils.configuration as conf
+from .customauth import Authenticator
+
+
+class Kerberos(HTTPKerberosAuth, Authenticator):
+    """Kerberos authenticator for SparkMagic"""
+
+    def __init__(self, parsed_attributes=None):
+        """Initializes the Authenticator with the attributes in the attributes
+        parsed from a %spark magic command if applicable, or with default values
+        otherwise.
+
+        Args:
+            self,
+            parsed_attributes (IPython.core.magics.namespace): The namespace object that
+            is created from parsing %spark magic command.
+        """
+        HTTPKerberosAuth.__init__(self, **conf.kerberos_auth_configuration())
+        Authenticator.__init__(self, parsed_attributes)
+
+    def __call__(self, request):
+        return HTTPKerberosAuth.__call__(self, request)
+
+    def __hash__(self):
+        return hash((self.url, self.__class__.__name__))