Merge pull request #1232 from PyBossa/twitter-importer

Twitter importer

Author: alejandrodob

Diff for: doc/changelog/index.rst

@@ -1,6 +1,7 @@
 Changelog
 =========
 
+* v1.3.0_
 * v1.2.2_
 * v1.2.1_
 * v1.2.0_
@@ -10,6 +11,7 @@ Changelog
 * v1.1.0_
 * v0.2.3_
 
+.. _v1.3.0: v1.3.0.html
 .. _v1.2.1: v1.2.2.html
 .. _v1.2.1: v1.2.1.html
 .. _v1.2.0: v1.2.0.html

Diff for: doc/changelog/v1.3.0.rst

@@ -0,0 +1,5 @@
+================
+Changelog v1.3.0
+================
+
+ * Add Twitter importer.

Diff for: doc/customizing.rst

@@ -252,6 +252,8 @@ them.
 .. _Twitter: https://dev.twitter.com/
 .. _`TWITTER_CONSUMER_KEY`: https://github.com/PyBossa/pybossa/blob/master/settings_local.py.tmpl#L52
 .. _`TWITTER_CONSUMER_SECRET`: https://github.com/PyBossa/pybossa/blob/master/settings_local.py.tmpl#L53
+.. note::
+    This will also enable the Twitter task importer.
 
 Facebook
 ~~~~~~~~
@@ -264,7 +266,6 @@ variables: `FACEBOOK_APP_ID`_ and `FACEBOOK_APP_SECRET`_ and uncomment them.
 .. _`FACEBOOK_APP_ID`: https://github.com/PyBossa/pybossa/blob/master/settings_local.py.tmpl#L54
 .. _`FACEBOOK_APP_SECRET`: https://github.com/PyBossa/pybossa/blob/master/settings_local.py.tmpl#L55
 
-
 Google
 ~~~~~~
 
@@ -331,7 +332,6 @@ type of users:
 
     ANNOUNCEMENT = {'root': 'Your secret message'}
 
-
 There is an example of the **ANNOUNCEMENT** variable in the
 `settings_local.py.tmpl <https://github.com/PyBossa/pybossa/blob/master/settings_local.py.tmpl>`_
 file, so you can easily adapt it for your own server. Basically, the
@@ -727,7 +727,7 @@ interested users.
 Enabling the Flickr Task importer
 =================================
 
-PyBossa has five different types of built-in importers. Users can use them to
+PyBossa has several different types of built-in importers. Users can use them to
 import tasks for their projects directly from the Web interface. However, using
 the Flickr one requires an API key and shared secret from Flickr in order to
 communicate with the service.
@@ -743,16 +743,29 @@ refer to `here <https://www.flickr.com/services/api/>`_.
 Enabling the Dropbox Task importer
 ==================================
 
-In addition to the Flickr importer, PyBossa also offers the Dropbox importer, which
-allows to import directly all kind of files from a Dropbox account. In order to
-use it, you'll need to register your PyBossa server as a Dropbox app, as explained
+PyBossa also offers the Dropbox importer, which allows to import directly all
+kind of files from a Dropbox account. In order to use it, you'll need to
+register your PyBossa server as a Dropbox app, as explained
 `here <https://www.dropbox.com/developers/dropins/chooser/js#setup>`_.
 
 Don't worry about the Javascript snippet part, we've already handled that for you.
 Instead, get the App key you will be given and add it to your settings_local.py::
 
     DROPBOX_APP_KEY = 'your-key'
 
+Enabling the Twitter Task importer
+==================================
+
+If you already have enabled the Twitter authentication, then the Twitter task
+importer will be enabled too. Otherwise, you will need to create an application
+in Twitter_ and copy and paste the **Consumer key and secret** into the next
+variables: `TWITTER_CONSUMER_KEY`_ and `TWITTER_CONSUMER_SECRET`_ and uncomment
+them.
+
+.. _Twitter: https://dev.twitter.com/
+.. note::
+    This will also enable PyBossa's Twitter login.
+
 Enabling Server Sent Events
 ===========================
 

Diff for: doc/user/overview.rst

@@ -137,7 +137,7 @@ built-in :ref:`task-creator`. You have to do the following:
    see several options. The first five are for using the different kinds of
    importers supported by PyBossa.
 
-.. image:: http://i.imgur.com/e9GhNlE.png
+.. image:: http://i.imgur.com/GbdkgEI.png
 
 The **CSV** importer, allows you to upload your own CSV file:
 
@@ -247,11 +247,11 @@ project, you will have to follow the next steps:
    :width: 100%
 
 
-3. And  click on the **Import Tasks** 
+3. And click on the **Import Tasks** 
    button. After clicking on it you will see several different options. The first
    five correspond to the different importers PyBossa supports:
 
-.. image:: http://i.imgur.com/e9GhNlE.png
+.. image:: http://i.imgur.com/GbdkgEI.png
 
 4. Click in the **Use an EpiCollect Project** one.
 
@@ -321,7 +321,7 @@ These are the steps:
 
 2. Then click on the **Import Tasks** button, and select the **Flickr importer**:
 
-.. image:: http://i.imgur.com/e9GhNlE.png
+.. image:: http://i.imgur.com/GbdkgEI.png
 
 3. Type the ID of the Flickr set you want to import the photos from, then click
 on the import button:
@@ -359,7 +359,7 @@ presenter of the project:
 * link: the link to the Dropbox page showing the file.
 * link_raw: the link to the raw file served by Dropbox. This is the one you'll have to use if you want to direct link to the file from the presenter (e.g. for using an image in a <img> tag, you'd do: <img src=task.info.link_raw>).
 
-In addition to these generic information, the Dropbox importer also will recognize
+In addition to this generic information, the Dropbox importer will also recognize
 some kind of files by their extension and will attach some extra information to
 them.
 
@@ -398,9 +398,71 @@ These are the steps:
 
 .. image:: http://i.imgur.com/u5vusQR.png
 
-2. Then click on the **Import Tasks** button, and select the **Flickr importer**:
+2. Then click on the **Import Tasks** button, and select the **Dropbox importer**:
+
+.. image:: http://i.imgur.com/GbdkgEI.png
+
+3. Fill in the two fields you will find in the form. The first one is for the
+source of your tweets. If you want them to be imported from a user account, then
+write it with the "@" symbol, like "@PyBossa". If you just want to import tweets
+containing a word on them (or a #hashtag), then type it there. The second field
+is for you to specify how many tweets you want to import. You can import as many
+as you want!
+
+Finally, click on the "Import" button, and you are done:
+
+.. image:: http://i.imgur.com/hYWxI19.png
+
+.. _twitter-import:
+
+Importing the tasks from a Twitter account or search result
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Another option for importing tasks is using the built-in Twitter importer. It
+allows to import tweets as tasks from either a specified Twitter user account, or
+from the results returned from a search to the Twitter search API.
+
+Tasks imported with it will have the tweet data attached to their info field,
+and can later be used from within the task presenter. This data is a direct
+transcription of the data returned by the Twitter API, in particular a Tweet_
+object.
+
+.. _Tweet: https://dev.twitter.com/overview/api/tweets
+
+Please notice that the values returned by the Twitter API may vary. However,
+the following fields are guaranteed to be always included in the info field of
+the tasks:
+
+* created_at: the date and time the tweet was made.
+* favorite_count: number of times the tweet has been marked as 'favorite'.
+* retweet_count: number of times the tweet has been retweeted.
+* coordinates: geographic coordinates of the place the tweet was made from. Note
+that this is not always available for every tweet.
+* tweet_id: the internal ID handled by Twitter to identify this tweet.
+* user: an object_ with information about the tweet author, as returned by the
+Twitter API.
+* text: the actual content of the tweet.
+
+In addition, an extra field "user_screen_name" has been added to the info field:
+* user_screen_name: the screen name (or 'handle') of the author of the tweet.
+
+.. _object: https://dev.twitter.com/overview/api/users
+
+For more information, please refer to the Twitter_ documentation.
+.. _Twitter: https://dev.twitter.com/
+
+.. note::
+    **When importing tweets from a search, retweets will be ignored!**
+
+So, to import tasks with the Twitter importer, do as follows:
+
+1. Navigate to your project's page and click in the **Tasks** section:
+
+.. image:: http://i.imgur.com/u5vusQR.png
+
+2. Then click on the **Import Tasks** button, and select the **Twitter importer**:
 
-.. image:: http://i.imgur.com/e9GhNlE.png
+.. image:: http://i.imgur.com/GbdkgEI.png
 
 3. Click on the "Choose from Dropbox" icon. You will be asked your Dropbox
 account credentials. then select as many files as you want:

Diff for: doc/user/project_settings.rst

@@ -258,10 +258,10 @@ navigation bar:
 
 Tasks can be imported using any of the PyBossa built-in importers, such
 as :ref:`csv-import` and :ref:`epicollect-import`. To set up an autoimporter,
-please refer to the instructions for :ref:`csv-import`, :ref:`epicollect-import`
-or :ref:`flickr-import`, as the procedure is the same:
+please refer to the instructions for :ref:`csv-import`, :ref:`epicollect-import`,
+:ref:`flickr-import` or :ref:`twitter-import`, as the procedure is the same:
 
-.. image:: http://i.imgur.com/nvTP7aX.png
+.. image:: http://i.imgur.com/20PHlHe.png
 
 The only difference is that the tasks won't be imported only once, but regularly,
 as explained. However, the same behaviour should be expected, so autoimporting a

Diff for: pybossa/__init__.py

@@ -26,4 +26,4 @@
 """
 
 
-__version__ = "1.2.2"  # pragma: no cover
+__version__ = "1.3.0"  # pragma: no cover

Diff for: pybossa/core.py

@@ -276,15 +276,22 @@ def setup_blueprints(app):
     for bp in blueprints:
         app.register_blueprint(bp['handler'], url_prefix=bp['url_prefix'])
 
-    # The RQDashboard is actually registering a blueprint to the app, so this is
-    # a propper place for it to be initialized
     from rq_dashboard import RQDashboard
     RQDashboard(app, url_prefix='/admin/rq', auth_handler=current_user,
                 redis_conn=sentinel.master)
 
 
 def setup_external_services(app):
     """Setup external services."""
+    setup_twitter_login(app)
+    setup_facebook_login(app)
+    setup_google_login(app)
+    setup_flickr_importer(app)
+    setup_dropbox_importer(app)
+    setup_twitter_importer(app)
+
+
+def setup_twitter_login(app):
     try:  # pragma: no cover
         if (app.config['TWITTER_CONSUMER_KEY'] and
                 app.config['TWITTER_CONSUMER_SECRET']):
@@ -297,9 +304,10 @@ def setup_external_services(app):
         print inst
         print "Twitter signin disabled"
         log_message = 'Twitter signin disabled: %s' % str(inst)
-        app.logger.error(log_message)
+        app.logger.info(log_message)
 
-    # Enable Facebook if available
+
+def setup_facebook_login(app):
     try:  # pragma: no cover
         if (app.config['FACEBOOK_APP_ID']
                 and app.config['FACEBOOK_APP_SECRET']):
@@ -312,9 +320,10 @@ def setup_external_services(app):
         print inst
         print "Facebook signin disabled"
         log_message = 'Facebook signin disabled: %s' % str(inst)
-        app.logger.error(log_message)
+        app.logger.info(log_message)
+
 
-    # Enable Google if available
+def setup_google_login(app):
     try:  # pragma: no cover
         if (app.config['GOOGLE_CLIENT_ID']
                 and app.config['GOOGLE_CLIENT_SECRET']):
@@ -327,24 +336,28 @@ def setup_external_services(app):
         print inst
         print "Google signin disabled"
         log_message = 'Google signin disabled: %s' % str(inst)
-        app.logger.error(log_message)
+        app.logger.info(log_message)
 
-    # Enable Flickr if available
+
+def setup_flickr_importer(app):
     try:  # pragma: no cover
         if (app.config['FLICKR_API_KEY']
                 and app.config['FLICKR_SHARED_SECRET']):
             flickr.init_app(app)
             from pybossa.view.flickr import blueprint as flickr_bp
             app.register_blueprint(flickr_bp, url_prefix='/flickr')
+            importer_params = {'api_key': app.config['FLICKR_API_KEY']}
+            importer.register_flickr_importer(importer_params)
     except Exception as inst:  # pragma: no cover
         print type(inst)
         print inst.args
         print inst
         print "Flickr importer not available"
         log_message = 'Flickr importer not available: %s' % str(inst)
-        app.logger.error(log_message)
+        app.logger.info(log_message)
+
 
-    # Enable Dropbox if available
+def setup_dropbox_importer(app):
     try:  # pragma: no cover
         if app.config['DROPBOX_APP_KEY']:
             importer.register_dropbox_importer()
@@ -354,7 +367,25 @@ def setup_external_services(app):
         print inst
         print "Dropbox importer not available"
         log_message = 'Dropbox importer not available: %s' % str(inst)
-        app.logger.error(log_message)
+        app.logger.info(log_message)
+
+
+def setup_twitter_importer(app):
+    try:  # pragma: no cover
+        if (app.config['TWITTER_CONSUMER_KEY'] and
+                app.config['TWITTER_CONSUMER_SECRET']):
+            importer_params = {
+                'consumer_key': app.config['TWITTER_CONSUMER_KEY'],
+                'consumer_secret': app.config['TWITTER_CONSUMER_SECRET']
+            }
+            importer.register_twitter_importer(importer_params)
+    except Exception as inst:  # pragma: no cover
+        print type(inst)
+        print inst.args
+        print inst
+        print "Twitter importer not available"
+        log_message = 'Twitter importer not available: %s' % str(inst)
+        app.logger.info(log_message)
 
 
 def setup_geocoding(app):

Diff for: pybossa/flickr_client.py

@@ -27,16 +27,15 @@ class FlickrClient(object):
     def __init__(self, app=None):
         """Init method."""
         self.app = app
-        self.client = None
+        self.oauth_client = None
         if app is not None:  # pragma: no cover
             self.init_app(app)
 
     def init_app(self, app):  # pragma: no cover
         """Method to init object following factories pattern."""
         from flask import session
-        from pybossa.core import importer
         self.app = app
-        self.client = OAuth().remote_app(
+        self.oauth_client = OAuth().remote_app(
             'flickr',
             request_token_url='https://www.flickr.com/services/oauth/request_token',
             access_token_url='https://www.flickr.com/services/oauth/access_token',
@@ -45,9 +44,7 @@ def init_app(self, app):  # pragma: no cover
             consumer_secret=app.config['FLICKR_SHARED_SECRET'],
             access_token_method='GET')
         tokengetter = functools.partial(self.get_token, session)
-        self.client.tokengetter(tokengetter)
-        importer_params = {'api_key': app.config['FLICKR_API_KEY']}
-        importer.register_flickr_importer(importer_params)
+        self.oauth_client.tokengetter(tokengetter)
 
     def get_user_albums(self, session):
         """Get user albums from Flickr."""
@@ -58,7 +55,7 @@ def get_user_albums(self, session):
                    '&primary_photo_extras=url_q'
                    '&format=json&nojsoncallback=1'
                    % self._get_user_nsid(session))
-            res = self.client.get(url, token='')
+            res = self.oauth_client.get(url, token='')
             if res.status == 200 and res.data.get('stat') == 'ok':
                 albums = res.data['photosets']['photoset']
                 return [self._extract_album_info(album) for album in albums]
@@ -70,15 +67,15 @@ def get_user_albums(self, session):
 
     def authorize(self, *args, **kwargs):
         """Authorize method."""
-        return self.client.authorize(*args, **kwargs)
+        return self.oauth_client.authorize(*args, **kwargs)
 
     def authorized_response(self):
         """Authorized response."""
-        return self.client.authorized_response()
+        return self.oauth_client.authorized_response()
 
     def get_oauth_client(self):
         """Get OAuth client."""
-        return self.client
+        return self.oauth_client
 
     def get_token(self, session):
         """Get token from session."""