feat: fetch responses from mastodon and bluesky (#23)

* feat: collect mastodon responses

* feat: queue operations for marking and deleting

* feat: delete response from queue when processed

* feat: fetch and render responses

* feat: response template for webmentions

* feat: collect urls when hooks are fired

* improvement: bluesky - get url from did

* feat: collect bluesky responses

* feat: panel queue view

* feat: allow up to for images for mastodon and bluesky #22

* feat: skip adding url to external posts #17

Author: mauricerenck

.gitattributes

@@ -9,6 +9,9 @@
 /.gitattributes export-ignore
 /.gitignore export-ignore
 /.htaccess export-ignore
+/.nvmrc export-ignore
+/.prettierrc export-ignore
+/.releaserc export-ignore
 /index.site.php export-ignore
 /package.json export-ignore
 /package-lock.json export-ignore

DEVELOPERS.md

@@ -0,0 +1,7 @@
+# Infos for developers
+
+Use `composer install` to install all dependencies. This creates an Kirby environment so you can test the plugin in the browser. You have to run a webserver for that and point it to the plugin root.
+
+Before pushing changes run `composer run build-release` this will run all unittests and remove dev dependencies.
+
+You can run the unittests by running: `composer test`. You can also use filters to only test specific things: `composer test -- --filter 'MastodonReceiver` for example

README.md

@@ -1,6 +1,6 @@
 # Kirby IndieConnector
 
-#### Send and receive Webmentions, send Mastodon and Bluesky posts or act like an ActivityPub instance
+#### Send and receive Webmentions, send Mastodon and Bluesky posts (POSSE) and get back responses, or act like an ActivityPub instance
 
 ![GitHub release](https://img.shields.io/github/release/mauricerenck/indieConnector.svg?maxAge=1800) ![License](https://img.shields.io/github/license/mashape/apistatus.svg) ![Kirby Version](https://img.shields.io/badge/Kirby-4%2B-black.svg)
 
@@ -10,8 +10,8 @@ IndieConnector is your interface to the IndieWeb. It'll help you to:
 
 -   Receive Webmentions
 -   Send Webmentions
--   Post to Mastodon
--   Post to Bluesky
+-   Post to Mastodon and Bluesky (POSSE)
+-   Collect responses from Mastodon and Bluesky
 -   Act like a ActivityPub Instance
 
 This plugins handles all the stuff around Webmentions and then normalizes the format and triggers a hook. Other plugins can then subscribe to this hook and use the data.
@@ -35,7 +35,8 @@ Use one of these three methods to install the plugin:
 -   [Sending Webmentions](docs/sending.md)
 -   [Activate the Panel view](docs/panel-view.md)
 -   [Post to Mastodon or Bluesky](docs/mastodon.md)
--   [Reply via Mastodon](docs/mastodon-replies.md)
+-   [Collect replies from Mastodon and Bluesky](docs/collecting-responses.md)
+-   [Get replies using brid.gy](docs/mastodon-replies.md)
 -   [Be a Mastodon Instance](docs/activitypub.md) (ActivityPub)
 -   [Subscribe to the hook](docs/hook.md)
 -   [Using webmention.io](docs/webmentionio.md)
@@ -48,6 +49,8 @@ Use one of these three methods to install the plugin:
 
 -   [x] Kirby 5 ready
 -   [x] Manual queue cleanup
+-   [x] Get replies from Mastodon and Bluesky without brid.gy
+-   [x] Option for using Kirby UUID permalinks in Mastodon/Bluesky posts
 -   [ ] Queue for sending webmentions
 -   [ ] Queue for sending mastodon posts
 -   [ ] Queue for sending bluesky posts
@@ -56,5 +59,3 @@ Use one of these three methods to install the plugin:
 -   [ ] Manual post to Mastodon and Bluesky
 -   [ ] Block hosts from within the panel
 -   [ ] Post complete texts to Mastodon and Bluesky splitted in threads
--   [ ] Option for using Kirby UUID permalinks in Mastodon/Bluesky posts
--   [ ] Get replies from Mastodon and Bluesky without brid.gy

blueprints/fields/block-webmentions.yml

@@ -14,3 +14,12 @@ fields:
         label: Mastodon URL
         type: url
         width: 1/3
+    blueskyStatusUrl:
+        label: Bluesky URL
+        type: url
+        width: 1/3
+    icSkipUrl:
+        label: Skip posting URL
+        type: toggle
+        default: false
+        width: 1/3

composer.json

@@ -53,7 +53,7 @@
             "Composer\\Config::disableProcessTimeout",
             "@php -S localhost:8000 kirby/router.php"
         ],
-        "test": "vendor/bin/phpunit --testdox",
+        "test": "vendor/bin/phpunit --testdox --colors=always tests \"$@\"",
         "build-test-package": "git archive HEAD -o indieConnector.zip --worktree-attributes",
         "build-composer": "composer install --no-dev --optimize-autoloader",
         "build-release": [
@@ -62,4 +62,4 @@
             "npm run build"
         ]
     }
-}
+}

docs/collecting-responses.md

@@ -0,0 +1,59 @@
+# Collecting Responses
+
+Collect likes, reposts and replies from Mastodon or Bluesky and show them on your pages.
+
+**This feature requires a SQLite database. [Learn how to set it up here](database.md)**
+
+When you use the IndieConnector to create posts on Mastodon or Bluesky whenever you publish a page (POSSE), you can now collect responses from those posts and show them as comments or in any other form on your pages.
+
+To use this feature you have to at least enable it in your config.php:
+
+```php
+'mauricerenck.indieConnector.responses.enabled' => true
+'mauricerenck.indieConnector.secret' => 'YOUR-SECRET
+```
+
+From now on the Response Collector will remember all post URLs. It will do so whenever the IndieConnector creates a new post on Mastodon or Bluesky or when you manually fill the Mastodon or Bluesky URL fields.
+
+## Collecting Responses
+
+To collect new reponses, you have to call a Webhook. The URL is:
+
+```
+https://example.com/indieConnector/cron/queue-responses?secret=YOUR-SECRET
+```
+
+Of course you have to change `example.com` to match your host and `YOUR-SECRET` to match the secret your set in your `config.php`.
+
+The plugin will then check the next 10 post URLs it knows and which haven't been check in the last hour.
+
+You can change how many URLs are checked and after how many minutes they should be checked again by setting:
+
+```php
+'mauricerenck.indieConnector.responses.limit' => 20
+'mauricerenck.indieConnector.responses.ttl' => 120
+```
+
+To check 20 posts in row check for the same URLs again after two hours (or 120 minutes).
+
+Depending on how frequently you want to check for responses you should adjust those values and schedule the cronjob which calls the Webhook URL.
+
+## Parse Responses
+
+New reposnes will be added to a queue. To process that queue, you have to call another Webhook:
+
+```
+https://example.com/indieConnector/cron/fetch-responses?secret=YOUR-SECRET
+```
+
+This Webhook will process 50 responses in a row. You can change that by setting:
+
+```php
+'mauricerenck.indieConnector.responses.queue.limit' => 100
+```
+
+New reponses will be processed as Webmention, so after setting everything up, you should see those reponses on the Panel View of IndieConnector in case you enabled it.
+
+## brid.gy
+
+If you used brid.gy to collect respones, please make sure to deactivate it, otherwise you might end up with duplicates.

docs/mastodon-replies.md

@@ -1,7 +1,8 @@
 # Replies via Mastodon
 
-If you want to enable people to reply to your posts via Mastodon, you currently have to use brid.gy to do so. I am currently working on a solution that will allow you to do this without brid.gy.
+If you want to enable people to reply to your posts via Mastodon, ~~you currently have to use brid.gy to do so. I am currently working on a solution that will allow you to do this without brid.gy.~~ [there is now a native solution for that](docs/collecting-responses.md).
 
+You can sill use brid.gy.
 Sign up for [brid.gy](https://brid.gy/) and sign in with with Mastodon. Choose cross posting. Follow all the steps. Brid.gy will then send you Webmentions whenever someone replies, likes or boosts your Mastodon post. This webmentions will then be collected by IndieConnector.
 
 ### Set the URL of your Mastodon post

docs/options.md

@@ -63,6 +63,8 @@
 | ----------------------- | ----------------- | -------------------------------------------------------------------- |
 | `post.prefereLanguage`  | `-`               | Use another language than your default language to use the text from |
 | `post.usePermalinkUrl`  | `false`           | Use the permalink url instead of the page url                        |
+| `post.skipUrl`          | `false`           | NEVER add the url to the post                                        |
+| `post.skipUrlTemplates` | `[]`              | Do not add the url to the post when using the given templates        |
 | `post.textfields`       | `['description']` | Text source fields for posting elsewhere                             |
 | `post.imagefield`       | `''`              | Image source field for posting elsewhere, must be one image          |
 | `post.allowedTemplates` | `[]`              | Set templates allowed to send webmentions                            |
@@ -90,6 +92,15 @@
 | `bluesky.handle`   | `‘‘`    | Your user handle                     |
 | `bluesky.password` | `‘‘`    | Your bluesky app password            |
 
+## Collecting responses ([details](responses.md))
+
+| Option                  | Default | Description                                                       |
+| ----------------------- | ------- | ----------------------------------------------------------------- |
+| `responses.enabled`     | `false` | Enable collecting responses                                       |
+| `responses.limit`       | `10`    | Number of posts to check for responses                            |
+| `responses.ttl`         | `60`    | Minutes after which a post url should be re-checked for responses |
+| `responses.queue.limit` | `50`    | Number of items to process per run                                |
+
 ## ActivityPub beta ([details](activitiypub.md))
 
 | Option              | Default | Description               |

index.js

@@ -2,7 +2,7 @@
 
 namespace mauricerenck\IndieConnector;
 
-use Kirby;
+use Kirby\Cms\App as Kirby;
 
 @require_once __DIR__ . '/dependencies/indieweb-comments.php';
 @include_once __DIR__ . '/vendor/autoload.php';
@@ -15,6 +15,9 @@
         'webmention-endpoint' => __DIR__ . '/snippets/webmention-endpoint.php',
         'activitypub-wm' => __DIR__ . '/snippets/activitypub-webmention.php',
     ],
+    'templates' => [
+        'indie-post-response' => __DIR__ . '/templates/pages/response.php',
+    ],
     'tags' => require_once __DIR__ . '/plugin/kirbytags.php',
     'blueprints' => [
         'indieconnector/fields/webmentions' => __DIR__ . '/blueprints/fields/block-webmentions.yml',