Add webhook support

[Lyncredible]

Summary

This change adds webhook support to enable real-time syncing of remote updates. It takes advantage of the webhooks functionality provided by Microsoft. This fixes #1620.

How does it work

1. The onedrive process stands up a local web server to receive incoming notifications (to be sent by Microsoft)
2. The onedrive process registers the public-facing url of the web server with Microsoft, via the POST /subscriptions endpoint.
3. When there is an remote update, Microsoft sends a notification request to the pre-registered url.
4. The onedrive process receives the notification request, and kicks off a new delta sync loop to fetch updates.

Code changes

1. onedrive.d contains the bulk of logic to manage the local web server and to register/renew subscriptions with Microsoft. It is also responsible for processing incoming notification requests. When such requests are received, it sends signals to the main sync loop.
2. main.d contains the logic to process incoming signals and execute delta sync to fetch updates.
3. The embeded web server is made possible via cgi.d from adamdruppe/arsd
4. New config options are added to config.d and also documented.

Testing

Unfortunately I could only test this manually with a very straightforward setup: personal OneDrive account with no shared folders. More testing is welcome!

What is next

Currently the public-facing url has to be statically configured. This should work well for most users either with static public IP or with dynamic DNS services. We could consider adding UPnP support to allow dynamic configuration via automatic port forwarding.

[Lyncredible]

My editor also removed a bunch of extra whitespaces. It is better to review this PR with the Hide whitespace changes option turned on (or by adding ?w=1 to the PR url)

[abraunegg]

@Lyncredible
Thanks for the PR, will review further when I have time.

One question I do have, and it is not immediately visible, is the license for cgi.d. If you can format your readme.md in the new arsd folder as per https://github.com/abraunegg/onedrive/tree/master/src/notifications so that it can be easilly seen that cgi.d is cgi.d copyright 2008-2021, Adam D. Ruppe. Provided under the Boost Software License. This will help folk who rely on the licensing side of things to be squeaky clean to enable the package to be including in distributions.

Regarding the '*' -> '-' changes, if you can put those back please given that is the standard I have been using.

Re the whitespace removal at the end of various lines - leave that one - that is on the whiteboard at the moment post #1494 being implemented, as there is a lot of extra whitespace that has crept in over the years.

I would also add, under the main readme.md (given the removal of 'While local changes are uploaded right away ... ' text), add under 'Features' Supports webhooks to subscribe to remote change notifications - or whatever else is suitably valid.

In terms of when this PR will be merged into 'master', most likely after #1494 is merged - so potentially you will need to think about a few updates based on those changes coming down the pipeline. I would PR merge this before that change, however, I am too far along the journey to stop, merge, backtrack then revisit all the language change code.

This will give also folk the opportunity to test this PR to really bake / shake this out in it's current format as well.

[Lyncredible]

One question I do have, and it is not immediately visible, is the license for cgi.d. If you can format your readme.md in the new arsd folder as per https://github.com/abraunegg/onedrive/tree/master/src/notifications so that it can be easilly seen that cgi.d is cgi.d copyright 2008-2021, Adam D. Ruppe. Provided under the Boost Software License. This will help folk who rely on the licensing side of things to be squeaky clean to enable the package to be including in distributions.

That is a great suggestion. I have updated the readme file accordingly.

Regarding the '*' -> '-' changes, if you can put those back please given that is the standard I have been using.

Alright, I have changed it back.

Re the whitespace removal at the end of various lines - leave that one - that is on the whiteboard at the moment post #1494 being implemented, as there is a lot of extra whitespace that has crept in over the years.

Cool. I will wait and see what to do after #1494 is merged.

I would also add, under the main readme.md (given the removal of 'While local changes are uploaded right away ... ' text), add under 'Features' Supports webhooks to subscribe to remote change notifications - or whatever else is suitably valid.

I have already added a line Real-Time syncing of remote updates under Features. Is that okay?

In terms of when this PR will be merged into 'master', most likely after #1494 is merged - so potentially you will need to think about a few updates based on those changes coming down the pipeline. I would PR merge this before that change, however, I am too far along the journey to stop, merge, backtrack then revisit all the language change code.

This will give also folk the opportunity to test this PR to really bake / shake this out in it's current format as well.

Sounds good to me!

[abraunegg]

@Lyncredible
Whilst this is merged into Master - we now have this depreciation warning:

src/arsd/cgi.d(3570): Deprecation: function `onedrive.OneDriveWebhook.serve.serveEmbeddedHttp!((cgi) => handle(cgi, parentTid), Cgi, 5000000L).serveEmbeddedHttp` function requires a dual-context, which is deprecated
src/onedrive.d(126):        instantiated from here: `serveEmbeddedHttp!((cgi) => handle(cgi, parentTid), Cgi, 5000000L)`

Please Fix

[abraunegg]

@Lyncredible
This PR, whilst it was merged, has been reverted via #1716

[Lyncredible]

@abraunegg Thanks for spotting the issue. I have created PR #1718 to address the deprecation warning. The fix can be seen in commit 5b89455.

In short, we need OneDriveWebhook.handle to be a static function, otherwise we would hit the member function "requires a dual-context, which is deprecated" warning. To workaround the problem, I changed OneDriveWebhook class to be a singleton class, even thought it does not have to be one.

The problem is deemed a bug and should be fixed in the compilers eventually. The singleton stuff could be undone when it is fixed. The root cause is described in the links below.

• https://issues.dlang.org/show_bug.cgi?id=5710
• https://forum.dlang.org/post/fkyppfxzegenniyzztos@forum.dlang.org

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.