Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add SDH-GameSync v1.0.0 #784

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

AkazaRenn
Copy link

@AkazaRenn AkazaRenn commented Mar 15, 2025

SDH-GameSync

A fork of Decky Cloud Save that I have rewritten (probably) over 70% of the code. The major differences between this plugin and Decky Cloud Save are:

  1. Support for per-game sync, it syncs games files based on the filter specific to the starting/stopping game, saved in the settings folder named <appId>.filter
  2. Support for screenshot upload, when a screenshot is taken, it can upload it to the cloud and remove the local copy automatically
  3. Refactored filters edit page that supports adding/removing both include and exclude sync rules in the same file with the ability to reorder entries
  4. Improved log page that supports separation of plugin logs and each sync logs, also a refresh button and allows resync to be triggered via controller

Task Checklist

Developer

  • I am the original author or an authorized maintainer of this plugin.
  • I have abided by the licenses of the libraries I am utilizing, including attaching license notices where appropriate.

Plugin

  • I have verified that my plugin works properly on the Stable and Beta update channels of SteamOS.
  • I have verified my plugin is unique or provides more/alternative functionality to a plugin already on the store.

Backend

  • No: I am using a custom backend other than Python.
  • No: I am using a tool or software from a 3rd party FOSS project that does not have it's dependencies statically linked.
  • Yes: I am using a custom binary that has all of it's dependencies statically linked.

Community

  • I have tested and left feedback on two other pull requests for new or updating plugins.
  • I have commented links to my testing report in this PR.

#771 (comment)
#778 (comment)

Testing

  • Tested by a third party on SteamOS Stable or Beta update channel.

@AkazaRenn AkazaRenn requested a review from a team as a code owner March 15, 2025 16:26
@EMERALD0874
Copy link
Member

Can you please give context as to why this needs to be a separate plugin listing/fork and not a PR to the original plugin?

@AkazaRenn
Copy link
Author

AkazaRenn commented Mar 15, 2025

@EMERALD0874 yeah it's actually pretty straightforward, the change is too large: https://github.com/GedasFX/decky-cloud-save/compare/main...AkazaRenn:SDH-CloudSaveFork:main?expand=1
This plugin's implementation is quite different from Decky Cloud Save under the hood. If I merge +4,566 -3,443 lines to Decky Cloud Save, it will simply be replace by SDH-CloudSaveFork, and there would definitely be a lot of current users unhappy with the change, that's something I want to avoid. Publishing it as a new plugin would allow users to choose between them.

@GedasFX
Copy link
Contributor

GedasFX commented Mar 15, 2025

@AkazaRenn crazy to see you again!

A bit of history: I have developed DCS as a means to just back up certain paths to somewhere as I had Lutris constantly delete my save files for no reason. It was my first plugin, and really my first time using python in a "more serious" setting, so that plugin really desperately needed a remake.

Over time people discovered the plugin and kept asking for features, that really I had no interest in adding, but as I was new to FOSS, and had time still, I added some requests. After a while, life hit hard, and I had honestly by far the busiest year of my life. Had no time to add features. Caved to pressure and managed to squeeze in the MUCH MUCH requested bi-sync feature, which flooded me with confused people how to use it. Left me really drained and just made me want to quit FOSS development entirely.

Final nail in the coffin, however, was the base plugin itself - it was purpose built for one feature only - backup, and the code base made any deviations very hard. Whether its backwards compatibility, changing users habit, or anything else, that plugin needed a complete rewrite. so that's why (almost a year ago) started work on ludusavi wrapper, which I finished today. Not much work, but thats how busy I just was xD

Simply put, I want to get rid myself of that plugin. It has outlived its usefulness, getting anything done is a delicate dance of backwards compatibility, convoluted testing, and support for features, I had no desire to see in the first place. If someone wants to take over, I am all for it, but as mentioned before, its best to just leave it deprecated, have people move on, and then remove it.

@GedasFX
Copy link
Contributor

GedasFX commented Mar 15, 2025

I am more than willing to review + test this, as regardless if it was gonna be a PR to original repo, then it would have to have my review anyway 😄

@AkazaRenn
Copy link
Author

Hiiiii @GedasFX , good to see you back! I'm so glad to hear the support from you and I cannot thank you enough for the ground work provided. Sorry that I said I could do it during the holidays but managed to delay it for over three months 😆

If you want to give a check on the plugin, fell free to get it from here https://github.com/AkazaRenn/SDH-CloudSaveFork/releases/tag/20250315161024

@GedasFX
Copy link
Contributor

GedasFX commented Mar 15, 2025

Its all good! Glad you are alright ^^ That was basically last week when you mentioned it to me anyway :D

I started to have a look at the code, and I believe it will have issues as you changed the base image of the Dockerfile, and I remember I had issues with CI, but may have been fixed since last I pushed.

@AkazaRenn
Copy link
Author

@GedasFX here comes the interesting part, the docker image now is no longer responsible for building the image since the decky binary will take care of it. That image, is where I steal the rclone binary from 😉

@EMERALD0874
Copy link
Member

@AkazaRenn Given Gedas' interest in releasing maintainership, could you handle this as either PR to their repo or a transfer of maintainership of the existing Decky Cloud Save? I'd prefer we not have a plugin on the store that isn't being actively maintained/supported.

@AkazaRenn
Copy link
Author

@AkazaRenn Given Gedas' interest in releasing maintainership, could you handle this as either PR to their repo or a transfer of maintainership of the existing Decky Cloud Save? I'd prefer we not have a plugin on the store that isn't being actively maintained/supported.

I'm not sure current users will be happy about it suddenly getting revamped into something totally different. To me the better way is to keep Gedas' plugin in store until it's broken, then we take it down and recommend users to migrate to mine.

@EMERALD0874
Copy link
Member

If it's something totally different, I'm concerned that repeatedly referring to your work as "a fork of Decky Cloud Save" and "Cloud Save Fork" could confuse users into thinking it's just an upgraded version of the original Decky Cloud Save like I thought. An original plugin name and a description change to "A fork of Decky Cloud Save with [features added]" may help here.

@AkazaRenn
Copy link
Author

@EMERALD0874 sure let me think about it, the major reason that it's called CloudSaveFork is because I suck at naming 🥹

@AkazaRenn
Copy link
Author

@EMERALD0874 how about "Game Sync"? Would it be too general?

@EMERALD0874
Copy link
Member

If that's a name you like, it's fine by me.

@AkazaRenn
Copy link
Author

@EMERALD0874 updated!

@AkazaRenn AkazaRenn changed the title feat: Add SDH-CloudSaveFork v1.0.0 feat: Add SDH-GameSync v1.0.0 Mar 16, 2025
@EMERALD0874
Copy link
Member

Please rebase from main and resolve merge conflicts so you aren't building multiple plugins. After that, I'll deploy to the testing store.

@AkazaRenn
Copy link
Author

@EMERALD0874 done, thanks for your help!

Copy link
Member

@EMERALD0874 EMERALD0874 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for testing. We may revisit the concern of having a fork of a plugin that accomplishes the same goal also being on the store, but I understand your view and want to find a solution that works for everyone. At least for now, I think I'm good with adding it as is if we get a good testing report.

@AkazaRenn
Copy link
Author

@EMERALD0874 sorry could you please trigger the pipeline again? I just found a stupid issue and had to make a fix

@pizzadude
Copy link

The plugin works fine but I did have to modify config.json to make the rclone bisync (which this new fork uses) command work reliably, otherwise any tiny error would require me to resync, which was annoying and time consuming.

    "additional_bisync_args": [
        "--conflict-resolve",
        "newer",
        "--resilient",
        "--recover",
        "--max-lock",
        "2m"

Sorry if this is the wrong place to post this, but there is no github issues page for SDH-GameSync so I decided to post it here. It's not really an issue, maybe a suggestion for the defaults or some would-be help for anyone stumbling across this on google.

@AkazaRenn
Copy link
Author

@pizzadude thanks for testing it out! It makes sense that a resync is required every time an error occurred, but as for the reason of the error, it may differ between the cloud providers, that's also one of the reasons I'm moving all bisync args into the config to make it possible for users to modify them, you can probably take look at the sync logs to tell what actually was caught by rclone that it thinks the sync should fail. As for the repo, you can find it here https://github.com/AkazaRenn/SDH-GameSync/

@GedasFX
Copy link
Contributor

GedasFX commented Mar 23, 2025

image
I tried to install this plugin from testing store, but I just cannot? It is likely due to to fact that I probably have something messed up in my steam deck. Investigating.

@AkazaRenn
Copy link
Author

image
I tried to install this plugin from testing store, but I just cannot? It is likely due to to fact that I probably have something messed up in my steam deck. Investigating.

Ahhh sorry that's the problem on my side, I imported a library that's not Python standard library (requests). I've made a fix but need the pipeline to be re-triggered.

@EMERALD0874
Copy link
Member

@GedasFX @AkazaRenn Sorry for the wait, the pipeline is deploying your changes now.

@AkazaRenn
Copy link
Author

@EMERALD0874 thank you so much!

@GedasFX
Copy link
Contributor

GedasFX commented Mar 23, 2025

I love bisync i love bisync i love bisync i love bisync i love bisync i love bisync
This testing took close to 2 hours 😄

Plugin Testing Report

Installed Plugins

  • Decky Ludusavi - 1.0.0
  • Decky Cloud Save - 1.4.2-pre1
  • Web Browser - 1.4.0-846761d
  • Game Sync - 1.0.0-b2cf955
  • ProtonDB Badges - 1.0.12-1
  • HLTB for Deck - 1.4.2
  • Emuchievements - 2.0.3
  • SteamGridDb - 1.4.0
  • vibrantDeck - 2.0.1
  • PowerTools - 2.0.2
  • Steam Deck HQ - 1.0-0338b3d
  • Junk-Store - 1.1.9-d7a47dc
  • IsThereAnyDeal for Deck - 1.0.3-b251a93

Specifications

  • SteamOS 3.6.22 (Stable)
  • Steam 1741737356 (Stable)
  • Decky 3.1.3 (Stable)

Issues

Has the following major blocking issue(s):

Cloud Provider
[1] Immediately after installation, you only have the option to select Cloud Provider. All buttons open store.steampowered.com. Maybe it is a clash with the browser plugin, please double check.

Bisync Initial load
[4] Bisync does not create a folder on drive remote? There are some steps needed to be taken to be able to upload anything.

  1. (for reproduction), change Sync Destination
  2. Click Start Global Sync
  3. Fail, as it will say "no prior roots, please resync"
  4. Click resync and fail since "destination folder does not exist".

Seems like its an issue with bisync specifically.

chtimes
[7] I got chtimes /home: operation not permitted. Did we ever fix this? This happened when I tried to pull changes by testing the "sync on start feature". This effectively hardlocked me out of testing anything more on the plugin.

Filters getting swallowed
[9] Switching between Target Filter and Shared Filter tabs, does not rerender the components, which leaves the old selection in place. Is an issue in case someone clicks save button. Technically avoidable by first clicking to "Sync Logs" tab.

Has the following minor non-blocking issue(s):

Confusing Documentation
[2] If you are not familiar with rclone config creation, current documentation and helpful tips are very lacking. Took me a while to get the config file and to the correct place. Also documentation paths are with incorrect casing (~/homebrew/settings/sdh-gamesync/ vs ~/homebrew/settings/SDH-GameSync/. While it does work, I highly suggest revisiting this to make it more convenient. If [1] is fixed, maybe less of an issue.

Advanced tab. "Sync Root" I was worried it will start a sync process, but it just opens a page. Then I see a familiar sight. Renaming this to "Configure Sync Root" might give good context here 😄. Same for Sync destination, but might not have enough space for it.

Screenshot system
[3] Screenshot upload system shows that upload fails, even though it clearly shows up on my drive.
20250323175016_1
20250323173842_1

Interestingly, the error only shows when "delete copy" is unselected.

Cosmetic
[5] Cloud Provider page, maybe has bottom of clouds cut off? They look very janky, but overall a non-issue.

Auto-Sync
[6] It appears that autosync on start is not suspending the game, which is different from base plugin. Maybe intentional.

Target Filter Creation
[8] Paste Filter (Replace) feature seems to just delete filter, and not actually replace anything, maybe some duplication checking? Feels confusing to use at least? Other buttons work great!

Summary

Features Checklist
List of features I checked, mostly to keep track of this massive plugin 😄

Installation.✔️
Cloud Provider Selection. ❌ See [1].
Manual Provider Installation. ⚠️ See [2].
Plugin Logs ✔️. Logs show, logs reload works perfect.
Upload Screenshots ⚠️. See [3].
Upload Screenshots. Destination ✔️
Global Filter Creation ⚠️. See [8].
Shared Filter Creation ❌. See [9].
Global Sync. ❌. See [4].
Global Sync. Sync on Stop. ✔️
Global Sync. Sync on Start. ⚠️. See [6].
Global Sync. Upload. ✔️
Global Sync. Download. ⚠️. See [7].
Sync Root Selection ✔️
Sync Destination Selection ✔️
Strict Game Sync

Other Suggestions:
The rclone.conf location was designed to isolate the rclone, which is one of decisions I kinda regret with DCS. I would move it to ~/.config/rclone/rclone.conf as that is the default, and if we use a solid section name like "sdh-gamesync", then it would almost surely not clash.

Closing Remarks
I really spent last 2 hours on this quick 15 minute adventure of testing this plugin 😄. For most part, I have to say, amazing job! I really love the sync logs UI, the helpful modals, the amazing sync filter! And most importantly, OMG I LOVE YOU FOR THE SCREENSHOT UPLOAD FEATURE!!!!
Most of the issues here are minor, and some are plaguing the base plugin too. Not too sure what to do about it. So I decided to leave a testing report of ALL issues I was able to find, and we can go from there on what needs fixing.

@AkazaRenn
Copy link
Author

AkazaRenn commented Mar 23, 2025

@GedasFX OMG I cannot thank you more for the thorough test! I'm going through all the problems you pointed out and trying to fix them.

[1] Immediately after installation, you only have the option to select Cloud Provider. All buttons open store.steampowered.com. Maybe it is a clash with the browser plugin, please double check

It's intentional to block all other buttons before a cloud provider has been configured, since you cannot do anything without that. As for the problem navigating to Steam instead, I've made it too strict finding the url from rclone stdout and a recent update on rclone adding a new line before the url broke everything, will fix in the next iteration.

[2] If you are not familiar with rclone config creation, current documentation and helpful tips are very lacking. Took me a while to get the config file and to the correct place. Also documentation paths are with incorrect casing (~/homebrew/settings/sdh-gamesync/ vs ~/homebrew/settings/SDH-GameSync/. While it does work, I highly suggest revisiting this to make it more convenient. If [1] is fixed, maybe less of an issue.

I'll leave it there for now since [1] will be fixed. Documentation will be updated for sure.

[3] Screenshot upload system shows that upload fails, even though it clearly shows up on my drive.

My mistake, mixed two conditions in one check, will fix in the next iteration.

[4] Bisync does not create a folder on drive remote? There are some steps needed to be taken to be able to upload anything.

Yeah that's a part that I couldn't figure out, rclone doesn't seem willing to create a destination by itself even when it's resyncing. Did you managed to solve it in decky-cloud-save? If not I may add a toast telling user to do that manually. Maybe I'll just leave it there since it's inside Advanced Options.

[5] Cloud Provider page, maybe has bottom of clouds cut off? They look very janky, but overall a non-issue.

It looks good on my side, shouldn't have issue because I mostly copy-pasted your code 😂

[6] It appears that autosync on start is not suspending the game, which is different from base plugin. Maybe intentional.

Yes it's intentional, now only game syncs (rclone copy or sync) will block the game as a way to improve the game launching time.

[7] I got chtimes /home: operation not permitted. Did we ever fix this? This happened when I tried to pull changes by testing the "sync on start feature". This effectively hardlocked me out of testing anything more on the plugin.

Ah damn totally forgot that. I'll add "--no-update-dir-modtime" to the default config hopefully that can solve the issue. Will fix in the next iteration.

[8] Paste Filter (Replace) feature seems to just delete filter, and not actually replace anything, maybe some duplication checking? Feels confusing to use at least? Other buttons work great!

It works well on my side, did you forget to save the change? The clipboard expires in 5 minutes so make sure you paste before it expires, and once pasted the filters list (GUI) should update, then click save to get it updated to the file.

[9] Switching between Target Filter and Shared Filter tabs, does not rerender the components, which leaves the old selection in place. Is an issue in case someone clicks save button. Technically avoidable by first clicking to "Sync Logs" tab.

I had a hard time solving that either so I finally decided to put the logs page between target filters page and shared filters page, so at least on controllers you won't have that issue 😂

Advanced tab. "Sync Root" I was worried it will start a sync process, but it just opens a page. Then I see a familiar sight. Renaming this to "Configure Sync Root" might give good context here 😄. Same for Sync destination, but might not have enough space for it.

They will be renamed to Set Sync Root and Set Sync Destination.

The rclone.conf location was designed to isolate the rclone, which is one of decisions I kinda regret with DCS. I would move it to ~/.config/rclone/rclone.conf as that is the default, and if we use a solid section name like "sdh-gamesync", then it would almost surely not clash.

I actually like keeping the config in the plugin settings folder because I'm the person who syncs all plugin configs to the cloud for backup and it worked pretty well. I'll leave it like this for now 😉

@AkazaRenn
Copy link
Author

AkazaRenn commented Mar 23, 2025

Updated with fixes in 859a91b

@GedasFX
Copy link
Contributor

GedasFX commented Mar 27, 2025

After I installed the plugin, the plugins stopped loading, but after a restart everything showed up normal... I will call this a fluke as been a while since i opened my deck again 😄

[1] Fixed ✅
[2] Not important for this release, will. call it resolved ✔️
[3] Fixed ✅
[4] Hmm, on my plugin it was not really an issue as bisync is opt-in, and usually users sync at least once before that. bisync copy does create an folder on remote. Bisync seems to not do that. Maybe can like try copy nothing initially? Can also catch exception and try doing initialize function: first copy, then run resync. This should avoid us all initial confusion.
[5] What issue, we have no issue 😄 ✔️
[6] Noted. ✅
[7] After adding the configuration option, I can't reproduce issue anymore, so I guess we are good! ✅
[8] Its really hard to explain, but here are my repro steps:

  1. select a row and click copy
  2. append new row
  3. select the new row (bottom)
  4. click "paste filter"
  5. observe the row getting deleted
    I will try fixing that because... 👇
    [9] I think I know what the issue is, will see if i can fix that.

The other 2 comments: ✅ and, epic! ✅

New stuff:
[10] Sync Failed toast opens the Target Filter page, which is a minor inconvenience. Did confuse me for a bit as to where I was 😄.

Some other thought I just got:
I find it kind of funny that screenshots upload destination is a big button, but sync destination is hidden under advanced settings 😄

@GedasFX
Copy link
Contributor

GedasFX commented Mar 27, 2025

So it turns out I cannot fork a fork, but just do this change, will fix it:
image

Because your GetFiltersFunction changes, it will rerun the effect, which will refetch the data.

@AkazaRenn
Copy link
Author

AkazaRenn commented Mar 27, 2025

@GedasFX
[4] I think I'll try to add a toast offering user to create the destination folder once they have setup a cloud provider, that should serve the purpose.
[8] copy/paste issue, I still cannot reproduce somehow...
[10] yeah I'm not sure if there's a way to set a different entry tab but couldn't figure out how

As for the screenshot destination, because screenshot is using copy and only syncs one way from local to cloud, it's harmless to change the destination, not like those involves downloading and even file deletions.

BTW I've figured out the filter tabs, turned out I really just need to create a wrap up function for shared filters.

Addressed everything but [10] in the latest commit db1f677

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: 👀 In review
Development

Successfully merging this pull request may close these issues.

4 participants