Feature Request: On-Demand Files

[IncPlusPlus]

Currently on-demand file downloading is not implemented. #105 was opened and closed due to inactivity. While I'd love to contribute this and make a PR instead of a feature request, this is a bit too daunting of a task for me. When "download files on-demand" is checked in the client on Windows, the file behavior is as follows.

• All files and folders in OneDrive exist in the local OneDrive folder but are empty. There is some sort of hook (described later) that downloads the file before it is opened or otherwise used by any program
• If a file is to be opened or otherwise used (such as copying or moving) by any program and it is not yet downloaded, it will be downloaded before it is used. It will be downloaded in-place (meaning it is not stored in a temporary location, to my knowledge). After it has been downloaded like this, the file stays downloaded and can be viewed/modified as many times as you like up until "Free up space" is clicked
• Files and folders in the OneDrive folder can be right clicked and and one of the right click options is "Always keep on this device". What this does is keep those files/folders stored locally instead of waiting for them to be used as described in the first bullet point
• (This one could be a stretch goal for a later addition) When right clicking on a folder in OneDrive, there is an option listed to "Free up space". What this does is removes any files that were downloaded (with the exception of files/folders which have "Always keep on this device" checked off)

[abraunegg]

@IncPlusPlus

Currently on-demand file downloading is not implemented

This is because the OneDrive API does not provide any capability to advertise what file has been tagged via OneDrive as 'on-demand' - thus there is zero mechanism to determine which file has this flag or does not.

For each OneDrive drive item the following is exposed: https://docs.microsoft.com/en-us/onedrive/developer/rest-api/resources/driveitem?view=odsp-graph-online

Until a driveItem property exists that represents that an item is 'on-demand', this cannot be supported by this client.

• If a file is to be opened or otherwise used (such as copying or moving) by any program and it is not yet downloaded, it will be downloaded before it is used. It will be downloaded in-place (meaning it is not stored in a temporary location, to my knowledge). After it has been downloaded like this, the file stays downloaded and can be viewed/modified as many times as you like up until "Free up space" is clicked

As this program is CLI based, to do this, your talking about a whole lot of GUI development to add this sort of functionality to a desktop manager (gnome/kde etc)

• (This one could be a stretch goal for a later addition) When right clicking on a folder in OneDrive, there is an option listed to "Free up space". What this does is removes any files that were downloaded (with the exception of files/folders which have "Always keep on this device" checked off)

By using the OneDrive API, there is no concept of 'a device' - thus, no way to track that this 'instance' is the instance where the file must always be kept. As per above, there is no driveItem resource property that represents even this option.

In short, the OneDrive API is missing this:

• a driveItem property to handle 'on-demand'
• a driveItem property to handle 'Always keep on this device'
• The Windows OneDrive client integrates with Windows & the NTFS file system to provide some additional capabilities - which all the graphical desktop managers & Unix file systems currently do not with this client.

As per #105, it might be worth asking the question here: https://github.com/OneDrive/onedrive-api-docs to see if there is any plans to support the missing driveItem properties that would potentially allow some sort of development in this area. Without these properties being exposed, this is a non-starter.

[IncPlusPlus]

This is because the OneDrive API does not provide any capability to advertise what file has been tagged via OneDrive as 'on-demand' - thus there is zero mechanism to determine which file has this flag or does not.

Like what was mentioned on #105, no API needs to exist for the 'on-demand' flag. It is more of a state of the program than a per-file flag. As for the 'Always keep on this device' flag, these are stored and acted on locally. There is no need to access any API to know which files to always store a copy of.

• ... After it has been downloaded like this, the file stays downloaded and can be viewed/modified as many times as you like up until "Free up space" is clicked

As this program is CLI based, to do this, your talking about a whole lot of GUI development to add this sort of functionality to a desktop manager (gnome/kde etc)

That's a fair critique. To avoid needing any GUI development. --free-up-space could be a switch for the onedrive command.

• (This one could be a stretch goal for a later addition) When right clicking on a folder in OneDrive, there is an option listed to "Free up space". What this does is removes any files that were downloaded (with the exception of files/folders which have "Always keep on this device" checked off)

By using the OneDrive API, there is no concept of 'a device' - thus, no way to track that this 'instance' is the instance where the file must always be kept. As per above, there is no driveItem resource property that represents even this option.

When managing driveItem instances, would it be possible to know which files should always be kept locally by matching the pattern with the contents of a keep_local_list similar to the sync_list?

In short, the OneDrive API is missing this:

• a driveItem property to handle 'on-demand'
• a driveItem property to handle 'Always keep on this device'
• The Windows OneDrive client integrates with Windows & the NTFS file system to provide some additional capabilities - which all the graphical desktop managers & Unix file systems currently do not with this client.

To summarize: Again, syncing files only on-demand is a property not of files or folders but of the local installation of OneDrive itself. Additionally, the Always keep on this device flag is something that is kept track of by the local installation of OneDrive. Since implementing the Always keep on this device flag through a GUI is impractical, would it be practical to instead use a keep_local_list similar to the sync_list? As for the integrations/hooks into Windows and the NTFS file system, I'm sure we could come up with similarly functioning workarounds.

[abraunegg]

@IncPlusPlus
OK .. so lets say this 'works' - you have the CLI checking files on OneDrive, and, if say a client feature flag is set, it will not download the file, but create a zero byte file + all folder structures so that everything exists locally.

You open 'vi' or OpenOffice or whatever in your GUI, or CLI ... how does the GUI or application or CLI now tell the onedrive client to go download the file (as this is now on demand) ? What if you are 'offline' with no working Internet access?

There would have to be major integrations into a number of places to make this work / simulate the Microsoft client on Windows.

[IncPlusPlus]

@IncPlusPlus
OK .. so lets say this 'works' - you have the CLI checking files on OneDrive, and, if say a client feature flag is set, it will not download the file, but create a zero byte file + all folder structures so that everything exists locally.

Correct. That's what the Windows client usually does.

You open 'vi' or OpenOffice or whatever in your GUI, or CLI ... how does the GUI or application or CLI now tell the onedrive client to go download the file (as this is now on demand) ?

That's the trouble... That's likely one of the NTFS/Windows hooks that OneDrive likely utilizes to perform this. I can help look into whether such a filesystem hook is possible with UNIX/Linux.

What if you are 'offline' with no working Internet access?

That's the purpose of the 'Always keep on this device' feature. It allows you to keep only the files you need around such that you have what you need when you're offline/on the go.

There would have to be major integrations into a number of places to make this work / simulate the Microsoft client on Windows.

Could you elaborate a bit?

[abraunegg]

@IncPlusPlus

Could you elaborate a bit?

I have not looked into it, but to recognise some sort of application FS access request would in my mind require some sort of kernel level FS hook.

Best for you to ask on the kernel developers mailing list for some pointers.

At the moment, I wont be looking into this any futher than what I already have .. back to beers & bbq & holiday for me.

[IncPlusPlus]

Thanks for hearing me out. Sounds like a plan. I'll look into this as I can and post updates here (if any). While I do research for this, would you mind keeping this issue open?

Happy holidays,
Ryan

[pdvrieze]

@abraunegg The approach I would see here would be to use FUSE interface (probably through libfuse - https://github.com/libfuse/libfuse). In that context the most straightforward approach would be to approach things as a caching solution - not unlike what the Android client does. So in Linux you don't even need empty files at all. Obviously this could be enhanced by allowing "pinning" of files to keep them from being evicted from the cache.