Akira File Format

[Alecaddd]

This is a behemoth and we will probably split this issue into multiple sub-tasks, but let's start collecting our approach here.

• Save and Open design file with .akira format.
• The file is a zip archive with an svg file and a hidden .git repository
• The file needs to generate a thumbnail to preview the currently saved design
• At every save triggered command, a new commit should be pushed to the local git repo
• Ability to visualize the commit history and revert back to a previously saved state

I'm still not sure if we should deal with undos the same way, or keep a limited amount of undos in memory as any other application.

[e200]

Did you think about use .akr as extension too? I think that how more short, more better.

Does this mean that Akira will have a dependency on git being installed on the system? Or do you intend to ship a lightweight git implementation with Akira and use that?

[albfan]

@terndrup in that case, we will use libgit2 so no need for git to be installed

[julientaq]

if there is some git thing running behind as a save you may end up with very heavy file (especially with one svg with everything inside). Maybe having a keep this version action that the user can trigger would be more efficient that saving every change with git. but i see where you're going 👍

Also, the idea of having git behind could be pretty useful to merge (got already some UX idea about how it could be done :D) or to fork a project. be sure that these options don’t get forbidden by your git implementations please :D.

[Alecaddd]

@julientaq since we're managing everything via SVG, which is basically just an XML string, we should be able to keep the file size quite low.
We'll need to test it anyway and see if it gets out of hand.
An option to delete older versions is being considered, also, to clear the git history and free up some space.

If you want to contribute with some design ideas, we'll need a popup window where you can scroll through the commits (probably named with the date and time) and the preview image for that commit.
A sort of time machine like experience in MacOs.

[snowparrot]

You should consider other free alternatives to zip as the algorithm is very outdated (from the 80s)

A good alternative:
https://en.wikipedia.org/wiki/7z

Also free and open.

[Jab2870]

Despite the primary format being SVGs, presumably bitmap images are going to be supported in Krita? If so, these will make the file size big, GIT was never really designed to manage binary files. One option might be to look at GIT LFS although I don't know how it would work with libgit2.

Another thought, if this is basically a git repo, would it be possible to manage it manually? If, as well as a .akira zip file, I would love to be able to open it from an uncompressed directory that I manually manage with git.

Lastly, will you facilitate the ability to push the git repository to a remote repository? Obviousl, if we are able to facilitate managing the repo manually, this would be addressed by that.

[jhrdina]

😄 If you really want to be future-proof and maybe want to support real-time online collaboration in a distant future (with independent offline edits and seamless conflict resolution, without a necessary need for central server), you might want to consider designing your JSON files and document representation as CRDTs (conflict-free replicated data types) and replace Git this way.

CRDTs naturally keep track of document state history, but at the same time allow you to specify automatic merging semantics for different subtrees of the JSON for the case of concurrent edits (Example for a list of shapes: You have a rectangle A, one offline user removes it, second offline user changes its dimensions. What happens after online sync? You can specify e.g. that list of shapes should have UPDATE WINS semantics, so that the rectangle gets "unremoved" and the dimensions change (update op.) doesn't get lost.) CRDTs design ensures that after sync everybody ends up with the same document state (usually formally proven).

There are interesting implementations in JavaScript, e.g.:

• based on δ-CRDTs (delta state CRDTs, a popular variant of CRDTs): ipfs-shipyard/js-delta-crdts
• or some other approach (here with predefined merging semantics, but on the other hand with infinite nesting support - not sure you really need this): automerge/automerge

I'm not sure how many native CRDTs implementations are there and yes, reimplementing custom library would require some time...

The thing is (in my opinion) it might be hard to add collaboration support into an offline-only app down-the-line, when your data-structures and file-format are already defined and are not designed for collaboration. (See Sketch - it still doesn't support real-time collaboration, does it? I don't have a macOS on hand, sorry...)

I hope I didn't disturb you too much! 😄 You know... I had to mention CRDTs, it is the new medicine that cures every problem, right? 😄 I really keep my fingers crossed for your project, I absolutely understand that you have some limited resources, but still, at least for the record, I had to mention this technology. 🙂

[gdelazzari]

The main issue I would have with the .akira file being a ZIP archive (or whatever kind of non-text file) is that versioning that would be an issue. I mean it's not that it won't work, but I usually make one Git repository for a project (unless it's a really big one) and have everything in there, from the (Akira) design files to notes to code, documentation, etc... It doesn't make any sense for such a use case to have the design file being an archived git repo inside another git repository. A lot of space would be wasted since the outer repo wouldn't be able to manage the internal one efficiently. This gets worse with every Akira file that is inside (I can see having multiple designs on a single repo).

I'm not sure what the best option would be.

If a user wants to store the Akira file inside a repository, the file should just represent the design with a text format (even SVG is totally fine). For users which won't store the file inside a repo, then something like what @jhrdina described seems really cool. Not sure about having that inside a repo, history would be duplicated anyway...

This doesn't seem to be a straightforward issue to solve. Assuming that my use case is shared by other people of course (i.e. if I'm the only one then who cares 😄 ).

The best solution I can think of is to have 2 formats ("raw" SVG without versioning and something specific with versioning) and make the user choose or choose automatically (i.e. when saving for the first time, traverse the directories and if a VCS repository is found then save in SVG (after asking the user for confirmation) otherwise use the versioned format.

Just sharing my two cents, hope it's appreciated

[Alecaddd]

Thank you all for your insightful suggestions and point of views.
We started working on this implementation and you can take a look at the initial work done on this PR: #110

@jhrdina you're not disturbing us at all, on the contrary, we need these suggestions since day 1 in order to avoid conflicts or technical issues while trying to implement new features in the future.
Would you be able to prototype a quick example, even not fully functional, to adopt CRDTs on the PR made by @Philip-Scott?

@gdelazzari thanks for your suggestion as well.
I personally don't see the problem as the .akira file is meant to be a standalone GIT repo itself to keep the full history of your design while shared across computers and users.

The idea here is to offer built-in version control for designers who have no idea what GIT is.
It's great that you know how to set up a repo and use your own version control system, but the majority of the users don't.

Offering a version control already into one single file out of the box is convenient and removes any technical barrier. The user can upload that single file on a cloud storage for backup, knowing that one file contains the full history and versions of the project.

You can save that file on a remote GIT repo as well if you want, but that seems quiet an overkill as you won't need to keep a history of versions of that file.

If in your case you decide to not use the built-in version control of Akira, you can simply export your design in SVG and set up your own repo.

[gdelazzari]

Thanks for the response and the clarification, I take for granted that you have a way better understanding of Akira's users needs than me, since I'm not a designer myself. In that case the .akira file format sounds good. CRDTs seem to be a really interesting thing though, hope you can do some testing with such a technology and evaluate them.

The user can upload that single file on a cloud storage for backup, knowing that one file contains the full history and versions of the project.

Not so relevant, but just to be complete ("mettere i puntini sulle i") it's also true that most cloud services already do version control by themselves. I don't think someone would want to disable Akira's built-in versioning because their favorite cloud storage already versions the files, but it's also to realize that (IMHO) we're going more and more into versioned file storage services/systems. Again, not so relevant, but still worth to consider in order to "see the thing in its entirety", I think.

Regarding the versioning inside the .akira files, I honestly don't think Git is a great idea. I mean, of course it's freaking battle-tested and everything, but is such a complex and full-featured system really needed? Also, how easy is it to make it manage the repository inside the archive? Does libgit provide user-definable hooks for opening, reading, writing, etc.. files so that they can be overridden with functions that operate inside the archive?

I think something ad-hoc could be implemented, since the file format has a relatively simple and known structure which should make it less harder.

CRDTs seem indeed the best solution IMHO, also being future-proof for (real-time) collaboration or whatever.

Again, just my two cents 🙂

[jhrdina]

Would you be able to prototype a quick example, even not fully functional, to adopt CRDTs on the PR made by @Philip-Scott?

I wish I had the time... unfortunately, I'm currently desperately busy with with my Master's Thesis and eventually finishing my Master's Degree. I don't think I will able to help with coding before the end of June. 😒

The general approach would be not to save the final document (graphical objects in your case), but rather single operations/deltas that lead to creation of the document (add rectangle to pos XY, move object with UUID af1b11234 to pos X'Y', remove object with UUID 197dff). These operations then get "replayed" on each file opening to get the current (or older) version of the document. Using these discrete operations, CRDTs can find out a lot about different user's actual intents and therefore can offer much better conflict resolution (in contrast to comparing two static finished documents).

I've found a C++ reference implementation of δ-CRDTs:
https://github.com/CBaquero/delta-enabled-crdts

The Automerge JS library mentioned above doesn't allow you to exactly specify the merging semantics, it does, on the other hand, offer some interesting advanced features - undo/redo support, relational JSON tables, optimized collaborative text editing... The underlying implementation might be a bit harder to port, because it uses JavaScript Proxies to "record" changes and also builds upon Immutable datatypes - these are great, as they enable structural sharing (you can have multiple snapshots of a document in time and share data that didn't change between snapshots) - the problem is I don't know any immutable data-structures implementation for Vala.

You can however play with the library in codesandbox, I've created, so that you can get the feeling of dev experience I would hope for. If you uncomment the last line, you can see in the console all the generated operations ready to be saved to file. You can also dig into CRDT internals using DevTools.

This is probably all I can do for you right now... :/

[Jab2870]

I think part of the problem might be mitigated if we have the ability not to "zip up" the file at the end. This would allow anyone with an existing understanding of git to use the akira directory as a sumodule rather than a binary file that cannot easily/efficiently be version controlled in part of a larger repository.