WIP: Initial Xet docs (incomplete)

[rajatarya]

This is the PR for Xet Storage documentation. This PR currently has partial content, so not ready for review yet.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[jsulz]

@rajatarya @ylow just did an update of these docs. Few things to note:

1. I did a structural update of headings so that the in-page navigation on the left sidebar would be easier. I did not love how that looked on https://moon-ci-docs.huggingface.co/docs/hub/pr_1622/en/repositories-storage
2. Moved up Deduplication section to be directly under Xet and futzed around with language to make that flow better
3. I updated the Using Xet Storage and all subsections (Recommendations and Current Limitations) with a first pass on the content there. I'm not particularly attached to any of it, but wanted to take a stab based on talking with Rajat to know what we wanted there.

@rajatarya I'm going to work in the screenshots of .gitattributes and pointer files next, along with the sequence diagrams next.

@ylow would you mind taking a look so far? I think we could use some help on the Security Model section. Also, any other feedback that you want to provide would be great ❤️

[jsulz]

@julien-c we've (@rajatarya @ylow and myself) have been looking at these docs are thinking about a larger rewrite where we have a single entry point for all content related to storage with an emphasis on Xet as the default and LFS as legacy.

The thinking being that from a reader's perspective it's somewhat odd to have a Storage Limits section and then a Xet-centric section about the Hub's new storage system.

This would mean the current Storage Limits documentation would move and merge with these docs.

Storage would reside under Repositories (just like Storage Limits does today), but would have subsections like the following:

• Storage
  • Usage
  • Recommendations and Limits
  • Plans
  • FAQ

(Xet architecture content would be moved to https://github.com/huggingface/xet-core for open source contributors)

The order and exact content on each page would be determined through this PR and with review from everyone that has an interest in the current Storage Limits documentation.

Does this sound reasonable to you, or should we continue to keep Xet documentation separate as we are in this PR?

[julien-c]

(@jsulz i'm stuck on something else today but will look tomorrow)

[julien-c]

@jsulz sorry about long response time.

In my opinion the most natural progression would keep to keep this doc page separate but put it right before the storage limits one, like so:

Storage limits are really more about pricing, and we will update that one as well to add mentions of Xet and not just LFS (in a subsequent PR)

Keeping this Storage Backends page separate will also make reviews etc more tractable.

EDIT: Alternatively, it could also be named just "Xet"...

[julien-c]

looks like a good start 🔥 let's iterate and ship a first version of this soon!

[jsulz]

@rajatarya just did a pass based on all of the feedback so far.

Only thing remaining is the Security Model section (which we could maybe merge without if we wanted and then iterate on later).

[ylow]

I will write the security model section tomorrow morning.

[julien-c]

lgtm (after the filename change)

[hanouticelina]

LGTM!

[jsulz]

[Wauplin]

🔥