Feedback on AssetServer for working with packed assets

[recatek]

I've found myself having to bypass the AssetServer entirely for working with packed assets for a few reasons, and I'd like to give feedback as to why and propose some changes as to what could make a future PackedAssetServer (or something similar) work better. In general, the AssetServer is built around accessing loose raw files by name and path, which breaks almost immediately when dealing with packed assets, even when providing a custom AssetSource.

For context, these are some of the design goals for my personal implementation of asset packing:

Pack Composition. Assets can be distributed arbitrarily across multiple packs and composed later. Packs may be divided by file type, or they could be divided semantically (e.g. assets for a particular DLC). It is not necessary to know what source pack file an asset is from in order to load that asset.

Pack Compression/Encryption. The entire pack file is compressed. This is fairly straightforward. This doesn't clash with the AssetServer/AssetLoader system since you can have your AssetLoader decompress the raw bytes just fine. Packs can also be encrypted for niche cases, which follows the same flow as decompression.

Hashed Paths Only. Packs contain no string paths, only a u32 or u64 hash ID for that file. I use fnv1a because of the const-fnv1a-hash crate that allows me to hardcode hashes in the binary without leaking names. The ID is a hash of the loose file path of the source file. These hashes can be salted to help confound datamining.

Asset Pack vs. Asset Source Directory. Loose files live in an asset source directory separate from the canonical Bevy assets directory. An offline tool crawls the asset source directory and builds the packs that then live in the assets directory. A game binary then ships with the packed assets in the assets directory, and no loose files.

Loose File Dev Mode. The pack system can be disabled and transparently switched to load loose files at runtime for dev iteration. This is gated behind a feature flag and this functionality is stripped from retail builds. In loose loading mode, the asset loader crawls the source directory and conjures a pseudo-pack of all those files that then goes through the normal hash-access lookup approach.

I think these are relatively standard among asset packing approaches in other engines, both public and proprietary. It is unusual to ship loose files with a game in a professional engine. As a note, I'm aware of bevy_asset_packer, and it has some similar goals, but it uses a single pack instead of multiple merged packs, and also uses memory mapping which isn't always portable (or safe). It also has no loose file mode.

---

The AssetServer makes it difficult to accomplish this goal currently, due to having a few assumptions:

• The AssetServer is very string-path-based. Nested files for example are addressed entirely by string (e.g. path/to/file.ext#nested). If instead there was an API that allows you to provide two hashes, one for the file path, and an optional nested file ID hash, it would alleviate this problem.
• The AssetSource/AssetReader assumes "path in, bytes out". This doesn't work for conjuring fake packs to support loose file loading. In general this flow is somewhat inverted. By the time I've walked and merged all of the assets in my packs, I'd might as well have loaded them, since I need to open a pack to know what's in it. There isn't enough mutable state stored in AssetSources to build and store a pack manifest.
• Not specific to this problem, but there's just a lot of confusing machinery here. I find it easier in general to spin up an IoTask, use async-fs to open files, process them myself inline, and dump them into ResMut<Assets<T>> handles than to do all of the work to make a custom AssetSource, AssetReader, AssetLoader, etc.

I'm not sure if the best approach is an alternate asset server, but it does seem appealing to have two of them. A LooseAssetServer or BasicAssetServer for dev or small games where this isn't an issue, and a PackedAssetServer for larger shipping builds. If the two were API-interchangeable for loading files, you could make the statement that the PackedAssetServer doesn't support things like asset processing or content watching/hot reloading.

[andriyDev]

I'm not sure if the best approach is an alternate asset server, but it does seem appealing to have two of them.

I definitely want us to support alternative asset servers (even though I don't think it's necessary here, see below. Unfortunately today our asset system and the asset server are somewhat coupled - namely asset handles store a bunch of stuff that is only relevant to the asset server. I would love it if we separated the AssetServer and configured it out.

---

Pretty much all of this can be done as an asset source. I don't think this needs to be a new AssetServer. An AssetReader can internally store the pack manifest as a RwLock<Option<PackManifest>>. If it's not there, load the pack manifest. After that, use the pack manifest to find the bytes. An AssetReader can also provide a decompressing reader.

Hashed Paths Only. Packs contain no string paths, only a u32 or u64 hash ID for that file.

There are two problems with this:

1. How do users reference an asset? What do they put in their "load" call? Do they put the u64 hash ID? How do users get this ID in their code? Do they need to hash their asset in the CLI just to access their asset? That seems very cumbersome.
2. If you switch from "loose file" mode to the "packed" mode, you now need to change how all your assets get loaded since you can no longer load paths. This is a big disadvantage, especially if you miss things. Or maybe the loose file mode also needs to load by hash ID? But then we're back to the first question.

than to do all of the work to make a custom AssetSource, AssetReader, AssetLoader, etc.

There's a reason for all these pieces. For example, the rendering folks have been loving the web asset source. I think you'd have to pry it from their cold, dead hands if you wanted to remove it. If this whole system already supports packed assets, then that's a good thing, and AFAIK it does.

It is not necessary to know what source pack file an asset is from in order to load that asset.

You can do this with another asset source: make a new CombinedAssetSource that just reads all the pack manifests, and then it looks up which pack to access when you try to read a path from it. Problem solved. Frankly I don't care about this though, since I think it's much more intuitive to say that each pack is a single source? It seems likely that your paths do have some meaning behind them, whether that be by type or by level or whatever - It seems a little unnecessary to hide that fact from callers.

Nested files for example are addressed entirely by string (e.g. path/to/file.ext#nested).

These are not "nested files". These are subassets - assets that live in the same file as another asset. Subassets suck for a lot of reasons, but they are also kinda necessary: glTF files store a bunch of assets all in one file. It is difficult to just load one of these subassets: here's an issue for that #21641.

---

I've tried starting a zip asset source, which would essentially be our "asset pack", but I ran into some blockers: #21641 (comment). As for composing these packs, we could also provide tooling for that. That seems fine.

[andriyDev]

You can give it a const new function that that takes in a path string

Ahh thanks for the explanation. I understand this now.

you'll need to build a hash manifest of everything first

I think this means that your file needs to be able to store these hashes (doesn't work for glTF well since it stores paths), or your loader needs to translate file paths into hashes, which means your hash manifest needs to store file paths anyway?

It also doesn't support the source-pack-agnostic path workflow AFAICT.

As I said above, we can totally make an asset source that does this. In your AssetReader when you read the pack manifests, you merge them such that you can lookup which pack to read based on the path. So pack1 could say it stores a.png, b.gltf, c.ron, and pack2 could say it stores d.txt, e.jpeg. Your merged manifest will look like a.png -> (pack1, offset=10), ...

it seems far easier to just load everything once and stick it in a HashMap<DataHashId, Handle>

The current asset system is intended for fine-grained loading: you load the things you need and those are kept in memory until you aren't using them. The approach you're proposing means you can only load/unload entire packs at a time - which also means there will be longer delays in loading since you have to load all the assets.

The main reason I don't use the AssetSource route is that it seems like it double-taps the files.

Yes, you'll need to do one read to read the pack manifest, and then another read to read the actual data. Except you can store the results of that pack manifest read in the AssetReader so you only do it once. Other than that, we only need one read per load: there's no other "double-taps". If you have something like a glTF, then loading it will parse all its data and create the appropriate subassets (this all belongs to one file read), then will invoke another load for textures it references (since that's a separate file).

[recatek]

Expanding the current Path based interface to take a more general DataUri or something similar, which can potentially be a hash, would be useful here.

With respect to customizing the AssetReader to first read a manifest and then resolve that into identifying paths, I think I see how it would be done, but it requires some outstanding features from your PR linked above. As you mention there, you ideally want to be able to add an AssetSource/AssetReader after App creation so that you can incorporate this into a loading screen phase. It's still challenging in the context of the current system because...

The current asset system is intended for fine-grained loading: you load the things you need and those are kept in memory until you aren't using them. The approach you're proposing means you can only load/unload entire packs at a time - which also means there will be longer delays in loading since you have to load all the assets.

This is why I was leaning towards a different AssetServer altogether, and why you would want the file loading process to not be sensitive to the source pack that a given file is in. A smart packer could analyze the file dependency chains and group files dynamically in terms of, say, assets used only by a specific level/zone vs. assets shared between multiple zones. Or it could duplicate files strategically to prevent unnecessary whole-pack loads for only one or two files.

[andriyDev]

Let me clarify something: the fact that we use the Path type is completely wrong. We want to eventually provide our own path type which is ~just a string (though for the common user we still want to support path-like joins. But yes ideally providing a hash in the asset "path" should be supported, and today it's annoying.

why you would want the file loading process to not be sensitive to the source pack that a given file is in

I feel like if you want this, it's unlikely that you want assets to be dynamically loaded at all? As in, it seems like you should just make loading packs explicit rather than looking them up? Otherwise loading one asset will also load a bunch of other junk you may not want (e.g., everything in that level).

If that's what you want, another way to implement this pack scheme is to create one big PackLoader (no longer an asset source/reader). Users need to manually load a pack by saying "load the pack at this file path" (though you can use AssetReader to load everything in a directory). The loader will just load all the assets (using NestedLoader<DynamicTyped, Immedate>) in the pack and then add them as subassets (where the subasset label is the your DataUri). Then you create a resource which maps your IDs to their handle, and you populate that resource whenever your pack assets are loaded.

My point here is to say I think we have all the pieces here to implement what you want on top of the existing asset system. The existing asset system certainly has some complexity, but we support a lot of different use cases. Deleting it all and replacing it with a much less capable system isn't practical.

Again, if you want your own asset server, I think we should be supporting that. Mainly that involves storing some type-erased state in the handle I think, and then configuring things out? I'm not sure... There's work to do there. I mean there's a lot of work to do clearly lol, between supporting adding/removing asset sources, zip sources, or disabling the AssetServer entirely. But we're working towards these!

[recatek]

Is there a long term plan for Bevy to do packaged assets the way Unity/Unreal/Godot/etc. do?

[andriyDev]

Yes IMO published games should not be using loose files like we have today. Personally I think zip files will be sufficient as "packs" in the medium term.