Delete DataFolder usage from analyzer engine

[Noggog]

Right now it's just used for getting fresh streams for the record frame analyzers. We should instead communicate the paths more directly so analyzers can be run in situations that dont have such a central Data folder concept.

[Noggog]

@Elscrux for this feature, how would you specify where all the mod files are when running the CLI? I can see this feature being useful for the API where code can specify the paths when making the call.. but the CLI seems to benefit from having a single folder to point to

[Elscrux]

I'd almost think this would be something going as far as core Mutagen concepts potentially. The IModListingGetter used in the load order or some more specialized interface would kind of make sense to include the full path I'd say.

Sorry, somewhat unrelated but one thing in this approach that I see right now is that the WithLoadOrder() call currently constructs a new link cache from the load order which you might not really want when you are working with a game environment anyway, where you already have a link cache.

Mutagen.Bethesda.Analyzers/Mutagen.Bethesda.Analyzers/AnalyzerRunnerBuilder.cs

Line 131 in 343c554

_linkCache = loadOrder.ToUntypedImmutableLinkCache();

Could be good to upgrade WithGameEnvironment() to use both link cache and load order directly?

Mutagen.Bethesda.Analyzers/Mutagen.Bethesda.Analyzers/AnalyzerRunnerBuilder.cs

Line 53 in 343c554

return WithLinkCache(gameEnvironment.LinkCache);

[Noggog]

I'd almost think this would be something going as far as core Mutagen concepts potentially. The IModListingGetter used in the load order or some more specialized interface would kind of make sense to include the full path I'd say.

I have been tinkering with a ModListingWithPath struct in the analyzer project that I could see bubbling up to mutagen proper at some point.

My question is less about that internal implementation, and more about the CLI's API for the user: How do you tell the CLI that it should analyze 12 different files from 12 different Mo2 folders, for ex?

I see right now is that the WithLoadOrder() call currently constructs a new link cache from the load order which you might not really want when you are working with a game environment anyway

If you're giving it a game env, it's using the env's LinkCache, which inherently (currently) has the LoadOrder. So that should be the same as passing in both.

If you're instead using the more direct WithLoadOrder call, we need to make a new LinkCache, as opening a call with both parameters would leave it vulnerable to a setup where the LO has items the LinkCache doesn't have, or vice versa. Imagine a LinkCache with A, B, C mods, but a LO passed with C trimmed off. That'd confuse the system. If we offered a variant taking in both, it would have to check and throw if things were out of sync like that

[Elscrux]

My question is less about that internal implementation, and more about the CLI's API for the user: How do you tell the CLI that it should analyze 12 different files from 12 different Mo2 folders, for ex?

Wouldn't the api be to pass in a load order that has these different paths? I'd assume Mutagen would have a way to construct this kind of game environment / load order. Probably as part of the GameEnvironmentBuilder where you can pass in individual mod paths. Currently I kind of hack this kind of behavior in with TransformModListings() and redirecting the mod listing to be pulled from the location I want but that's more of a hacky workaround.

If you're giving it a game env, it's using the env's LinkCache, which inherently (currently) has the LoadOrder. So that should be the same as passing in both.

No I totally get that but sorry that probably wasn't clear. My talk about load order/link cache stuff was just to motivate that we should probably upgrade the WithGameEnvironment() call to use both link cache and load order as they are definitely synced in there. So you can pass a game environment and keep the link cache and the load order without needing to recompute one.

[Noggog]

Wouldn't the api be to pass in a load order that has these different paths? I'd assume Mutagen would have a way to construct this kind of game environment / load order

At the moment, no it doesn't actually. Almost all the mechanisms are "what is the data folder?" + "What are the entries from that data folder you want to load?"

You can hack it like you've mentioned, but that might mess up some of the related logic. Stuff like locating strings will look in the data folder still for BSAs and loose strings files, etc

[Elscrux]

At the moment, no it doesn't actually. Almost all the mechanisms are "what is the data folder?" + "What are the entries from that data folder you want to load?"

Sorry I didn't express myself properly. I meant more it more like, "Mutagen would need to have to away to construct this kind of game environment". Do you think that could be a possibility?

What do you think about the WithGameEnvironment() things I reported?

[Noggog]

It would be kind of a can of worms, rather than a quick add. It would have downstream rammifications on lots of other tooling. For example, the StringsFolderOverlay systems that find strings files would need to be upgraded to not just look at a folder for strings for any mod, but instead have a separate lookup per mod

Before we did these types of changes, I'd want to ask what the goal of running in separate locations is. Is it just to avoid running through a tool like mo2 that brings all the mod content to one place? For example, xEdit itself i imagine would also have trouble working with systems not within one data folder. I think it's the more rare situation that tooling is trying to pick apart mo2 internals itself

[Elscrux]

It would be kind of a can of worms, rather than a quick add. It would have downstream rammifications on lots of other tooling. For example, the StringsFolderOverlay systems that find strings files would need to be upgraded to not just look at a folder for strings for any mod, but instead have a separate lookup per mod

Yeah it would be a bigger project for sure I'd assume. Also things using the data directory service would need to be checked if they are compatible with this idea.

Before we did these types of changes, I'd want to ask what the goal of running in separate locations is. Is it just to avoid running through a tool like mo2 that brings all the mod content to one place? For example, xEdit itself i imagine would also have trouble working with systems not within one data folder. I think it's the more rare situation that tooling is trying to pick apart mo2 internals itself

It's partly to avoid something like MO2 yeah, but it also opens up many new scenarios when you're not bound to how MO2 works. I initially had a problem with MO2 because being able to track file changes with a file system watcher doesn't work properly when run through MO2. This was just the first part where running via MO2 is just a limitation when it comes to realizing something like Creation Companion. It's also been a challenge because in MO2 of course you just have all files in one big data folder, but if you'd like to do specific operations on a mod level where you just look at the files of one mod that is not possible. I assume that this would be the most common use case for other people too, be able to reference one mod's assets directly.
I realize it's an extra abstraction layer, but it has really helped a lot for my use case at least and it's an essential part at least of the design of Creation Companion now and made a lot of stuff much easier. I can work with multiple DataSource objects and use DataSourceFileLink objects to reference a unique DataRelativePath in a certain DataSource.
I don't want to destroy the concepts Mutagen has been running on, but if there could maybe be an optional "mode" of how you could construct your load order to run on this non-central data folder, but a distributed data folder approach, that would be super helpful for me, and hopefully also other people in the future. I can also check this out in Mutagen myself if you want for sure, lmk!

[Noggog]

I initially had a problem with MO2 because being able to track file changes with a file system watcher doesn't work properly when run through MO2

Yeah, Synthesis been having lots of trouble with Mo2's VFS as well. Im hoping Nexus' new mod manager that doesn't use a VFS will be a better route. Can just work with the actual disk on the OS straight up again.

Yeah it would be a bigger project for sure I'd assume

the complexity depends on how deep you want to take it. Because, for example, if the analyzer suite just takes in a load order of mod objects... well then the data folder is not applicable really. The analyzer suite can just use the mod objects and be done with it.

It'd be up to the user to make sure those mod objects were constructed appropriately and had the correct data that was desired.

I dont think the analyzer suite itself has to walk that road, as it can just take the LO/LinkCache to be used straight up.. and leave it to the user to figure out how those were constructed.

What do you think about the WithGameEnvironment() things I reported?

I think that already exists?

Mutagen.Bethesda.Analyzers/Mutagen.Bethesda.Analyzers/AnalyzerRunnerBuilder.cs

Line 51 in 343c554

public AnalyzerRunnerBuilder WithGameEnvironment(IGameEnvironment gameEnvironment)

Was there a missing aspect on that call im missing that you wanted to discuss?

[Noggog]

Though I do wonder if the analyzer suite would be interested in the data folder mode, where it is more aware of the the actual concept of a data folder.

This might be used by analyzers that aren't focused on mutagen object records directly, but instead would throw topics on the data folder being malformed in some way for a user's install.

Perhaps that'd be via an optional WithDataFolder call that isn't required

[Elscrux]

What do you think about the WithGameEnvironment() things I reported?

I think that already exists?

Mutagen.Bethesda.Analyzers/Mutagen.Bethesda.Analyzers/AnalyzerRunnerBuilder.cs

Line 51 in 343c554
public AnalyzerRunnerBuilder WithGameEnvironment(IGameEnvironment gameEnvironment)

Was there a missing aspect on that call im missing that you wanted to discuss?

Do my previous explanations help at all? I'm not sure how I could explain it differently. The idea is to keep using the same load order when passing in the link cache of a game environment, and not having to construct a new one when we have the load order already in the game environment.

#507 (comment)

Could be good to upgrade WithGameEnvironment() to use both link cache and load order directly?

#507 (comment)

If you're giving it a game env, it's using the env's LinkCache, which inherently (currently) has the LoadOrder. So that should be the same as passing in both.

No I totally get that but sorry that probably wasn't clear. My talk about load order/link cache stuff was just to motivate that we should probably upgrade the WithGameEnvironment() call to use both link cache and load order as they are definitely synced in there. So you can pass a game environment and keep the link cache and the load order without needing to recompute one.

[Elscrux]

Yeah it would be a bigger project for sure I'd assume

the complexity depends on how deep you want to take it. Because, for example, if the analyzer suite just takes in a load order of mod objects... well then the data folder is not applicable really. The analyzer suite can just use the mod objects and be done with it.

It'd be up to the user to make sure those mod objects were constructed appropriately and had the correct data that was desired.

I dont think the analyzer suite itself has to walk that road, as it can just take the LO/LinkCache to be used straight up.. and leave it to the user to figure out how those were constructed.

Isn't this still relevant when the record frame analyzers come into play? Then the analyzer suite is currently trying to access the mod in the data folder. A mod object won't help because it doesn't have any info about where it came from.

Additionally, when we add asset analyzers in the future, or look at the existing missing asset analyzers that run on records, this is also a concept that we need to keep in mind maybe? Currently it would only support something in the data folder, though of course when you add mod folders into the mix now it's getting a bit more complex. Does this asset exist (fine, just look if it exists in any folder). Is this asset doing what it is expected to? Now we might have multiple overrides of an asset and need to take that into account, just like we do with records with multiple overrides in multiple mods.

Though I do wonder if the analyzer suite would be interested in the data folder mode, where it is more aware of the the actual concept of a data folder.

This might be used by analyzers that aren't focused on mutagen object records directly, but instead would throw topics on the data folder being malformed in some way for a user's install.

Perhaps that'd be via an optional WithDataFolder call that isn't required

A data folder mode of some sorts would be a great side product of what I'm thinking here. Definitely a use case that would be applicable to the average mod user.