[Discussion] YAFC 2.0

[SWeini]

Make new YAFC from scratch. Same as now, only better, and easier to maintain.

Motivation

It became quite difficult to add new features and many of the core systems need a rework. It might be easier to rewrite from scratch than to refactor all core systems one after another.

Changes

External Program vs. Mod

I'm assuming here that YAFC stays an external program. However, Helmod and Factory Planner provide a good alternative and currently the main selling points for YAFC are:

• Best solver (simplex method using OR-tools)
• Cost analysis (even if it has some problems)
• Used on separate monitor (preferred by some, others prefer an in-game solution)

Therefore we should think really hard about going the mod route before rewriting from scratch.

Lua Runtime vs. JSON Export

It is quite difficult to duplicate factorio's Lua runtime 100% accurately. In the past there have been plenty of mods that failed to load or needed adjustments in that area. There is an easier way: Factorio supports a bunch of command line parameters to export most relevant data:

• --dump-data can replace the whole Lua runtime with a simple json parser
• --dump-icon-sprites generates easy-to-use icons for each item/fluid/recipe/entity. Right now not all icons are shown 100% accurately. Could fix all those bugs and remove a bunch of code at the same time.
• --dump-prototype-locale generates easy-to-use translations for each item/fluid/recipe/entity. Same here, currently not everything has correct locales, could remove a bunch of code.

In general, this would remove a quite large part of the codebase and replace it with something simpler. As long as factorio can load a specific set of mods it is guaranteed that YAFC can also use it, and that the data/icons/locales do match perfectly.

All the data should be zipped and saved as a modpack file. YAFC project files would specify such a modpack and loading a project file simply needs to load the data, resulting in significant performance improvements during load.

With all data being easily available it will be possible to show a diff of two modpack files (e.g. after updating mods). I used Foreman in the past, just to see what changed after updating mods, because they provide modpack files (via its own exporter mod).

This change needs a different way for mods to specify scripted mechanics. The current way does kind of work, but it's also very limited because it doesn't support anything that can't be expressed in normal factorio prototypes. Not fitting into factorio prototypes is the main reason to add scripted mechanics.

We would also lose the ability to exclude mods completely. This is currently a great feature for mods that only provide item compression (deadlock stacking) or transportation (transport drones). However, you will always have the possibility to create a modpack file with those mods disabled.

There is a machine-readable API declaration, I already tried using this to generate a full data-stage json parser, but it needs too much manual adjustment and the API changes almost every factorio version which complicates things further, even though most API changes are backwards compatible.

GUI

Many contributors have expressed concerns in the current way of doing GUI things. I don't want to pinpoint specific problems here, but there is plenty of room for improvement. The main goal of rewriting the whole GUI layer is a strict separation of the GUI layer, making it possible to write tests for the rest of the codebase.

Maybe even think about changing the GUI framework.

Recipe

Currently everything is converted to fake recipes which works OK-ish for some things (e.g. boilers) and less so for others (e.g. spoilage). The goal is to replace recipes with something more abstract and flexible. Recipes are just one specialization of it.

Unspecified Inputs

Allow fluid inputs to not define a specific temperature (and use whatever is available). Allow fluid fuel and item fuel to not define specific fluids/items (and work with an abstract fuel).

This makes it possible to not care about the specific fuel at all or change the fuel for a whole sheet at once. But it is also important for the next point.

Cost Analysis

Currently gives a good estimation for most recipes, but is totally wrong on others. Areas to improve are:

• prod/speed/efficiency modules
• fluid/item fuels
• proper temperature handling
• proper cost for electricity
• multi-surface analysis with inter-planetary logistic costs
• rewrite ore-density detection for 2.0
• configurable cost function
• configurable optimization target.

Serialization and Undo/Redo System

I didn't look into this topic too closely yet, but I've heard complaints about code quality and ease of understanding in this area. Might be worth looking into ways to drastically change this, hopefully to the better.

Summary Sheets

There have been discussions about the value of the legacy summary sheets and the new summary sheets. Do a proper requirements analysis in this area.

Sheets and Sheet Solver

This might be the area that can mostly be kept. Putting a sheet into OR-tools has some complicated code, but it is a complicated thing to do.

There are probably a few improvements that can be made. Yes, all of them could probably be done in the current system, but many of the above topics complicate things.

• multiple beacon types
• surface-to-surface logistic costs
• in general more support for multi-surface mods (building restrictions and such)

[ApocDev]

Lua Runtime vs. JSON Export

It is quite difficult to duplicate factorio's Lua runtime 100% accurately. In the past there have been plenty of mods that failed to load or needed adjustments in that area. There is an easier way: Factorio supports a bunch of command line parameters to export most relevant data:

--dump-data can replace the whole Lua runtime with a simple json parser
--dump-icon-sprites generates easy-to-use icons for each item/fluid/recipe/entity. Right now not all icons are shown 100% accurately. Could fix all those bugs and remove a bunch of code at the same time.
--dump-prototype-locale generates easy-to-use translations for each item/fluid/recipe/entity. Same here, currently not everything has correct locales, could remove a bunch of code.

--dump-data and --dump-prototype-locale would definitely be the most useful (and required together), however, the --dump-icon-sprites takes quite a while to run. We would need to have an "initialization" phase for each project, or to re-initialize whenever mods change. I'm not against it at all, but it is ~180MB of disk space additionally with full Py and a few other mods after being zipped. (Call it 175MB?)

Would it make startup much faster? Definitely. But it would definitely add a lot more "overhead" to the first startup. (Plus, you have to accept the notification in Steam when you dump the sprites, which is awkward)

GUI

Many contributors have expressed concerns in the current way of doing GUI things. I don't want to pinpoint specific problems here, but there is plenty of room for improvement. The main goal of rewriting the whole GUI layer is a strict separation of the GUI layer, making it possible to write tests for the rest of the codebase.

Maybe even think about changing the GUI framework.

My thoughts on the UI:

1. It tries to be ImGui without the polish, and feature set. (Global key states? Horizontal scrolling?)
2. Do we even need to have an immediate mode GUI? Especially with a render loop like we have, it's meant more for games, not for desktop applications like we're doing here. Even WinForms would be a slightly better choice in this case (yes, I will take a shower after saying that...)
3. What are the primary goals of a new UI?
   • Size?
   • Speed?
   • Ease of use?
   • DX?
   • Developer familiarity?
   • Flexibility?

Would a web-based technology fit best? (Electron, Photino, etc) Or something XAML based? (MAUI, Uno, Avalonia, etc)

4. The current UI lacks a lot of consistency. Some basic hotkeys (like escape) aren't actually consistent. A lot of mouse related hotkeys don't work (why can't we middle click a tab to close it? Or create a new one?) A lot of these behaviors are easily solved with a more robust UI framework

Personally, I'd push more towards web technologies, since they have "solved" all these potential issues a long time ago, and provide way simpler ways of making UI changes. Plus, you separate style from form. Want 30 different "themes"? No big deal, just add more CSS. Nothing changes elsewhere.

Recipe

Currently everything is converted to fake recipes which works OK-ish for some things (e.g. boilers) and less so for others (e.g. spoilage). The goal is to replace recipes with something more abstract and flexible. Recipes are just one specialization of it.

So more of a "Product Transformation" than a recipe?

Unspecified Inputs

Allow fluid inputs to not define a specific temperature (and use whatever is available). Allow fluid fuel and item fuel to not define specific fluids/items (and work with an abstract fuel).

This makes it possible to not care about the specific fuel at all or change the fuel for a whole sheet at once. But it is also important for the next point.

This would be great to have. But at the same time, how would you properly calculate that on a global scale? If you wanted "waste" fluids to be available for fluid fuels, but not "primary" fluids? What about only specific types of fluids? Would you have to mark the fluid as "use as generic fluid fuel" in each production tab in the exports?

I agree with the idea, but there are a lot of edge cases to think about. This would also involve adding more "custom items" or something on top to represent abstract "ideas" for items. (Eg; electricity)

[DaleStan]

tl/dr: With the exception of the UI, I don't see a good argument here for a rewrite.

I'm assuming here that YAFC stays an external program. However, Helmod and Factory Planner provide a good alternative

I agree here. There are multiple mod-based calculation tools, but I'm not aware of any other external tools that support arbitrary mod packs. Also, I doubt the simplex solving or the milestone analysis can be accomplished in a reasonable time in Lua. My test Pyanodon project takes half a second to process the milestones, or three seconds if I apply the single-thread restriction that Factorio will inflict.

It is quite difficult to duplicate factorio's Lua runtime 100% accurately.

I'd say more that there's been no real attempt to duplicate Factorio's Lua runtime 100% accurately. I, at least, have never gone through the definition of LuaHelpers and tried to figure out which ones should be supported.

That said, the Lua interop code was one of the last things I managed to figure out, partly (mostly?) because the Lua/C interop is so much of ... *gestures* ... that.

--dump-data can replace the whole Lua runtime with a simple json parser

This will take some work, but I don't think it warrants a full rewrite. I'd start with a search&replace from LuaTable to JsonElement (assuming System.Text.Json) in FactorioDataDeserializer and DataParserUtils, and then fix the various Get methods.

I'd also be interested to hear how you're planning to determine the concrete type for an ItemID or an EntityID without the assistance of the defines.prototypes table, especially if somebody builds a spoiling loop (e.g. X spoils into Y spoils into Z spoils into X). We didn't use that table until recently because we had carefully arranged things so we never saw a ModuleID or AmmoID before we'd seen the module or ammo itself. Once people realized they could make modules that spoil into other modules, that trick broke.

Note that the three --dump parameters cannot be used together, so you have to start Factorio three times (Not a bug, apparently). This can triple the loading time if Factorio decides not to obey the cache-prototype-data config option. Also, make sure you use --config, so yafc can run Factorio even if the user is currently playing Factorio.

--dump-icon-sprites generates easy-to-use icons for each item/fluid/recipe/entity.

This shouldn't need much rewrite either. Prefix the icon-rendering code with "go load that png file" code, and then continue to the existing render code for Yafc's custom sprites and the fluid-with-temperature icons.

YAFC project files would specify such a modpack

Assuming the same behavior as the current deserializer, this seems like a bad idea. The deserializer is pretty well encapsulated. Putting the modpack file information in the project file means you have to pause the project file deserialization to load the mod data. Alternatively, you could work some sort of magic so RecipeRow and friends contain strings, not FactorioObjects.

Instead, I'd tie the modpack file to the mod folder, and probably store it in the mod folder. Knowing exactly which mods generated the modpack also makes it easier to determine if the pack is out of date.

it is ~180MB of disk space additionally with full Py and a few other mods after being zipped.

I'm not sure what other mods you're using; I only get 114 MB zipped for py. (Input: 109 MB of sprites, 1.4 MB locale, and 62 MB of data.raw, using pymodpack.)
Packing the sprites into a .tar.lz file first reduces the size to 84 MB, (26% reduction) though that would require rewriting the icon rendering to iterate through the icons, instead of through the objects.
(Using lzip -9 because we don't care about compression time. That can happen on a background thread while the user is using yafc.)

84 MB isn't great, but it's not bad either, compared to nearly 1 GB for the full modpack.

Maybe even think about changing the GUI framework.
Even WinForms would be a slightly better choice in this case
Would a web-based technology fit best? (Electron, Photino, etc) Or something XAML based? (MAUI, Uno, Avalonia, etc)

WinForms, MAUI, and WPF are all non-starters, since they don't run on Linux. My votes are for something that supports hot-reload of the UI first, and XAML second.

The UI is threaded through everything, and is one of the few justifications I see for a rewrite. There were also some decisions, such as putting MathUtils and logging in Yafc.UI, that make it harder to judge exactly how nasty a UI rewrite will be.

Some basic hotkeys (like escape) aren't actually consistent.

That should be a bug report (or PR); escape should always close the active panel. It's possible to simultaneously open multiple panels of the same type, though.

A lot of mouse related hotkeys don't work (why can't we middle click a tab to close it? Or create a new one?)

Because no one implemented that. I'm not aware of any UI frameworks where the tab control is able to automatically create/remove tabs.

The goal is to replace recipes with something more abstract and flexible.

There's not much detail here, but I don't immediately see how this needs a rewrite either. As long as the recipe replacement can be stored in an IObjectWithQuality<>, the serializer will still be happy.

I'm pretty sure I can wrangle the serializer into reading/writing an IObjectWithQuality<> or FactorioObject property, if that's what you need. (Maybe typed as an IFactorioObjectWrapper or marked with an attribute?)

Allow fluid inputs to not define a specific temperature

This feels to me like the opposite of what you said elsewhere: #418 (comment) Also, an arbitrary-temperature fluid input would still have to respect the minimum and maximum temperatures set in the corresponding FluidIngredientPrototype.

This makes it possible to not care about the specific fuel at all

This seems to me to be the same as "don't link the fuel", or maybe the existing edit-the-yafc-file feature of setting "fuel":null. What makes this different?

rewrite ore-density detection for 2.0

This one I have figured out, using --generate-map-preview and --report-quantities, but it requires some sort of modpack cache, since it adds at least one more factorio execution to load time, and more in multi-planet packs.

I also tried to execute the noise expressions in roughly the same way we do for 1.1, but that got numbers that were sometimes negative or incorrectly zero, and other times apparently valid but significantly different than the --report-quantities results.

Serialization and Undo/Redo

As unpleasant as this is to debug, I wouldn't touch it. It's well encapsulated, pretty reliable, and covered by several tests.
It might be better if it had used JsonSerializer.Serialize and JsonSerializer.Deserialize, with appropriately located [JsonConverter]s, but there are much more significant pain points.

In addition, I'm concerned about both the second-system effect and what I'd call "Netscape 6 effect" where Netscape decided to delete all their existing code and start over, and lost all their market-share in the process.

[SWeini]

tl/dr: With the exception of the UI, I don't see a good argument here for a rewrite.

Thanks a lot for your input to that matter. I very much appreciate it.

There are multiple mod-based calculation tools, but I'm not aware of any other external tools that support arbitrary mod packs. Also, I doubt the simplex solving or the milestone analysis can be accomplished in a reasonable time in Lua. My test Pyanodon project takes half a second to process the milestones, or three seconds if I apply the single-thread restriction that Factorio will inflict.

There is also Foreman2, but its not actively developed. Don't even think it got converted to 2.0 yet.
I have no fear of milestone analysis or simplex-solving reasonably sized sheets (less than ~200 rows). My fear is the cost analysis part, which runs simplex on a gargantuan sheet (every recipe at least once, with improvements it will be a lot larger than currently)

I'd say more that there's been no real attempt to duplicate Factorio's Lua runtime 100% accurately. I, at least, have never gone through the definition of LuaHelpers and tried to figure out which ones should be supported.

Nothing is impossible. However, the maximum we can achieve is "currently no open bug reports", but that means that tomorrow we might have to fix another bug. Or we need yet another "defines" dump with a new factorio version. Btw, the defines dump might be still needed, unless we convert to being a mod.

That said, the Lua interop code was one of the last things I managed to figure out, partly (mostly?) because the Lua/C interop is so much of ... gestures ... that.

Agree, the Lua/C interop is just plain ugly, and I looked at it closely to fix some bugs. I can tell you that there are also a few sleeping bugs in there, where a top-level function pops too many values from the stack (from what I saw). Calling that method from another method would break the lua stack.

--dump-data can replace the whole Lua runtime with a simple json parser

This will take some work, but I don't think it warrants a full rewrite. I'd start with a search&replace from LuaTable to JsonElement (assuming System.Text.Json) in FactorioDataDeserializer and DataParserUtils, and then fix the various Get methods.

Maybe you are right and a change would be easier to do than I thought. Still need to think about how to support mod integration and how that would affect the described hot swap.

I'd also be interested to hear how you're planning to determine the concrete type for an ItemID or an EntityID without the assistance of the defines.prototypes table, especially if somebody builds a spoiling loop (e.g. X spoils into Y spoils into Z spoils into X). We didn't use that table until recently because we had carefully arranged things so we never saw a ModuleID or AmmoID before we'd seen the module or ammo itself. Once people realized they could make modules that spoil into other modules, that trick broke.

Loading order of prototypes and creating references between them has nothing to do with the lua vs. json question. That's another part that needs improvement.

Note that the three --dump parameters cannot be used together, so you have to start Factorio three times (Not a bug, apparently). This can triple the loading time if Factorio decides not to obey the cache-prototype-data config option. Also, make sure you use --config, so yafc can run Factorio even if the user is currently playing Factorio.

Yes, I'm aware of that, but it only needs to be done when mods change, not every time you start YAFC. You can even work on multiple YAFC projects of different modpacks at the same time without having to regenerate those files all the time, which is different from now where you either have to use multiple mod folders or restart factorio and switch mods there (which takes longer than 3 dumps).

--dump-icon-sprites generates easy-to-use icons for each item/fluid/recipe/entity.

This shouldn't need much rewrite either. Prefix the icon-rendering code with "go load that png file" code, and then continue to the existing render code for Yafc's custom sprites and the fluid-with-temperature icons.

Yes, this one could be easily replaced without touching anything else. Except that YAFC-integration code of mods is free to add any prototypes, so those would be without icon now. Unless you leave the current code, but that leaves us with the worst of both worlds.

YAFC project files would specify such a modpack

Assuming the same behavior as the current deserializer, this seems like a bad idea. The deserializer is pretty well encapsulated. Putting the modpack file information in the project file means you have to pause the project file deserialization to load the mod data. Alternatively, you could work some sort of magic so RecipeRow and friends contain strings, not FactorioObjects.

You misunderstood my proposal. I meant a different file, not the .yafc project file itself. Can be a simple zip with the contents of the --dump-... commands, and the same zip can be shared between multiple .yafc project files.

WinForms, MAUI, and WPF are all non-starters, since they don't run on Linux. My votes are for something that supports hot-reload of the UI first, and XAML second.

I worked with Avalonia in my previous job (medium sized desktop application (at least 10x YAFC size) that runs on Windows and Mac). Was pleasant to work with.

The goal is to replace recipes with something more abstract and flexible.

There's not much detail here, but I don't immediately see how this needs a rewrite either. As long as the recipe replacement can be stored in an IObjectWithQuality<>, the serializer will still be happy.

You are correct, everything can be done in the existing codebase as an isolated "small" rewrite.

Allow fluid inputs to not define a specific temperature

This feels to me like the opposite of what you said elsewhere: #418 (comment) Also, an arbitrary-temperature fluid input would still have to respect the minimum and maximum temperatures set in the corresponding FluidIngredientPrototype.

Sorry for being a bit short on my words. Obviously "arbitrary" means "anything in between the min/max defined". Heat needs the same support for min/max temperatures as fluids. What I'd like to see is when a building needs "any heat above 200°" you can leave the exact temperature unconfigured and any reactor generating at least 200° heat can satisfy that demand. This does not contradict my earlier comment you linked.

This makes it possible to not care about the specific fuel at all

This seems to me to be the same as "don't link the fuel", or maybe the existing edit-the-yafc-file feature of setting "fuel":null. What makes this different?

I didn't know about the "fuel":null trick. But that would only remove the fuel, right? I still want to link it properly.

What I like is, imagine multiple buildings needing liquid fuel (e.g. oil burners, glass works with multiple recipes) and multiple liquids being made (e.g. tar processing plus acetylene/syngas on top of it). The way to do this now is to have multiple copies of let's say oil burners to consume all the different fluids. But that can break when the oil burners are not enough anymore to void all the different fluids while let's say your glass works run on acetylene. To fix this I have to juggle around the multi-fluid rows to the glassworks now or assign fluids in a better way until it works. I like to not make a decision upfront which recipe is fueled by which fluid. I want n recipes consuming X MW of fluid energy and I want m additional rows each converting a single fluid to fluid energy.

In addition, I'm concerned about both the second-system effect and what I'd call "Netscape 6 effect" where Netscape decided to delete all their existing code and start over, and lost all their market-share in the process.

I'm aware of the risks that why I wanted input from you guys.

If I understood your opinion correctly, you agree that certain parts need improvement, but you'd just refactor in the existing codebase. Is that correct?

My main reason for doing a rewrite is that doing all those improvements will effectively be a full rewrite. It might be easier to start from a clean slate instead of being hold back by the tech debt in other systems when integrating the new code into the existing code.

[DaleStan]

I'd also be interested to hear how you're planning to determine the concrete type for an ItemID or an EntityID

Loading order of prototypes and creating references between them has nothing to do with the lua vs. json question. That's another part that needs improvement.

The reason this is relevant is because the current loader reads burnt_result = "something" and uses the prototypes table, which isn't part of data-raw-dump.json, to decide whether to create an Ammo, Module, or Item. Do want to store the string and do a second pass for cross-referencing?

Assuming the same behavior as the current deserializer, this seems like a bad idea. The deserializer is pretty well encapsulated. Putting the modpack file information in the project file means you have to pause the project file deserialization to load the mod data.

You misunderstood my proposal. I meant a different file, not the .yafc project file itself. Can be a simple zip with the contents of the --dump-... commands, and the same zip can be shared between multiple .yafc project files.

I'm imagining a yafc file that looks something like this:

{
  "settings": {
    //...
  },
  //...
  "modpackFile": "c:\\games\\Factorio\\mods\\yafc-data.zip",
  //...
}

To make this work, we have to find the modpackFile property, load the specified modpack, and then go back and reread the earlier portions of the project file. The "Item.whatever" strings in the file are converted to the corresponding FactorioObject, which doesn't exist until the mod data is loaded.

What I like is, imagine multiple buildings needing liquid fuel (e.g. oil burners, glass works with multiple recipes) and multiple liquids being made (e.g. tar processing plus acetylene/syngas on top of it)

If I understand you correctly now, within the current framework, I'd create a "Fluid.generic-liquid-fuel", display it in MW, add that as a valid fuel for each fluid burner, and add recipes that convert 1 MJ of each liquid fuel into 1 MJ of generic liquid. For extra points, I'd silently inject these recipes into the solver if the user linked generic liquid, but didn't add any of its production recipes.

For generic items, I'd make the conversion recipes spit out the waste product (burnt_result), but I'm not sure if/how to silently inject those recipes. Maybe that means fluid conversion recipes shouldn't be silently injected either.

If I understood your opinion correctly, you agree that certain parts need improvement, but you'd just refactor in the existing codebase. Is that correct?

Yes, that's correct. My experience is a third variant, where someone told one of our senior developers to "ship that product." He did, but he forked the codebase and broke or removed several features other customers relied on. It was nearly a decade of fixing bugs in one version, and then having them crop up in the other, before we finally merged the two forks.

[SWeini]

The reason this is relevant is because the current loader reads burnt_result = "something" and uses the prototypes table, which isn't part of data-raw-dump.json, to decide whether to create an Ammo, Module, or Item. Do want to store the string and do a second pass for cross-referencing?

Yes, the clean way to solve this is a two-pass system where either the second pass resolves the references or the first pass does nothing except creating the correct instances.

I'm imagining a yafc file that looks something like this:

{
"settings": {
//...
},
//...
"modpackFile": "c:\games\Factorio\mods\yafc-data.zip",
//...
}
To make this work, we have to find the modpackFile property, load the specified modpack, and then go back and reread the earlier portions of the project file. The "Item.whatever" strings in the file are converted to the corresponding FactorioObject, which doesn't exist until the mod data is loaded.

An implementation detail should never prevent a general idea from being implemented. If everything else fails, read that single data from the yafc file in a separate pass upfront.

If I understand you correctly now, within the current framework, I'd create a "Fluid.generic-liquid-fuel", display it in MW, add that as a valid fuel for each fluid burner, and add recipes that convert 1 MJ of each liquid fuel into 1 MJ of generic liquid. For extra points, I'd silently inject these recipes into the solver if the user linked generic liquid, but didn't add any of its production recipes.

For generic items, I'd make the conversion recipes spit out the waste product (burnt_result), but I'm not sure if/how to silently inject those recipes. Maybe that means fluid conversion recipes shouldn't be silently injected either.

You understood correctly. Item conversion recipes would include the burnt_result. For items there is a further complication because machines can accept multiple fuel categories. I wouldn't inject conversion recipes silently, though.

Yes, that's correct. My experience is a third variant, where someone told one of our senior developers to "ship that product." He did, but he forked the codebase and broke or removed several features other customers relied on. It was nearly a decade of fixing bugs in one version, and then having them crop up in the other, before we finally merged the two forks.

Ending up with two codebases that need maintenance is a nightmare, fully agree. I had good experience with a full rewrite of a software product, but we could partition the work so that it was 6 parts that were rewritten one after the other without much overlap. Overall took 2 years, but each part took "only" a few months. Unfortunately I don't see well separated parts in YAFC that you could rewrite without massively affecting the rest of the codebase.

Btw, this discussion doesn't need to end in a decision to a) do nothing or b) restart from scratch. Maybe the result is a decision to tackle topics a, b and c in a specific order and not touch d and e at all. Currently I don't see a vision of how to go forward, that's why I started the discussion.

[ApocDev]

We don't need to have a "modpack" file to store the data/images.

I've been working on this in my own time (separate from any goings-on here).

I'm finding the best approach for the data loading is as follows:

1. On project creation (or mod list change, or mod version(s) change), run the --dump-data and --dump-prototype-locales
2. Parse the necessary data from the raw dump into a less chaotic format (we don't care about collision boxes, fluid boxes, "glow" temperatures, etc)
3. Simply utilize %AppData% (or Linux equivalent) to store projects in their own directories. No more "point me to your one, single, yafc project file". A project is now a directory, with it's own settings file, it's own "data" files, etc.
4. When loading a project, just parse the data we already cleaned up from the initial dump pass into simple POCOs.
5. Run a 2nd pass to set up object references. (Eg; Recipe.Results[0].ItemRef = db.GetItem(Recipe.Results[0].Item and so forth.) This takes < 10ms, if not faster. There just isn't a lot of stuff to set up references for. And you know the POCOs already exist in some form.

I've been doing this with my "rewrite", and the dump phase takes basically as long as Factorio takes to execute those two commands. And loading a project is ~2-300ms, most of which is spent updating the UI with status updates.

Icons are simply loaded directly from mod zips (but could be done via the icon dump instead, which would be much easier, and faster, albeit with the long create/update time for when mods change).

Project structure (%AppData%/Yafc folder on Windows for instance):

Yafc
│   recent-projects.json
│   uiSettings.json
│
└───projects
    └───test
        │   project.json
        │   settings.json
        │
        └───data
                data-assembling-machines.json
                data-fluids-temperature-variants.json
                data-fluids.json
                data-fuel-conversion-recipes.json
                data-items.json
                data-recipes.json
                data-simple-entities.json
                data-technologies.json

In this case, I've dumped the relevant prototypes in a post-dump format, with vastly simplified types. I also included the "split" fluids (for temperatures) as well as fluid fuel conversion recipes (I actually came to the same conclusion as @DaleStan for converting at 1MJ. But I just have distinct recipes with a special category to hide them from the UI as normally selectable.)

Basically, pay all the data costs once, when we actually need to look at data updates. The rest is just clever caching.

The only downside is maybe milestones, but even that is pretty trivial with a separate "sync" mod to let us know when techs are unlocked, or TURDs selected, etc.

In my personal "rewrite", I'm focusing on the UI and basic data patterns as needed. The rest of the codebase is more or less reusable with some small tweaks to change access patterns. (At least, that's my current goal 🤞)

[DaleStan]

This change needs a different way for mods to specify scripted mechanics. The current way does kind of work, but it's also very limited because it doesn't support anything that can't be expressed in normal factorio prototypes.

What does this look like? What language is it written in? The two choices I see for language are JSON and Lua. Even if no other external tools use the same script patching, I don't think we don't want to impose a third language on mod developers. JSON has the advantage that we don't need an external library, but Lua has a more natural way to specify alterations/removals. (At least to me. JSON Patch exists, but makes me shudder.)

Is it possible for Yafc to inject these fixes, or do they have to be baked into the mod? The Ultracube fix works just fine if moved to data-final-fixes, but the textplates fixes both rely on the ability to edit the return values of require.

What other abilities should the scripted mechanics allow?
For example, I have some as-yet-unused code that lets you tell Yafc a particular item or fluid depends on having at least one of any number of other objects. (Some source objects break cost analysis, and there's no way to require all.) I also have some half-baked code that lets you tell Yafc a particular recipe requires all of two or more researches.