[Discussion] The ease of understanding versus the amount of features

[shpaass]

I want to outline a balancing act in the development of Yafc.
A project must have features to be useful, but each new feature makes it harder to add more.
If the features are held back too much, then the project dies to a fork. If the code gets too hard to understand, then the project stagnates because no one can add new features, and eventually it dies to a competitor.
Therefore, each feature is a balancing act between how useful it is and how much it complicates the codebase.

When ShadowTheAge was developing Yafc, he didn't have to worry about the ease of understanding because he was the main contributor by a landslide. Right now, the situation is different - the project is kept alive by various people. @DaleStan is the most active contributor, @SWeini helps with critical issues, @veger reviews the PRs, I help to keep the lights on, and many others contribute over time.

To summarize, we do have to take the ease of understanding into account. Not just by asking if the feature is done well, but if this feature needs to be added at all.

I'm writing this to compare my experience with others. If you've worked with the Yafc codebase, then you are welcome to share how easy or hard you found it to understand.

[craig-johnston]

I think you should assign one or a small group of people to make the call on:

• Accepting a feature at all
• If an implementation of a feature is good enough
• What important changes they'd like to see prioritised

For the last point, there needs to be someone (or some people) who recognise that the current way something is being done needs to change. Simon gave a good example of overhauling the way the optimisation works (moving away from recipes and pseudo-recipes). But I'm sure there are UX and similar large decisions that also need a leader(s) to decide that's the right way.
Basically anything that may be controversial but is being done for the genuine belief it's better in the long run.
This is also true for non-functional decisions like how the code should be structured.

[DaleStan]

UI: I don't remember having much trouble figuring it out. There is too much business logic in the UI code, though.

L10n: I didn't understand it until I read the Factorio docs three or four times, but I eventually figured it out well enough to build the i18n on top of it.

Serialization: Oh horrors. I'm pretty sure I lost SAN points figuring this out. It brings to mind what Terry Pratchett said about elves: "It's wonderful. It provokes wonder. It's marvelous. It causes marvels. ... It's terrific. It begets terror."
In theory, it should be easier to use ICustomJsonDeserializer than to interact with the core serialization system, but since it's no longer used, it could be removed to simplify things.

Table solver: The connection from Yafc to ORTools is just plain ugly. I think that's more the fault of ORTools than anything else. I can force my way through it when I need to, but it never makes me happy.
Having the solver use recipes (technically recipe rows, since recipes don't specify crafter, fuel, or modules) appears to be unpopular, but I haven't seen any suggestions for what it should use instead.

Cost estimation: It does something and numbers (usually) come out. Sometimes even useful numbers.
The assumptions that all recipes will eventually become available (false for py) and that there's no inter-surface logistics costs (false for SA and SE) make the numbers less useful than they otherwise could be.

Lua parsing: Works well enough, as long as you also have https://lua-api.factorio.com/latest pulled up on a second monitor. It often helps to have the lua dump (from https://github.com/DaleStan/yafc/tree/dump-data) open too.
I experimented with generating code from prototype-api.json, but many defaults are missing or difficult-to-parse (e.g. "clamp(level, 0, 64)"), and it would probably require separate 1.1 and 2.0 parsers.

[maxreader]

---

Documentation

Coming from having worked on other shall we say, "complicated," codebases, the overall process of "how do I figure out what I even need to change" is about par for the course, with so far the exception being the solver portion. Having clearer descriptions/definitions of terminology in that area would go a long way, especially for links, constraints, link coefficients. In general, documentation is king. My experience thus far is that some of the summary stuff has been pretty helpful, just needs more of it, and more clear documentation on how to implement it and which sort of extension/plugin/whatever uses it to maximize its usefulness.

---

Refactoring/Readability

My overall suggestion regarding refactoring in the small would be "leave things as good as or better than you found it." If you're touching and testing a known spaghetti area that would be reasonable enough to at least improve readability in, it should pretty much be an obligation to do so for your feature to get merged. Doesn't necessarily need to be in the same PR.

That's coming from some more professional non-FOSS development experience, where the time-spent expectation is 50% enhancements and 50% fixes/non-enhancement development. Not really feasible to do exactly that with FOSS stuff, hence suggesting that features get paired with a refactor/fix whenever possible.

---

Feature Triage, Scope Definition and Conventions

I'm not super concerned about the "if this feature needs to be added it at all" part, but haven't been party to development here for long enough to see how many frivolous requests get made. My recommendation there would be to effectively make a "charter" that defines the overall project goals and scope, and that becomes your guidepost for "does this feature request make sense."

To take a very recent example I saw as a sort of "XY" problem relevant here in the Discord, I would not say "YAFC should have a built in (general purpose/manual entry) calculator" is a reasonable feature request, because it's not really solving any of the Factorio-specific problems here. You can easily do that in a plethora of other software and I would consider to be a waste of effort for YAFC. Their root problem was that they wanted to see the inserter throughput count divided by the number of machines... which DOES seem like a reasonable display option that's directly tied to what YAFC is meant to do. So for individual feature requests, I would say to require an example of which specific action they want to do that requires some sort of manual fiddling/external calculation that isn't feasible directly in YAFC today, then solve that specific problem in a way that actually eliminates user effort instead of just moving that effort around.

From a more principled standpoint, I could buy "figure out all the ratios you need for your factory without needing to ever be manually typing numbers/math into an external tool or calculator" as one of those guideposts. Take any principle like that, apply it to the issues we have NOW, see if it actually is a good guidepost/heuristic for what you'd otherwise have decided. If it is, codify it as convention.

[ApocDev]

UI - Having worked with ImGui quite a lot in the "recent" past (5+ years at this point), dealing with the UI is "fine". A lot of missing obvious behaviors (like properly scrolling scroll areas, horizontal scrolling, actually detecting mouse contains, global keydown/up handlers that aren't tied to some mouse action), and the fact that most of the UI related stuff is tightly coupled with a bunch of business logic, makes it very difficult to follow the logic.

I shouldn't have to look through 100+ lines of code to figure out that "oh, this button has a Ctrl+Click behavior that only works when the planets align". UI code should be dead simple and call external services/etc to do the actual logic... but instead most logic is buried directly in UI action handlers.

Testing - Has anybody actually tried testing complex features that require UI interactions? ... you can't... But then again, you can't submit the PR because there are no tests... but you can't add tests because you can't run the UI in a test environment...

Solver - The decision to use float over at least double seems awkward. It may have made sense when most people were still on x86, but I don't even know if you can still buy an x86 PC anymore. If you can, I doubt you're really playing Factorio on it anyway... The world has come a long way. There are a lot of "hacks" to deal with FPU precision simply because we chose floats instead of doubles/decimals. Memory is cheap, and YAFC isn't going to suddenly start using GBs of memory by using higher precision math. Sure, JSON serializers only go "up to doubles", but that's what string<->decimal converters are for (also why most financial DTOs use strings for $ values, because the precision matters). Granted, it's a game, and we usually don't care about 20+ decimals of precision, but the point still stands. Compounding errors and all that.

Feature Selection - Who decides what a "good feature" is? Is my need to have hotkeys to do consistently tedious actions something that would be denied? What about someone who wants to add a "Pink" theme? What about a feature that is "niche" for some specific modpack? Who gets to decide what can/cannot be added?

On that same note, what is "good enough"?

---

For me personally, the most painful parts of doing much of anything in the codebase is the UI. Even with ImGui, you have proper component renderers to ensure consistency. I can make an "item button" look however I want, because there's no such thing as an "Item Button". Add in the business logic dropped in the middle of button click handlers... and it's a nightmare sometimes. Just look at ProductionTableView.OpenProductDropdown - 50 lines before you see that there is a Ctrl+Click handler... and the method itself is THREE HUNDRED lines long...

I understand some of this stuff is tech-debt from a long history of development (inconsistent naming schemes, large files, permanent temporary hacks, etc)... but I can't help but pull my hair out when I want to add even the simplest of things...

[shpaass]

Who decides what a "good feature" is? [...] Who gets to decide what can/cannot be added?

The maintainer with the context from contributors.

Is my need to have hotkeys to do consistently tedious actions something that would be denied?

For Yafc to be alive, it needs to cater to its userbase and be maintained. Essentially, every change is a question: "How would it affect the devs and users?" I've added a mention of it in Contribution, and the Feature Design template for a dev to express how their feature would affect that. In a nutshell, a feature wants to make the lives of both the devs and users better.

What about someone who wants to add a "Pink" theme?

If that was an example of a cosmetic feature, then it would be evaluated with the same core question - "How would it affect the devs and users?"

What about a feature that is "niche" for some specific modpack?

Yafc was created to help playing Pyanodons, so the main target-audience is the Pyanodons userbase. Other modpacks are supported based on the dev interest in them, but they still get evaluated by the same core question.

On that same note, what is "good enough"?

From what I understand, you refer to @craig-johnston "[decide] If an implementation of a feature is good enough". Using the same core question "How would it affect the devs and users?", the minimum requirement is "It is at least the same for users, and at least an improvement for devs". In other words, the user experience must not degrade, and the dev experience must get better. The reason for such a requirement is that, as you mentioned, the codebase is not in a good state right now, so we need to improve it.

I understand some of this stuff is tech-debt

Not just some. Most of this stuff is the tech-debt that Yafc needs to address.

[ApocDev]

Not just some. Most of this stuff is the tech-debt that Yafc needs to address.

If the main issue is that the code base is hard to work in, maybe that should be the focus, and just implement a feature-freeze until it's complete?

[shpaass]

If the main issue is that the code base is hard to work in, maybe that should be the focus, and just implement a feature-freeze until it's complete?

Unless our contributors love to refactor, I think the feature-freeze would be endless because no one would want to just refactor. Therefore, it's more of a "If you want to add a feature, then please make the overall result less complex than the initial state."

[SWeini]

copy of my discord post from a few days ago:

My vision is that yafc has all features that are necessary for modeling/balancing whole factory, that includes a bunch of stuff we don't support at the moment like rocket logistics. You may exclude any scripted mechanics as those are inherently difficult to model, otherwise it would have been a recipe.
The biggest problem I had when I was more active is that the core itself is not flexible enough. Those dummy recipes are not easy to work with and are quite limited. The GUI code is not well separated from the logic. And the Lua prototype parsing lacks clarity.

[veger]

I find the code base of YAFC is too nasty to be fun to work with. The mixture of business logic and UI (as mentioned already). The ImGui itself having rendering and actions in the same place. I don't like these kind of frameworks, I see them in Frontend applications as well (glad I am backend engineer 😉). The serializer being 'somehow' working (I definitely lost points on it it too). It is definitely getting better with the contributions of DaleStan though. The lack of any documentation at all, again since we forked the community edition this has become a focus and much better!

I do find YAFC the only planning tool for complex Factorio that makes sense to use. Back then I tried them all and YAFC has all I needed. Never tried others since then. Soon I found I still I missed some features and got involved in the by fixing stuff and adding features. Fixing stuff and improving YAFC was fun as it immediately made my Factorio life easier. I continued this with our fork, espcially being able to finally merge the loads of PRs that were stuck in limbo, but at some point the minor stuff as fixed (for me at the least). And I went slowly to a reviewer role to continue supporting YAFC. I have to admit I don't play Factorio (a lot) nowadays, so that might be the reason that I do not find new annoyances anymore trigging me to fix them. This is mainly due to time constraints: you can spend your time only once, and with some young kids running around time is even more limited 😄 But I am pretty sure I'll pick up Factorio in the future as it is my number 1 game ever, and I still didn't finish a full Pyanodon playthrough yet (started enough).

As goes for all of us, we work on YAFC for fun, it is not our main job. So adding 'chores' that no one likes will drive away motivation for sure. So refactoring YAFC into something that is actually understandable and maintainable might be too big a project. What might be a nicer approach is to start from scratch (at the least it would be for me), as we can get rid of all the legacy, half implemented plans, etc. But even with this approach I doubt it will ever reach the same feature level as we currently have.

For me the most fun is to have a coding project that I can fall back to. I have a certain need of writing code, which either gets fulfilled at my day job or on my free time. If my day job prevents scratching this itch, I see myself coding more on my spare time, and the other way around. YAFC in whatever form (I can understand all of the code if I am motivated enough) is fun for me as improving it (by code or reviewing useful code) as it helps me making plans for my next Pyanodon playthrough.

So whatever route is going to be chosen, I will stick to the project in whatever form I can provide my contributions.

sorry for the not so coherent rambling. I am not feeling well last couple of days, but still wanted to chime in on this

[shpaass]

start from scratch

I considered it once. After I looked at the number of lines in Yafc (23k+ iirc), I stopped considering it. The amount of work would be monumental compared to even fixing all the stuff we mentioned so far.

[veger]

Agreed, we'll never reach (my guess) the same level of features. At least not anytime soon.

[SWeini]

To be honest, I always felt a bit depressed when seeing the technical debt and realizing that solving all of it means basically restarting from scratch. Therefore I kept my involvement to a minimum, only jumping in when something critical popped up that stood in the way of me using it. I really like the fact that I can use YAFC on my second monitor, but the idea of planning mods starts to grow on me.

Pro mod:

• Better integration into save (reading available techs/recipes, surface effects, custom day-time-cycles, ...)
• Better integration when executing the plan (blueprint export/import, shopping list, ...)
• Mod-compatibility via scripts (that can do anything) instead of prototypes (which are really limited)
• Prototype data and icons are easily available

Con mod:

• No access to Google OR-Tools
• Complex GUI is somewhere between difficult and impossible
• Sheets are stored in the save game (other players might see this as a pro, I don't know)
• Forced to use Lua

Obviously YAFC as a mod wouldn't be YAFC anymore, it would be something else entirely.

But I still wonder how much demand for YAFC exists if there was a planner mod that uses simplex solver on a large scale and provides cost analysis. The one thing that makes YAFC stand out for many players is the good solver due to OR-tools.

By the way, these thoughts triggered me to write Production Rates Calculator Extreme - possibly as a precursor to a planner mod.

Then there is the whole udp feature that got introduced recently, which could improve many of the integration aspects, for the cost of a more difficult installation procedure. Maybe one could also use udp in the other direction and call OR-tools from a planner mod.

When starting from scratch the whole mod vs. external program topic needs to be seriously discussed.

I'm more inclined to contribute if I didn't have to work in this mess of technical debt and I could see a great vision. I'm not scared of starting from scratch, even have some CLI tool that does data reading, solving and cost analysis.

Sorry for the wall of text, I don't want to steal this whole discussion, but if you have an opinion on my thoughts I'd be very much interested.

[ApocDev]

To be honest, I always felt a bit depressed when seeing the technical debt and realizing that solving all of it means basically restarting from scratch. Therefore I kept my involvement to a minimum, only jumping in when something critical popped up that stood in the way of me using it. I really like the fact that I can use YAFC on my second monitor, but the idea of planning mods starts to grow on me.

Pro mod:

* Better integration into save (reading available techs/recipes, surface effects, custom day-time-cycles, ...)

* Better integration when executing the plan (blueprint export/import, shopping list, ...)

* Mod-compatibility via scripts (that can do anything) instead of prototypes (which are really limited)

* Prototype data and icons are easily available

Con mod:

* No access to Google OR-Tools

* Complex GUI is somewhere between difficult and impossible

* Sheets are stored in the save game (other players might see this as a pro, I don't know)

* Forced to use Lua

Obviously YAFC as a mod wouldn't be YAFC anymore, it would be something else entirely.

But I still wonder how much demand for YAFC exists if there was a planner mod that uses simplex solver on a large scale and provides cost analysis. The one thing that makes YAFC stand out for many players is the good solver due to OR-tools.

By the way, these thoughts triggered me to write Production Rates Calculator Extreme - possibly as a precursor to a planner mod.

Then there is the whole udp feature that got introduced recently, which could improve many of the integration aspects, for the cost of a more difficult installation procedure. Maybe one could also use udp in the other direction and call OR-tools from a planner mod.

When starting from scratch the whole mod vs. external program topic needs to be seriously discussed.

I'm more inclined to contribute if I didn't have to work in this mess of technical debt and I could see a great vision. I'm not scared of starting from scratch, even have some CLI tool that does data reading, solving and cost analysis.

Sorry for the wall of text, I don't want to steal this whole discussion, but if you have an opinion on my thoughts I'd be very much interested.

While I agree with most of this, why not simply have the best of both worlds? I've already created a tiny mod to write all the unlocked technologies and TURD selections to a file that gets updated whenever research is finished, or the game is saved or loaded. Then there's a file watcher looking for that file in the mods folder, constantly updating the milestones and unlocked tech.

This keeps all the benefits of both (except maybe the in game GUI, but you have way more freedom externally anyway). Expanding it to export more data is trivial. You only need to update info when major things change. Plus, "installing" the mod is trivial too since yafc already knows your mod folder (or can be extended to have a secondary mod folder).

I still dont quite understand the need for manually parsing all the prototype info when you can simply export it from the source of truth.

[shpaass]

I still wonder how much demand for YAFC exists if there was a planner mod that uses simplex solver on a large scale and provides cost analysis.

I'll start with a preface that I'm glad you shared your opinion.

When I think of a mod alternative, I always wonder why the existing planner mods haven't implemented better solvers yet. Probably there are big problems on the way. Regarding the demand, I would also consult the devs of the existing planner mods.

I deeply resonate with the saying "You either write legacy or support legacy." In other words, I think that the benefits of a cleaner architecture are not worth the risks and effort of writing the whole thing from scratch.

If we want to further discuss the idea of writing an alternative from scratch, then I suggest to make a separate [Discussion] issue, because I think this topic is different from the original "features vs complexity."

[SWeini]

If we want to further discuss the idea of writing an alternative from scratch, then I suggest to make a separate [Discussion] issue, because I think this topic is different from the original "features vs complexity."

See #542, feel free to continue the discussion there.