Machine readable format

[burner1024]

I'm thinking that it could be beneficial to have IESDP data in machine readable format. Currently there are multiple tools all parsing and reading IE files on their own: DLTCEP, NearInfinity, iesh, probably others. Inevitably, differencies and inconsistencies arise. But if file formats were described in a structured way, we could have a single, truly definitive source of information, and pull updates from it (semi)automatically. (Which is my ulterior motive for BGforge MLS.)

As an example, take a look at sfall docs: functions listed in a yaml file, with some python scripting it converted to markdown, and markdown is published with jekyll, resulting in a nice site.

If you look at opcode description, it's already basically yaml, with html additions. Could probably converted into true yaml semi-automatically.

To be clear, this is not about IESDP looks, just internal data representation: binary file formats, script actions/triggers, effects.

Tagging @Argent77, @4Luke4, @AvengerTeamBG, @FredrikLindgren, @ALIENQuake.

[lynxlynxlynx]

Something like this might have been useful 15 years ago, but now that almost everything is already written, I don't see anyone gaining anything by rewriting all the parsers, writing generators and rewriting the way the data is stored.

On the IESDP side of things the last part may still happen though, since it could then convey the data better (eg. #6 #19 #26).

[burner1024]

almost everything is already written

MLS isn't, hence this issue.
Obviously, I can't speak for devs of other tools, but MLS would benefit.

@lynxlynxlynx let me put it like this: if a such pull request was coming your way, would you be opposed to it?

[lynxlynxlynx]

Depends on what everything you'd stick in it. Eg. if you want to make the file formats machine readable, that's great and wouldn't need to affect deployment, since jekyll has the needed support: https://jekyllrb.com/docs/datafiles/

[burner1024]

Do you want to define a format for, say, opcodes? Or I should do that myself?

[lynxlynxlynx]

I don't know what you mean, since the opcodes are already easily machine readable. Everything is parametrized and the description is plain html most of the time (a few liquid tags here and there).

[burner1024]

Well, html with liquid parts is not exactly machine readable.
Still being parametrized is exactly why it's easier to start with them.

To make an example, maybe something like this
_data/opcodes.yml

- n: 0
  name: AC vs. Damage Type Modifier
  type: stat
  param1: AC Modifier
  param2: Type
  bg1: 1
  bg2: 1
  bgee: 1
  iwd1: 1
  iwd2: 0
  pst: 1
  doc: |
    Applies the modifier value specified by the 'AC Modifier' field to the category specified
    by the 'Type' field.

    Known values for 'Type' are:
      - 0   All
      - 1   Crushing
      - 2   Missile
      - 4   Piercing
      - 8   Slashing
      - 16  Base AC setting (sets the targets AC to the value specified by the 'AC Modifier' field.

    If the targets AC is already 'AC Modifier' or below, this effect will do nothing).

    Each modifier type to AC from this opcode is capped to the range [-20, 20]. Each AC type total is capped to the range [-32768,32767].

  notes:
    - |
      IWD1 and PST use a slightly different version. The "Base AC" sets to **field - 1** instead.
    
      IWD2 uses different parameters altogether.

[ALIENQuake]

The "Known values" can also be one of yaml key with values, right?

[lynxlynxlynx]

The notes are sometimes interspersed in the description, not always at the end. And there can be several of different severities, so the proposed format is not good enough. I also see no way to avoid needing to clean up the descriptions on the user side of things, eg. to get rid of broken links.

And I'm definitely opposed to cramming them all into one file. That's a simple step users can do if they need it.

[burner1024]

The "Known values" can also be one of yaml key with values, right?

Right, but I'm not sure if will be possible to parametrize them and keep current html layout unchanged. And I don't know how I'd use this data yet. If anyone needs it, they are welcome to chime in.

The notes are sometimes interspersed in the description, not always at the end. And there can be several of different severities, so the proposed format is not good enough. I also see no way to avoid needing to clean up the descriptions on the user side of things, eg. to get rid of broken links.

Severity is easy to deal with, adding a separate stanza (warning, important) should be enough.
Being interspersed is harder. Not sure if there's a way to parametrize this properly. I'll search around, if not, I guess keeping the doc monolithical will have to do.

Could you clarify the bit about broken links? Which ones do you mean, why are they broken?

And I'm definitely opposed to cramming them all into one file. That's a simple step users can do if they need it.

Jekyll can read data from dirs, something like _data/opcodes/0.yml. Though it does start to sound like simply moving/renaming opcodes dir, but I'm trying to work out something that'll work for actions and stuff with minimal changes later.

[lynxlynxlynx]

Links: opcodes link to each other via anchors, so you'd end up with broken links unless you replicated the naming and therefore layout (all in one file).

Actions and triggers won't be much better; except for less data, the same problems apply.

[burner1024]

Depending on how this works out, links might be kept and lead back to IESDP:

Maybe it's better to start with something else than opcodes, indeed. Here, I took a shot at actions. Sample data is stored in the data files.
I wanted to avoid duplicating info (action numbers in both filenames and files themselves, etc), but liquid is not flexible enough, so the overall system is almost the same as opcodes, just doesn't require setting game to 0 to filter it out (can be just skipped).
You can launch it locally, check out BG1 and IWD2 action pages.

[lynxlynxlynx]

Looks good. From what I can tell, action descriptions only use colors (can be replaced) and links (markdownify takes care of that?) on top of what's shown above.

[burner1024]

Loaded BG2 actions.
There's minor difference in styling, mostly because markdownify wraps everything into paragraphs. I made adjustments, but didn't go out of my way to make it exactly the same before getting feedback.
(One thing I did fix intentionally is ugly codeblocks)

If all's good, next step would be to load actions from other files, run comm/md5 to find and delete the identical ones, then some diffs to find and combine those that only differ in wording, then move all the rest as variants.

[lynxlynxlynx]

Sounds good, but let's resolve #85 first, so it's clearer how some of the more interesting actions turn up (since you skipped a few).