Adds a per-level persistence system.

[MistakeNot4892]

Description of changes

• Adds /atom/Serialize() to return an assoc list of relevant values suitable for saving out.
• Adds /atom/Preload() and /atom/PreloadData() and hooks in SSatoms flush to deserialize atoms during flush.
• Adds /datum/level_data/load_persistent_data() and /datum/level_data/save_persistent_data() as entrypoints for per-level persistence.
• Adds a fire() override to SSpersistence to do a periodic level data save.
• Converts the innards of the previous persistence system to use the new save/load handlers and procs.

General flow of level persistence:

• Saving:
  • SSpersistence periodically iterates the z-level list, finds levels that want to serde, and calls save_persistent_data()
  • Levels return a list of instances to get_persistent_instances(), instances have Serialize() called and return a list of modified fields.
  • Fields are serialized (to JSON with the default handler) and written to disk.
• Loading:
  • SSmapping initializes and calls preload_persistent_data() and load_persistent_data() on relevant /datum/level_data z-level objects.
  • load_persistent_data() creates the base instances and (for atoms) sets __init_deserialisation_payload with the data loaded from tile.
  • SSatoms flush calls Preload() on all deserialized atoms which pre-populates vars on the atom.
  • Ssatoms proceeds to Initialize() atoms as normal.

TODO

• Implement area serde.
• Implement datum serde.
• Implement non-turf loc for serialized atoms.
• Test with implementation on tradeship.
• Test that wall construction serializes.
• Test that wall removal serializes.
• Test that removing flooring serializes.
• Test that adding flooring serializes.
• Test that roofing turf serializes.
• Test that changing turf height serializes.
• Get simplemob serde to work.
• Test legacy persistence:
  • Graffiti
  • Filth
  • Trash
  • Paper
  • Books
• Implement some kind of grandfather check to convert old persistence format to new.
• Test legacy migration.
  • Graffiti
  • Filth
  • Trash
  • Paper
  • Books
• Implement first-run level generation (suspend level gen if we initialized a level via serde)
• Test level generator reordering (split into 2nd PR?) Reverted these changes.

Future work

• Completely integrate /decl/persistence_handler into this system instead of the bespoke serde on SSpersistence. ended up largely doing this in this PR.
• Add a bespoke system for serializing area changes (blueprints etc) on a z-level.

Why and what will this PR improve

Adds a framework for handling persistence in the future.

Authorship

Myself.

Changelog

Nothing player-facing.

[MistakeNot4892]

Depends on #5175

[out-of-phaze]

i think you'd have to actually add if(_persistent_save_running) return hooks into move and other client-driven/verb things to make it block anyway, is the thing. i'm not sure waitfor = TRUE would be enough. so it'd only be a partial solution either way

[MistakeNot4892]

We could potentially suspend all subsystems and kick people back to the lobby during saves, maybe? Similar to how DMMS suspends fluids and such until the run finishes.

[MistakeNot4892]

I'm going to call datum and area serde out of scope for this PR, I don't have any clue how to handle them in a clean way.

The atom side of this appears to be working fine, I've tested with a debug item and it all seems pretty good.

EDIT: Went ahead and just let it instance datums with the serde data as a list arg, in case it's needed in the future.

[MistakeNot4892]

Slipped and accidentally added turf serde, limited testing on Shaded Hills but needs proper testing before I sign off on it.

[MistakeNot4892]

Depends on #5182

[MistakeNot4892]

Testing notes:

• shutter state needs to be serialized on walls
• fluids need to be serialized on floors
• flooring is busted

[out-of-phaze]

needs rebase

[MistakeNot4892]

Current outstanding issue with this one is that I cannot for the life of me seem to get map templates placed during init to flag the turfs as changed. Generating on Shaded Hills and restarting leaves template-shaped sections of mask turfs. Not sure what the issue is, it happened even after I made all ChangeTurf() calls and relevant sections of DMMS flag the turfs directly. I am assuming that something is newing /turf directly and not flagging.

[out-of-phaze]

Current outstanding issue with this one is that I cannot for the life of me seem to get map templates placed during init to flag the turfs as changed. Generating on Shaded Hills and restarting leaves template-shaped sections of mask turfs. Not sure what the issue is, it happened even after I made all ChangeTurf() calls and relevant sections of DMMS flag the turfs directly. I am assuming that something is newing /turf directly and not flagging.

does /area/fantasy/outside/point_of_interest have AREA_FLAG_ALLOW_LEVEL_PERSISTENCE? it doesn't seem so, that'd mean at least a few non-passthrough templates (chemistry shack etc) would fail to save

[MistakeNot4892]

does /area/fantasy/outside/point_of_interest have AREA_FLAG_ALLOW_LEVEL_PERSISTENCE? it doesn't seem so, that'd mean at least a few non-passthrough templates (chemistry shack etc) would fail to save

Genuinely no idea why I didn't think of this. Thank you :(

[MistakeNot4892]

This is now working as far as I can tell. Main things I am worried about in live testing is accidentally cooking legacy persistence data, but it should at least write out a backup before it does any kind of migration of that data.

I am not committing any changes that actually use this system in full yet, I'm going to save that for after we've done playtesting (probably via Pyrelight or maybe Scav)

[MistakeNot4892]

Realised belatedly that I need to get area serde in before this will work as intended, due to map templates setting area, but I will do that in a second PR, I don't want to fiddle with this one further currently.

[out-of-phaze]

i frankly really don't like that we still have the legacy persistence system, it really doesn't work for anything but papers and dirt (the generic 'filth' stuff was extremely annoying). but i guess some maps want a middle-ground re: persistence.

once this is in i'll try to make it faster

[out-of-phaze]

this could use /tmp/ possibly

[PsyCommando]

Yes please. I really wish /tmp was used more. It helps make transient vars a whole lot more obvious.