Supporting multiple versions of Minecraft

[avdstaaij]

The problem

There is a significant limitation (perhaps even design flaw) in GDPC: it cannot support multiple versions of Minecraft simultaneously. Or, differently said, new features in the library cannot be used when writing a generator for an older version of Minecraft, and older versions of the library (and thus old generators) cannot be used with newer versions of Minecraft.

There are two causes of this limitation:

Cause 1: GDPC depends on a minimum version of GDMC-HTTP, which has the same fundamental limitation: Each version of GDMC-HTTP only supports one specific version of Minecraft, with neither backwards nor forwards compatibility.

Cause 2: GDPC interacts with "Minecraft data" (blocks, entities, items) in a few places (see e.g. #98). I attempted to limit this as much as possible, but couldn't fully eliminate it. Currently, I believe these places are:

• The functions to rotate and flip individual blocks.
• The utility functions to generate NBT data (signData/placeSign, bookData, etc.).
• The WorldSlice constructor.

In an ideal world, it would be possible to write a generator using the latest version of GDPC and apply it to any out of a range of supported Minecraft version, and to take an old generator and to apply even to newer Minecraft versions, but the limitations above prevent this.

Why cause 1 must be solved first

I believe that adressing the minecraft data interactions (cause 2) is not useful, or has very limited use, before either GDMC-HTTP becomes backwards compatible with Minecraft versions or GDPC becomes backwards compatible with GDMC-HTTP versions. Abstractly, the reason is as follows: if GDPC inherently supports Minecraft versions X through Y, but requires GDMC-HTTP version >= Z, and GDMC-HTTP version Z requires Minecraft version Y, then by transivitiy, GDPC still only supports Minecraft version Y. There is no use to supporting Minecraft versions that are not supported by the minimum required version of GDMC-HTTP.

For a concrete example, consider issue #98. Issue #98 is about the sign helper functions like placeSign. The NBT data format for signs changed in Minecraft 1.20. If these functions would be updated to the >=1.20 format, they would no longer support <1.20 (cause 2). In #98, it is suggested to change the sign helper functions to check the version of Minecraft that is currently in use (this is not quite trivial, as I will explain under "Solving cause 2", but not impossible), and change their behavior depending on it. In principle, this is a good idea. However, consider the following. GDMC-HTTP 1.4.0 requires Minecraft 1.20.2, and later versions of GDMC-HTTP thus far have only supported later versions of Minecraft. So, if we want to implement features in GDPC that require functionality from GDMC-HTTP 1.4.0 (such as the improved withinBuildArea support), GDPC will have to require GDMC-HTTP 1.4.0, and thus, by transitivity, Minecraft 1.20.2. So the behavior switch in the sign helpers would become useless.

Solving cause 1

Cause 1 can be solved from either the GDMC-HTTP side or the GDPC side.

The GDMC-HTTP side

To solve cause 1 on the GDMC-HTTP side, GDMC-HTTP needs to become backwards compatible with regards to Minecraft. GDPC could then simply depend on the latest version of GDMC-HTTP. One way to do this might be for there to be versions of the mod for a range of Minecraft versions instead of just one: "GDMC-HTTP 1.4.0 for Minecraft 1.20.2", "GDMC-HTTP 1.4.0 for Minecraft 1.19.2", etc. By solving this on the GDMC-HTTP side, everyone who uses GDMC-HTTP would benefit, not just those who use GDPC.

I do not know how possible this is or how much work it would take. @Niels-NTG, I would like to hear your input on this!

The GDPC side

To solve cause 1 on the GDPC side, GDPC needs to become backwards compatible with regards to GDMC-HTTP. It might be possible to do this: we could introduce an abstract class Interface, and add multiple implementations such as GDMC_HTTP_1_3_3_Interface, GDMC_HTTP_1_4_0_Interface, etc. This would even allow things like AmuletInterface, so it may be beneficial regardless. All features that require interface support (such as build area enforcement) would then move to these concrete interface classes, with some classes not supporting all features and perhaps logging warnings when this is the case. However, I am not entirely sure if all interface-dependent features can be so easily grouped together.

Solving cause 2

Once cause 1 is solved, we can consider solving cause 2. To reiterate, I believe the places where GDPC interacts with Minecraft data are:

• The functions to rotate and flip individual blocks.
• The utility functions to generate NBT data (signData/placeSign, bookData, etc.).
• The WorldSlice constructor.

WorldSlice

First of all, WorldSlice. This one might be the most problematic, since it essentially parses Minecraft's entire world format. I think the best way to make this Minecraft version-independent is to completely get rid of the GET /chunks call and instead rely on higher-level GDMC-HTTP calls. GDMC-HTTP now supports getting 3D regions of blocks and biomes, and there is now a /heightmaps endpoint, so this could very well be possible. If we don't switch to higher-level calls, there is no way to get around maintaining multiple implementations of the WorldSlice parser for various versions of Minecraft.

Blocks, entities and items

The other two parts, rotating/flipping blocks and NBT helpers, are fairly similar. They both deal with blocks, entities or items.

In #98, it was suggested to make a NBT helper check the active Minecraft version and switch on it. This approach could in principle be used by all NBT helpers and block transform functions. However, it has a few disadvantages:

• Currently, these functions do not actually involve GDMC-HTTP at all. I would prefer to keep it that way, since it allows you to use them in situations where there's no Minecraft instance, such as when creating a model/schematic for later use. Making them require GDMC-HTTP access also means that the functions that didn't require an Editor instance now will, which will cause problems with cyclic imports and will certainly cause breaking changes in a lot of places.
• This approach won't help users who manually specify blocks/entities/items instead of using helper functions like signData.
• This won't allow us to support future Minecraft versions, though there are few approaches that will.

I think the ideal place to do version-conversions for Minecraft data is right before sending to or after receiving from GDMC-HTTP. This way, you capture all occurences of Minecraft data that needs to be converted, without requiring breaking changes anywhere else.

It seems that Amulet learned from the mistakes of MCEdit, because Amulet Core has a really clever solution to the version-conversion-problem in the form of PyMCTranslate. It uses a "universal format" for blocks (and I believe entities and items as well) which should not be directly used and can therefore be changed without breaking compatibility. When you read or write a block, you specify the Minecraft version you want to work in, and PyMCTranslate translates between that version and the universal format. And while I'm not sure about this, I believe this architecture even enables support for future Minecraft versions, since you could update PyMCTranslate without updating the rest of Amulet Core.

I think it would be possible to utilize PyMCTranslate in GDPC by using it to convert to or from the current Minecraft version right before interacting with GDMC-HTTP. The only problem is that I'm unsure if this is allowed by its license, specificially its non-conpetition restriction. GDPC with GDMC-HTTP might fall under "competition" as described there. But if we could use it, this seems like it would be an ideal solution.

Summary

To support multiple versions of Minecraft with the same version of GDPC, the following is needed:

1. GDPC must become backwards compatible with regards to GDMC-HTTP, or GDMC-HTTP must become backwards compatible with regards to Minecraft. As long as this is not the case, supporting multiple Minecraft versions internally in GDPC is of little use.
2. GDPC must internally support multiple versions of Minecraft, which requires fundamental changes in its block rotation/flipping system, its NBT data utility functions and in WorldSlice.

[Niels-NTG]

Looking back at the process from updating GDMC-HTTP from Minecraft 1.19 to 1.20, it doesn't seem possible to support multiple versions of Minecraft simultaneously in the same distribution. The jar files are compiled for a specific Minecraft-Forge version.

Having said that, I don't think there is anything fundamentally different between the APIs of these versions of Minecraft that would prevent me from back-porting the features I've created for GDMC-HTTP 1.4 (Minecraft 1.20) to Minecraft 1.19. The code relies a lot on code that's internal to Minecraft, so writing a wrapper so I could compile two versions simultaneously isn't trivial. Still, I do know that other mod developers have done this. They often even support very different mod frameworks such as Fabric + Forge at the same time using wrappers for either and a core of shared code.

To keep it simple I could start with creating a branch from my current 1.20 version and start back-porting all the features I've created to Minecraft 1.19. The user will still need to install a specific version for their version of Minecraft, which I will likely indicate by suffxing the files with the Minecraft version (eg. gdmc_http_interface-1.5.0-minecraft-1.20.2.jar, gdmc_http_interface-1.5.0-minecraft-1.19.2.jar). Currently I'm not in a position to say what my timeline on this would be. It's tempting to start working on this right away, but I've papers to write ;) .

On the topic of the NBT method: I do believe that working with the NBT format should be encouraged. It's a universal format that can contain both blocks and entities. And has the DataVersion attribute, which Minecraft can use to convert the NBT data to a format of the its current version of the game. However, I do agree that understanding this format shouldn't be a requirement for the end user, nor should client programs such as GDPC be required to be built upon NBT data structures. Being able to place blocks and entities with an easy to understand JSON format is important for making GDMC-HTTP easy to use.
Instead I would like to design a new endpoint for some future release that allows the user to mix blocks and entities within the same payload, which then GDMC-HTTP packs into a single NBT structure placement instruction. In addition the user would be able to set a target DataVersion for all requests, which would would GDMC-HTTP instruct to convert that data to the current Minecraft version when receiving write requests. But also when GDMC-HTTP returns data to the client it is converted to the data version specified by the user.

[avdstaaij]

Thanks for reading that wall of text ;). Having separate GDMC-HTTP jars with the same API but for different Minecraft versions is exactly what I had in mind for "Cause 1 on the GDMC-HTTP side"! If that's maintainable, I think it would be the best solution. You will get more and more jars in each release, though.

Also, don't worry, there's no need to hurry with this. The GDMC competition uses a set version anyway, and if the GDMC-HTTP API remains backwards compatible, multiple-Minecraft-support will even work retroactively!

What do you mean with "the NBT method" and "the NBT format"? For blocks, I don't intend to discourage using NBT data (it's just one of the three components, along with ID/name and states). There's no DataVersion tag for those, though. NBT is just a generic data format, some formats that use NBT have a DataVersion field (like structure files), some don't (like schematic files).

The potential new endpoint you describe sounds very interesting! If you can let Minecraft itself do all the conversions, then that would greatly help with "Cause 2" as well. Can Minecraft convert in both directions (new->old and old->new)?

[Niels-NTG]

I've started to see if and how I could back port the currently latest version of GDMC-HTTP (1.4.4-1.20.2) to work with Minecraft 1.19.2 such that there is complete feature parity no matter the Minecraft version. I'm happy to report this was relatively painless and from some initial testing it looks it behaves the same as 1.4.4 as well, including the huge performance boost introduced with version 1.4.0.

I think I will be able to release a 1.5.0 update soon. This will be especially useful going forward when we want to include future Minecraft versions like the upcoming 1.21.

Sorry for the confusion. I didn't explain what I meant with the "NBT method" very well. What I'm basically saying it that sending and receiving (S)NBT-formatted structure files via the POST /structure and GET /structure endpoint in GDMC-HTTP already allows for placing/reading both blocks and entities in a single action, very much alike my proposed PUT /place endpoint. The main downside of this approach is that it requires the client to understand the NBT format, which isn't as easy reading and writing JSON.

When you're working with an older Minecraft version it cannot convert data to a version newer than it, so there are limitations.

[avdstaaij]

This sounds very good!

If Post /structure and GET /structure are more performant than the JSON endpoints, GDPC could switch to them under the hood. The communication with the HTTP interface is abstracted away.

When you're working with an older Minecraft version it cannot convert data to a version newer than it, so there are limitations.

That is unfortunate. That means this wouldn't work for the "new generator, old Minecraft" scenario. But it's better than nothing.

[Niels-NTG]

Update from my side: with update v1.4.6 GDMC-HTTP now has full support for both 1.19.2 and 1.20.2 versions of game!

[avdstaaij]

Fantastic! Then I can go look into the GDPC side (cause 2). The changes will certainly be breaking, so this will be slated for 8.0, probably after this year's competition.

I've been given permission by the developers of Amulet Core to use it in GDPC for Minecraft data conversion, which will help greatly. It will however come with some usage limitations for GDPC, since GDPC will then inherit the usage limitations from Amulet (Core)'s proprietary license: it can't be used commercially and can't be used in a system that competes with Amulet. I feel like these limitations are worth the benefit, however: Amulet's data conversion system is well-maintained, and, due to its use in Amulet, very well-tested.

The design I currently have in mind is as follows:

• Editor.__init__ will gain a required parameter minecraftVersion, which it will store. This will be the "target version" for the generator, which may differ from the "active version" - the version of the world being edited. Right before writing to or right after reading from the world, Editor will convert all Minecraft data between the two versions using Amulet Core's translation system.
• All functions that interact with or generate Minecraft data (block transformation, signData, etc.), will also gain a required parameter minecraftVersion. They will then interact with/generate data in that version of Minecraft, probably internally using Amulet Core gain so they don't need a separate implementation for each version.
• For convenience, the functions above will also gain equivalents that take an Editor parameter instead of a version one. They will then use the version stored in the editor.
• Functions that already require an Editor instance (e.g. placeSign) won't require API changes.