Need to support new fields in protobufs 33.5

[kamahen]

A numbere of fields have been added to the protobuf specification ... these need to be reviewed and the bootstrap re-run.

[kamahen]

Protobufs is now v34.0, so I'll use that as the target.

I have all the tests working (and I've mostly fixed the bootstrap process although it still requires a bit of manual tweaking), but the Makefile and READMEs are somewhat messy (I don't see a point in switching to cmake). If it's urgent, I'll make a PR now and tidy up later; otherwise, it might be over a month before I can tidy things up.

As a separate thing: does it make sense to add git@github.com:protocolbuffers/protobuf.git as a submodule? This would make it easier in future to keep things up-to-date (and check on what has changed), although I had problems building it, so I just downloaded the binaries from the releases page.

[kamahen]

I should add that the current version of protobufs works fine with protoc V34.0, provided the .proto file doesn't contain any of the new features in the protobuf in the .proto file. So, there's probably no rush to update contrib-protobufs except for one of the tests not passing (and that test, amongst other things, does a sanity test that all the .proto features can be handled).

If you want to see what I've done so far: master...kamahen:contrib-protobufs:protoc-34.0

[JanWielemaker]

That looks like a decent amount of work. Just finish it when you find the time as there is no rush. I hope that the bootstrap.sh is not part of the regular build procedure? The regular build should only use CMake, Prolog and a C(++) compiler to remain portable.

[kamahen]

The bootstrap in the contrib-protobufs submodule is only needed when there's a new version of the protobufs compiler (protoc) and that new version adds some features that causes an "unknown feature" error from the current plugin (e.g., the latest one added the ability to specify the edition in a .proto file); as far as I can tell, none the new protoc features affect protoc-gen-swipl (the SWI-Prolog protobuf compiler plug-in.

There's a separate test called bootstrap which simply checks what's in the bootstrap directory; it currently uses the system version of protoc (as installed by apt install protobuf-compiler python3-protobuf), which might not be the latest. (The tests are slightly problematic in that they create files in the source directory rather than in the build directory; this can be fixed but it's a bit of work and I'm not sure it's worth spending the time.)

As to adding https://github.com/protocolbuffers/protobuf.git as a submodule and whether we can use "protobufs from the system" ...

The problem is that the system protobufs compiler (protoc) can be older than the latest release and will therefore fail to compile the latest release's .proto files. For example, on my Debian system, protoc is 3.21.12 but the latest release is 3.40.0. To do the bootstrap (to add features from the latest release), I need to access a few files from the repository (descriptor.proto, plugin_in.proto) plus the latest protobufs compiler and a few associated "include" files.

[It's possible that I could relax this by adding some kind of handler for skipping unknown fields; but currently, the code throws an exception if it encounters an unknown field - which is why the bootstrap test failed: it tries to compiler the descriptor.proto, plugin_in.proto as a sanity test.]

The descriptor.proto and plugin.proto files are only in https://github.com/protocolbuffers/protobuf.git ; if I don't want to build the compiler from the sources, the compiler binary is in the release zip file (e.g., in the "Assets" section of https://github.com/protocolbuffers/protobuf/releases/tag/v34.0).

Right now, I'm using a clone of https://github.com/protocolbuffers/protobuf.git plus https://github.com/protocolbuffers/protobuf/releases/download/v34.0/protoc-34.0-linux-x86_64.zip

I can continue to do things this way - anyone who wants to update the protobufs package will need to manually clone the repository and download the zip file. I thought that I could simplify things a bit and also keep track of the protoc version by doing something similar to how the rocksdb pack adds the Facebook repository as a submodule (and I could probably build the protoc binary from that submodule, so it wouldn't be necessary to download the zip file). But if this is a problem, then I can just document how to clone the repository and use it.

BTW, bootstrap.sh is only used during the bootstrap (which is only needed when there's a version change of protoc) and not during any of the tests. I need to clean it up a bit, but it's mainly an aide-memoire for the steps needed to do the bootstrap.

I could instead add the two .proto files from the source repository, plus the Linux executable and its associated "include" files into contrib-protobufs repository, but I wonder if that's a good idea; it's probably better to have instructions for doing the bootstrap by cloning the source repository and downloading the executable (or cloning the source repository and building the executable).

[kamahen]

Sorry for the long-winded comment ... I think I can get most of what I want by adding tidying up the boostrap.sh (perhaps renaming it), and uncommenting the wget (or maybe I should use curl) step. This only needs to be run if there's a change in the syntax of .proto files, which I'll need to document.

Is there a way to find out the build directory? (E.g., run cmake with some appropriate flag to do the protobuf-bootstrap, and access the build directory from bootstrap.sh) Similarly, changing the tests so that they don't write into the source directory requires knowing where the build directory is.

[JanWielemaker]

uncommenting the wget (or maybe I should use curl) step

I think curl is a bit more widespread. You can always have cmake checking for curl and raise a warning if you want to do the requested step and curl is missing. Or simply document it ...

Is there a way to find out the build directory?

Yes. cmake has 4 variables: CMAKE_CURRENT_SOURCE_DIR, CMAKE_CURRENT_BINARY_DIR, CMAKE_SOURCE_DIR and CMAKE_BINARY_DIR that point at the source and build dir. The CURRENT version for the subdirectory configured by the CMakeLists.txt and the non-CURRENT for the overall project (i.e., swipl).

[kamahen]

It appears to be fairly straightforward to change my bootstrap to use the cmake build directory instead of writing to the source directory (the way many Makefiles do), triggered by a small change to a download.sh script whenever Google's protobufs version changes. I might also convert my unmaintainable Makefiles to cmake, but not sure -- this would require learning a bit more about cmake.

The READMEs need extensive revisions -- they were incomplete and often wrong anyway (it's amazing how much I've forgotten about what I was doing a few years ago).

[JanWielemaker]

... And ideally code the scripts in Prolog 😄 . That is how I managed the scripts in the ssl package.

[kamahen]

It's easier to write simple bash scripts with the e and -o pipefail flags. :)
Anything more complicated would probably be best with cmake, which is somewhat like Prolog without variables (make also).
[I am currently reading the cmake tutorial.]