Skip to content

Latest commit

 

History

History
93 lines (73 loc) · 3.08 KB

CONFORMANCE_TESTS.md

File metadata and controls

93 lines (73 loc) · 3.08 KB

Swift Protobuf Conformance Tester

Google's protobuf project includes an extensive "conformance test suite" that exercises various encoding and decoding features of protobuf implementations to help ensure that they are all interoperable.

SwiftProtobuf currently passes Google's entire conformance test suite. We have integrated the conformance test with the SwiftProtobuf test suite to help ensure that we remain conformant in the future as well.

If you have Google's protobuf source code on your system, then a simple environment variable setting will run the conformance test as part of every make test. We strongly recommend anyone making changes to SwiftProtobuf run these tests regularly.

Preparation

The conformance test suite requires Swift 3.0, standard command-line tools such as make and awk, and a full source checkout of Google's protobuf project.

The Makefile assumes by default that the protobuf project is checked out in an adjacent directory. If this is not true, you should set the GOOGLE_PROTOBUF_CHECKOUT environment variable to the full path of the protobuf checkout.

Running the Tests

If the Makefile can find the protobuf project sources, then the following should suffice:

$ make test

This will build and run the SwiftProtobuf test suite, verify the code generator, and finally run the conformance tests.

You can also run the conformance tests by themselves:

$ make test-conformance

Test Environment

The Makefile at the root of the SwiftProtobuf project has the following lines, which specify how to run the installed protoc program on your system, and where to find the Google protobuf source tree:

PROTOC=protoc
GOOGLE_PROTOBUF_CHECKOUT?=../protobuf

If these do not match your system, you can specify one or both on the command line

$ make PROTOC=[path] GOOGLE_PROTOBUF_CHECKOUT=[path] test

If you have problems

The most common problem area is building Google's conformance host program. You may find it easier to switch to the directory where you have checked out Google's protobuf sources and build the host program manually:

$ cd protobuf
$ ./configure
$ make -C src
$ make -C conformance

You can then manually run the conformance test using the following commands:

$ cd swift-protobuf
$ ../protobuf/conformance/conformance-test-runner --failure_list failure_list_swift.txt .build/debug/Conformance

Known Issues

At this writing, all of the conformance tests succeed.

Report any issues

If the conformance test prints out any "unexpected failures", please look in the Github Issues to see if the problem you're seeing was already reported. If not, please send us a detailed report, including:

  • The specific operating system and version (for example, "macOS 10.12.1" or "Ubuntu 15.10")
  • The version of Swift you have installed (from swift --version)
  • The version of the protobuf source code you are working with (look at the AC_INIT line near the top of configure.ac)
  • The full output of the conformance test run, starting with "CONFORMANCE TEST BEGIN"