What is a reference implementation?

[benjagm]

Context

In the last OCWM #907 , we discussed about what project to fund in the future and there was an idea to fund Reference Implementations. But what is a reference implementation?

We need to come up with a definition that clearly defines what a reference implementation is and complement it with clear eligibility criteria, commitments, and quantitative compliance data from Bowtie.

Some definitions:

• Wikipedia

Expectations for this issue:

• Provide definition
• Provide the eligibility criteria:
  • What makes those implementations different from others?
  • Can we have more than one by language?
  • Are they committed to implementing experimental capabilities?
• Define the bowtie compliance value

Next steps:

• Agree on the criteria.
• Highlight reference implementations.
• Support implementations:
  • Funding.
  • Promotion.
  • Others.

[jviotti]

I think the options that were discussed in the call were:

• Either the implementations made/maintained by the TSC, under the argument that a TSC member either knows what they are doing, or if a non-TSC member really knows what they are doing, they will likely get elected into the TSC
• Or implementations with a perfect or close to perfect score in Bowtie

I personally like option 1 more. The official test suite is not super vast (you can still get it to 100% coverage with various issues), and if we consider reference implementations to serve as a reference of how things should be done, there is no way that thought leadership will come from outside the TSC.

[gregsdennis]

if we consider reference implementations to serve as a reference of how things should be done

A reference implementation isn't an example of how things should be done. To the contrary, generally reference implementations are simplistic examples that serve as explanations of concepts without optimizations. 100% accuracy is their goal, not performance, and most cases they are not viable for use in production environments because of this.

[benjagm]

What could the requirements be?

• XX% bowtie coverage in the modern drafts?
• Implements the experimental capabilities?
• Provide specific documentation to guide/support other implementers?
• Member of the TSC?

[jdesrosiers]

A reference implementation isn't an example of how things should be done. To the contrary, generally reference implementations are simplistic examples that serve as explanations of concepts without optimizations. 100% accuracy is their goal, not performance, and most cases they are not viable for use in production environments because of this.

I was thinking of it as a reference of how things should behave, not how things should be implemented. These are the implementations that people go to to explore how things are supposed to work. For someone in the JavaScript ecosystem trying to understand how dynamic references are supposed to work, we'd recommend they use @hyperjump/json-schema as a reference rather than ajv that has bugs in its dynamic reference implementation that don't look like they'll ever be fixed.

Maybe "reference implementation" isn't the right term. Maybe we just need to define clearly what that means to us in this context.

[jdesrosiers]

What could the requirements be?

XX% bowtie coverage in the modern drafts?

I don't think requiring a specific % coverage is necessary. I think coverage is just a factor that the TSC will take into account when voting on an implementation. Some tests are more important than others. For example, we have tests for things that should never appear in a real schema despite being a valid schema. If those tests don't pass it's not a big deal, but if a single test for a common behavior fails, that's a problem.

Implements the experimental capabilities?

Yes! It should be a requirement that they not only implement the latest draft, but are active in implementing and keeping up with proposed new keywords and other features.

Provide specific documentation to guide/support other implementers?

I'm not quite sure what that would mean. I would expect these maintainers to be active in supporting other implementers such as answering questions in the #implementers Slack channel, but I think this could more generally be a requirement for being active in the community.

Member of the TSC?

Yes! That doesn't necessarily mean that any implementation maintained by a TSC member should be considered a reference implementation, just that being a TSC member is a prerequisite to be nominated. The main reason I think this requirement is important is because it shows they are active in the community and that they are likely to keep up with the latest developments. If they have a great implementation, but aren't willing to be involved with the TSC, I don't think they're like likely to keep up with the latest developments.

This requirement isn't about gatekeeping. Anyone with those qualifications and is not yet a part of the TSC would be invited to join. But, it will look like gatekeeping from the outside so we might need to be careful how we say it.

[karenetheridge]

100% accuracy is their goal, not performance

Agreed. A reference implementation can be 100% compatible with the spec and be incredibly slow, and possibly not full-featured. I can't imagine that an implementation in, say, Prolog would be very fast, nor worthy of putting into production anywhere, but it would still have educational value.

For example, we have tests for things that should never appear in a real schema despite being a valid schema. If those tests don't pass it's not a big deal

Disagree. If it's not a big deal, it shouldn't be part of the acceptance test suite. We should only be testing for things that are in the specification and therefore we expect to be functioning properly. The only exceptions I could see here are with format implementations (whose status we already acknowledge by placing in the "optional" directory), which are specified externally, sometimes with a large number of requirements, where we don't expect implementations to re-implement themselves, therefore are dependent on external libraries where 100% compliance isn't always available. Do you have a particular example you were thinking of here?

But, it will look like gatekeeping from the outside

That's because it is gatekeeping, and I don't think it's valid. What if I resigned from the TSC tomorrow because of time commitments? Does that mean my implementation would be automatically ineligible for consideration (or removed from the list if we already had one), even though it's up to date and I continue to maintain it? TSC membership would be desirable, and is bound to be highly correlated, but I don't think it should be mandatory.

[jdesrosiers]

Agreed. A reference implementation can be 100% compatible with the spec and be incredibly slow, and possibly not full-featured. I can't imagine that an implementation in, say, Prolog would be very fast, nor worthy of putting into production anywhere, but it would still have educational value.

I'm thinking more and more that "reference implementation" isn't the right word because although you're right according to the typical understanding of the term, I don't think that's what this discussion is about. I almost objected to the use of the term in the meeting, but I let it go.

This is about implementations that are commonly used as a reference to explore and understand JSON Schema features and capabilities. How it's implemented, what language it's implemented in, or performance characteristics aren't really relevant. So, "commonly referenced" is a key characteristic. It's not just any implementation that has good test coverage, or stays up-to-date, or has some educational value. It's more about implementations that have a high level of importance to the community.

The reason I'm not considering just any good implementation is because the point of this exercise is to determine which implementations we should be donating to. We don't have the funds to give to everyone who deserves it, so we should be prioritizing those that have the most impact. We can expand on that when we have more funds to do so, but I think that's the place to start.

If it's not a big deal, it shouldn't be part of the acceptance test suite. We should only be testing for things that are in the specification and therefore we expect to be functioning properly.

There are things that are in the specification that should never appear in a real schema even though they are technically valid. One example is a dynamic reference with a JSON Pointer fragment. This is technically valid because dynamic references fall back to behaving like references when they don't find a matching dynamic anchor, but that's not a proper way to use a dynamic reference and shouldn't appear in a real schema. If it does appear in a schema, it's always a mistake. If an implementation doesn't pass that test and passes all tests for proper use of dynamic references, it's not a big deal. I would have no reservations whatsoever about using that implementation or recommending that implementation to others.

There are a couple other cases I could mention, but I don't want to derail this discussion.

What if I resigned from the TSC tomorrow because of time commitments? Does that mean my implementation would be automatically ineligible for consideration (or removed from the list if we already had one), even though it's up to date and I continue to maintain it?

I see what you're saying, but I see things a little differently. Remember, this is about determining which implementations get funded. I don't think it's unreasonable to make participating in the community a requirement to receive funding. At least while our funds are so limited, I think we should be putting them where they have the most impact and that means the people who are actively participating. I think we should be prioritizing funding those who are driving the evolution of JSON Schema over those who are just reacting to change. When the days come where we have more to distribute, we can give more and expect less, but I don't think that's where we are right now.