Stronger emphasis on reaching Recommendation

[plehegar]

[raised by the 2020 AC Review]

In the interest of broader interoperability on the web we'd like to see more emphasis placed on groups taking specs to Rec status rather than being maintained perpetually in CR. We suggest that this is prioritized in the next process iteration.

When a specification remains perpetually in Candidate Recommendation there seems to be no requirement that issues raised during horizontal review are addressed (as would be required before transitioning to Recommendation).

Related issues: #402 and #406

[michaelchampion]

When a specification remains perpetually in Candidate Recommendation there seems to be no requirement that issues raised during horizontal review are addressed (as would be required before transitioning to Recommendation).

There are several perma-threads tangled up here and it's hard to resolve them separately:

1. Traditional Recommendations get wide / horizontal review at one point in time, but traditionally are not revised as problems are discovered and the rest of the world changes. So the Recommendation does not define how to get broader interoperability on the web.

2. That led to Living Standards / Perpetual CR, where the WG accepts that it's not worth their time to try to make a snapshot authoritative given that review doesn't catch all bugs and external reality changes faster than the Process can move. They focus their energy on ensuring the current draft fixes known problems rather than creating a snapshot and getting it horizontally reviewed.

3. But with a Living Standard, there's no obvious chokepoint at which to insist that wide/horizontal review feedback is taken seriously by a WG.

So if you want to insist on an HR chokepoint, you should also insist that the process emphasize perpetual maintenance. Maybe Recommendations auto-expire after x number of years? Maybe adopt the WHATWG innovation of putting "support tables" inline in specs that show the reader the current results for interoperability (+ A11Y, I18N, etc.?) tests in various implementations, and let the reader assess how well the spec is implemented and maintained.

But more fundamentally, the W3C Process can't force the external world to value the Recommendation label enough to use it as a way to pressure a group to do the extra work needed to achieve it. The browser engine monoculture exacerbates this problem because the way things work in all the Chromium browsers + iOS Safari is the de facto standard whatever W3C says.

My personal takeaway: People in the W3C community should invest less of their brainpower in tweaking the formal Process Document, and more energy into improving the informal culture. If a WG is not addressing issues raised in horizontal reviews, that's a COMMUNICATIONS problem more than a Process problem.

• Maybe the WG doesn't understand the feedback well enough to act on it
• Maybe the people affected by the problem aren't telling the WG and implementers that it's important to them
• Maybe the WG isn't listening their stakeholders who suffer from the problem
• Maybe the horizontal review group isn't addressing a problem real stakeholders have, so the feedback is being politely ignored by everyone
• and so on.

[nigelmegitt]

Maybe Recommendations auto-expire after x number of years?

In my view this is the wrong way around. Instead, I made a proposal at #406 (comment) to make CRs expire on inactivity, with ways to make it fairly easy for WGs to decide the timeout period and to do refreshes to reset the clock.

[michaelchampion]

Can we agree that unmaintained Recommendations and CRs should lose whatever claims to authoritativeness they possess?

"Maintained" in some rare cases might mean that a qualified group has re-assessed the spec and determined that there are no issues to fix and that it still meets the need for which it was crafted. But in general, if a spec isn't getting continually updated it's becoming irrelevant.

But fundamentally, what real problem is "stronger emphasis on reaching Recommendation" trying to fix? I don't think it's "broader interoperability on the web", because web interoperability is getting better at the same time "emphasis on reaching Recommendation" is declining, and it's generally accepted that the current WD or even latest GitHub commit is the version developers should reference.

[chaals]

@michaelchampion wrote

Can we agree that unmaintained Recommendations and CRs should lose whatever claims to authoritativeness they possess?

That depends...

The only claim to authorative that a CR has is that the Working Group believes it has described an appropriate solution to a problem, in a way that could be used as a basis for interoperable development. Given the many structural barriers (most particularly language and timing) that limit the participation in a Working Group, the broader review and the checking that the WG solution works at least for those who are interested in implementing at the bleeding edge is a useful reality check.

If the CR is neither updated (perhaps just to say "status - same as before"), nor becomes a Recommendation, it seems reasonable to ask whether something is going wrong - has the Working Group changed its mind about something, is there just not enough information to know if their proposal is actually any good, or is it just that having laid out a solution for the benefit of the Working Group they don't have enough motivation to clarify this for the rest of the ecosystem by moving the document along the process.

The latter problem is something we could work out how to address. Given that W3C has a competent techincal staff, perhaps we should be asking them to do more of the final cleanup that results in a spec that is ready actually becoming a Recommendation.

For a Recommendation, we can probably agree that where "unmaintained" means that there are known errors that have been brought to W3C's attention, it is important that we have a mechanism to update the published Recommendation that actually works, whether the changes are small ones or really a new version with significant improvements, new or re-designed features, or the like.

My reluctance to specify a particular rule for this is based on the fact that there are different kinds of recommendation, written differently and operating in different environments. The XML Recommendations are very widely used, and while it might be helpful to have some minor corrections, it seems unlikely for the most part that we need a way to write a complete new version of XML - it seems that the market has divided into those who are basically happy with what there is, and those who want something new and shiny (like curly brackets and the opportunity to invent new approaches to internationalisation each time you make a vocabulary, instead of having a standard way of making it work). On the other hand, it seems plausible that the specification of e.g. like Verifiable Claims will evolve, for example to include details of a protocol allowing interoperability.

None of this seems to militate against reaching Recommendation, but instead suggests that we need to be clear about the ways we do maintenance.

[frivoal]

When a specification remains perpetually in Candidate Recommendation there seems to be no requirement that issues raised during horizontal review are addressed (as would be required before transitioning to Recommendation).

Issues, including those raised by Horizontal review, are expected to be addressed in order to enter the CR phase. To go from CR to REC, you need:

• implementations (generally proven by passing tests)
• an AC Review.

The rest, including issues from HR groups, is expected to be addressed earlier.

A few chosen bits from the Process supporting that (found both in the 2019 and 2020 versions of the Process):

Advancing to Candidate Recommendation indicates that no further improvement is expected without additional implementation experience and testing. A Candidate Recommendation is expected to be as well-written, detailed, self-consistent, and technically complete as a Recommendation, and acceptable as such if and when the requirements for further advancement are met.

To publish a Candidate Recommendation […]:

• …
• must show that the specification has received wide review

For all requests to advance a specification to a new maturity level other than Note: (note: these criteria are invoked in the transition to CR)

• must formally address all issues raised about the document since the previous maturity level.

When publishing an updated version of an existing Candidate Recommendation or Recommendation, technical reports are expected to meet the same maturity criteria as when they are first published under that status.

So, so far, at least on a first pass, CR, not REC, is where horizontal review issues (and any other issue) are expected to be addressed, and Process 2020 changes nothing about that.

However we also have this bit (which is also in both P2019 and P2020):

However, in the interest of replacing stale documents with improved ones in a timely manner, if flaws have been discovered in the technical report after its initial publication as a CR or REC that would have been severe enough to reject that publication had they be known in time, it is also permissible to publish an updated CR or REC following the usual process, even if only some of these flaws have been satisfactorily addressed.

The goal is that you're not stuck and unable to republish anything if you've only addressed some of the issues that have been raised since the last CR, so that you can keep an up to date document on TR. The intent is that you're expected to address all the issues over time, but may only address a few at a time.

However, I can see how this effectively removes the obligation to deal with issues at all. The transition to REC would force you to deal with all issues, but if a working group doesn't push a spec to REC, and stays in CR indefinitely, there's never a forcing function to address all issues.

I think this is a problem worth addressing, but we should stick to the intent of CR, and address it at that phase. Encouraging people to go to REC may also be a good thing, but not for that reason once we plug this hole. Note that the same problem exists about the lack of requirement to address all issues filed against a REC before the next publication of the same REC. Mostly, I agree with @michaelchampion that if a WG is not addressing issues raised by HR groups, we have more than a Process problem, but a simple tweak to the process could provide a bit of a nudge towards good behavior.

Here what I think we could do: for each issue opened against a CR (or REC) which has not yet been solved by the time the Working Group wants to publish the next CR Snapshot (or REC), the working group has to (a) make an explicit Group Decision to defer the issue the a subsequent publication, and (b) mark the issue explicitly in the spec.

(a) makes it so that a group cannot accidentally or negligently ignore issues. They need to make an explicit decision, which creates an opportunity to discuss the issue within the group, and needs consensus to move forward. Things can be deferred, but they cannot be forgotten. That decision also becomes an opportunity to file a Group Decision Appeal, should people find that the decision to defer is made in bad faith and is a "wontfix" is disguise.

(b) Gives visibility into the problem to readers of the spec who aren't following the day to day tribulations of the WG.

[nigelmegitt]

[Responding as a Chair, and with my own observations, not necessarily BBC's]

... what real problem is "stronger emphasis on reaching Recommendation" trying to fix?

@michaelchampion Where to start?! The comparison here is between shipping a Rec or remaining at CR.

• Recommendations are W3C's core product, and any Rec track work that doesn't get there is an overhead. My understanding is that members pay their fees with the expectation that W3C will publish and maintain Recs. (I'm not saying the proportion should be zero, because it's healthy to make some mistakes and recognise them)
• CRs don't demonstrate that anyone has implemented them; Recs do.
• CRs don't demonstrate that the specification can be implemented, because they do not need to be tested.
• CRs have no authoritativeness whatsoever, other than as a WG statement "here's a thing that we believe would be good and could be implemented"
• Building interdependent specifications on potentially poorly implemented (if test suites are inadequate), or not implemented foundations is an obvious recipe for future problems; any feature can be modified or removed by a CR iteration, so it is a requirement to have at least some stable reference points. Those are what we call Recommendations.
• W3C has relationships with other SDOs that wish to reference W3C standards: if W3C's specifications turn out to be faulty then W3C's reputation will be damaged and folk will go elsewhere. Note that other SDOs may not have any requirement that the specifications they reference are Recs, but they nevertheless do care about the quality: the W3C's checks and balances on quality are only demonstrated by progression to Rec.

... web interoperability is getting better at the same time "emphasis on reaching Recommendation" is declining, ...

I have not seen any evidence to support this statement and suspect that it is false. As more and more fine-grained feature-based specifications are incubated and implemented by potentially only one UA, if they do not advance through the Process's maturity levels, web interoperability declines.

In particular, web interoperability measurability declines for specs not at Rec because there is no stable comparison point and no requirement for the existence of tests.

... and it's generally accepted that the current WD or even latest GitHub commit is the version developers should reference.

This is an independent and orthogonal problem that deserves separate consideration: if we are concerned about which version developers should reference then we have the same problem whatever the maturity level is, when there may be multiple proposals for features or wording, either in pull requests, editor's drafts, working drafts or CRs. As demonstrated by P2020 it is possible in principle to iterate Recs, adding "non-Rec" features as notifications to developers.

In my view the version referencing issue is not an argument against emphasising reaching Recommendation. Rather, it is possibly an argument in favour, depending on the publishing WG's approach.

[nigelmegitt]

Here what I think we could do: for each issue opened against a CR (or REC) which has not yet been solved by the time the Working Group wants to publish the next CR Snapshot (or REC), the working group has to (a) make an explicit Group Decision to defer the issue the a subsequent publication, and (b) mark the issue explicitly in the spec.

@frivoal probably a minor nit, that could be covered by the word "solved" (!), but noting that this could be abused, there would also need to be the opportunity for the WG to reject the issue as "no change needed" having given it due consideration.

[michaelchampion]

@nigelmegitt wrote

As more and more fine-grained feature-based specifications are incubated and implemented by potentially only one UA, if they do not advance through the Process's maturity levels, web interoperability declines.

Hmm, that's an interesting hypothesis. It's not consistent with my experience when I worked on the Edge team; the folk wisdom in the browser implementer community was "CR is good enough to get real interoperability" I freely admit that my subjective experience and the implementer folklore could be wrong, what evidence could we use to shed light here?

https://caniuse.com/#home and https://caniuse.com/#index is the best-known resource to look at features by interoperability status and standards maturity. I don't see an easy way to break out the data across time to see if standards support / interoperability is increasing or decreasing, but it looks pretty good at the moment.

But to the question of whether "CR is good enough" or " we need stronger emphasis on reaching Recommendation", skimming the the CanIUse support tables doesn't show any obvious correlation between broad interoperability and Recommendation status. I don't get paid to argue about this stuff anymore :-) so I'm not inclined to do a more rigorous analysis of the data, but feel free to make a data-based case if I'm missing something.

Some of @nigelmegitt 's bullet points for why it's important to move from CR to Rec are formally true but empirically irrelevant if we assume that actual web developers look at CanIUse rather than w3.org/TR to determine whether a spec is mature enough to use.: The support tables tell the reader whether a spec is actually implemented and thus whether it can be implemented, and whether the implementations are good enough to get interoperability. The other points seem to focus on the value of Recommendations for W3C rather than the value to the community.

So, I continue to be unconvinced that "broad interoperability" is a compelling rationale for having the process document more strongly emphasize the need to reach Recommendation. There might be other reasons, especially if there are real examples of specs remaining in CR for a long time with unresolved A11Y, I18N, security, etc. issues that were raised by reviewers. I would welcome data with which to assess that hypothesis. In #443 (comment) @frivoal suggests some tweaks to the CR entrance and re-publication criteria that seem useful to prevent or mitigate such problems.

[frivoal]

@nigelmegitt fair point. Replace "solved" by "addressed"

[nigelmegitt]

formally true but empirically irrelevant if we assume that actual web developers look at CanIUse rather than w3.org/TR to determine whether a spec is mature enough to use.

@michaelchampion I'm wary of assumptions (my own included); do you know of any data about how developers make these determinations?

[michaelchampion]

Again, it was folklore in the browser developer community when I worked on Edge that website developers used tools such as caniuse.com and MDN web docs rather than the W3C. I'm not expert on website analytics, but the first source of data I'd check is the https://en.wikipedia.org/wiki/Alexa_Internet#Alexa_Traffic_Rank for the various sites. Installing a Chromium extension https://chrome.google.com/webstore/detail/webrank-seo/mkhilblbmkdnapffblmecglknalglfji into Edge and checking the sites I see (low rank means higher traffic):

Site	Traffic Rank	Trend
https://www.w3.org/TR/	5,514	generally falling over the past year
https://caniuse.com/	10,853	rising over the past year
https://developer.mozilla.org/en-US/	236	flat for the last year

So, at first glance, there's less traffic to caniuse.com than w3.org/TR (although this doesn't indicate what developers are doing). But there's MUCH more traffic to MDN Web Docs than W3.org/TR, and presumably it's mostly developers who look at MDN Web Docs. So I may have mis-spoken referring to CanIUse.com, but am probably right that developers use MDN Web Docs much more than w3.org/TR. And MDN uses caniuse.com data , see https://hacks.mozilla.org/2019/09/caniuse-and-mdn-compat-data-collaboration/, so maybe I was indirectly right :-)