Mark experimental/provisional APIs

[kohr-h]

As a consequence of #1312 and #1267, it seems to me that we should have a way to mark certain new APIs as experimental and subject to change. It just happens sometimes that the first implementation is not optimal and there are good reasons to revise it. I guess this will also become more important in the future after we actually cross the 1.0 line.

I suggest that we establish a standard way for marking brand new features and APIs as experimental for some period of time (measured in releases I guess) and explicitly warn about the possibility of breaking change. It kind of goes hand in hand with a deprecation process that we will also need to adopt at some point.

The simplest way I can think of is something like

.. warning::
    This API is experimental and may change in the future.

[adler-j]

I disagree with this. Anything that goes into ODL core should NOT be experimental.

The proper way to solve this is to consider all code on GitHub as experimental, and only the minor numbered releases (0.6, 1.0, 1.1) as fully supported. Users should then, in theory, use the latest stable release if they want non-experimental code, and anyone working from the github release should be aware that things might change.

However, to do this, we need to actually start making releases more often.

[kohr-h]

I disagree with this. Anything that goes into ODL core should NOT be experimental.

The proper way to solve this is to consider all code on GitHub as experimental, and only the minor numbered releases (0.6, 1.0, 1.1) as fully supported. Users should then, in theory, use the latest stable release if they want non-experimental code, and anyone working from the github release should be aware that things might change.

Well, that's good as an ideal, but it's not always obvious upfront how a good API should look like. New features always go through review, and that's good for catching the more obvious flaws, but sometimes things just slip through.

Some of this boils down to the question of how much of an API guarantee we are willing to give. Currently there's nothing like that explicitly written down anywhere, and maybe we get away with breaking changes more easily before a 1.0 release. But at some point user annoyance becomes an actual issue, and I think it's important to make things explicit at some point.

An alternative we should also consider is to push big changes through contrib first, although that will in fact mean 2 API changes instead of 1.

So however way we go, the very least we need IMO is a proper deprecation process.

However, to do this, we need to actually start making releases more often.

Actually your approach speaks against releasing more often, because the more often we release, the less time we have to fix API-related issues.

[adler-j]

Perhaps going the numpy way with "version added" would help users see e.g. how mature some feature is? Overall I feel that anything in a numbered release should be mature enough that users shouldn't feel worried about using it (especially given that noone reads the doc anyway and they would just be negatively surprised).

The idea of first going for contrib sounds awfully complicated in my opinion. Tensorflow does this and it is a mess.

So, my suggestion would be (starting from 1.0):

• Start using "version added" (e.g. add .. versionadded:: 1.10.0 to Notes).
• Add a proper deprecation process, e.g. .. deprecated:: 0.98 along with a PendingDeprecationWarning.

[kohr-h]

Largely agree with this suggestion. I had a look at the .. versionadded thing before and thought it was nice, but for that you need versions 😉. See also #1335.

So how about this:

• From 1.0 on APIs are stable and only change for a good reason.
• Current master is experimental and subject to API change without notice. However we should really ensure that people can use master without worries, which means it's never broken in obvious ways, and we don't make breaking changes light-handedly.
• New features (and signature parameters) get a version tag as you suggest.
• Deprecated features and APIs are warned about when used, with a visible warning.
• Deprecation happens over N minor releases, with warnings issued during N releases.

However, in particular the deprecation process, and a bit also the new feature tagging, assumes that releases happen in more or less intervals. So the question is what such an interval could be.

[adler-j]

My overall feeling is that it's more important that something happens than that it happens perfectly. E.g. overall we want to:

• Keep master in a usable state (no crashes or other bugs etc) but perhaps with API changes.
• Release as often as is reasonable. With a somewhat steamlined release process we should be able to do it every N months (for N ~1-4). But we shouldn't feel any pressure to do it according to some exact schedule (at least while the user base is relatively small).

[kohr-h]

Release as often as is reasonable. With a somewhat steamlined release process we should be able to do it every N months (for N ~1-4). But we shouldn't feel any pressure to do it according to some exact schedule (at least while the user base is relatively small).

I think we need to have a target interval. If we just say "maybe it's time to release" whenever we feel it's adequate, we kind of end up in the current situation automatically 🤷‍♂️ . Everybody has their obligations and no time, basically. But if there's a pop up in the calendar every N, say 3, months, it's at least more likely that we start to discuss. Of course, nobody wants a rigid release schedule.

[adler-j]

Imo, 3 months is a good target for releases. We could even try scheduling them

[kohr-h]

Okay, how about issues of the form Release 0.7 | Target date 2018-05-31 that tell you in the title when it's due? We can use them to track stuff to include as we are already doing.

The alternative would be milestones, but I'm a bit shying away from them since they require more administration and tend to be overlooked more. Also, for a new issue/PR it's sometimes hard to foresee when it's gonna be done, so assigning to a milestone tends to be largely random for bigger issues or PRs.

Finally, we could use the Projects pane to organize stuff. It would be a nice alternative to making lists by hand in the release issue.

Here's a suggestion:

• We create a project "Release X.Y" and drag PRs and issues to it.
• We make an issue of the form Release X.Y | Target date <3 months ahead> and link to the project page.