Validate "or later" licenses

[pombredanne]

The master branch implementation treats "or later" licenses as separate keys with an eventual aliases.

The alternate-or-later-handling branch implementation treats "or later" as keywords and not as separate license keys.
If this later implementation ends up a winner, if would make sense to add validation to license symbols to check if a license supports an "or later" version or not to avoid stupid things like "MIT or later"

[steven-esser]

#11 has been merged. Is there anything left to do here?

[RazerM]

Based on Annex D: SPDX license expressions, shouldn't I be able to parse Apache-2.0+, for example?

>>> from license_expression import get_spdx_licensing
>>> licensing = get_spdx_licensing()
>>> licensing.parse("Apache-2.0+", validate=True, strict=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<redacted>/python3.9/site-packages/license_expression/__init__.py", line 559, in parse
    self.validate_license_keys(expression)
  File "<redacted>/python3.9/site-packages/license_expression/__init__.py", line 466, in validate_license_keys
    raise ExpressionError(msg)
license_expression.ExpressionError: Unknown license key(s): Apache-2.0+

[pombredanne]

@RazerM Thank you for chiming in!

We use the bundled SPDX license list (derived from ScanCode) to validate strictly and we do not consider non-catalogued usages of the + trailing sign as a special construct: if you (really) want to support some Apache-2.0+ license you would need to add this explicitly to a symbol list for this to validate.

You will note that even at SPDX, the use of trailing + is generally out of favor when not flat-out deprecated as it is for GPL/LGPL in the https://spdx.org/licenses/ "Deprecated License Identifiers" section in favor of using proper dedicated licenses keys such as GPL-2.0-or-later. So technically the spec would allow GPL-2.0-or-later+ and GPL-2.0-only+ which is at best weird if not fun and byzantine.

We do support all common usage of a + on A/L/GPL and GFDL licenses with aliases and this covers 99.99% of the case seen in the wild as provided in the list such as here: https://github.com/nexB/license-expression/blob/da6f49db9ef1b44a0d5d52851e60ed14dff82c78/src/license_expression/data/scancode-licensedb-index.json#L7188

In earnest I only ever saw an explicit usage of an "Apache-2.0 or later" license notice in the wild a handful of times; and it is a minor point in the grand scheme of things until there will be such thing as another version of the Apache license which is a fairly unlikely event to happen in the next few years.

This said, we still have a handful of rules in scancode and we report an Apache 2.0 in these cases, so this is not an entirely unknown quantity albeit odd and rare in the wild:

• https://github.com/nexB/scancode-toolkit/blob/develop/src/licensedcode/data/rules/debian_common_apache-2.0-plus.RULE
• https://github.com/nexB/scancode-toolkit/blob/develop/src/licensedcode/data/rules/apache-2.0_1098.RULE
• https://github.com/nexB/scancode-toolkit/blob/develop/src/licensedcode/data/rules/apache-2.0_1099.RULE

So to recap:

1. we do not support generic trailing + suffix but instead only use concrete license identifiers. So we no not exactly support everything in the SPDX grammar. This is a design choice and this avoids weird shots.
2. if you need some Apache-2.0+ now you can add it locally to your symbol list either as a new key or an alias for the Apache-2.0
3. Apache-2.0+ is at best a harmless oddity IMHO but we detect it in ScanCode as plain Apache-2.0
4. I am OK to add Apache-2.0+ as a SPDX id alternative to the main Apache-2.0 in ScanCode and thhis will then trickle down here too
5. I am not against adding an apache-2.0-plus to the licensedb if there is some minimally visible usage in the wild and we would give it an Apache-2.0+ SPDX id.

[pombredanne]

Just to explain further, here I am adding Apache-2.0+ as an alternative SPDX id for apache-2.0

>>> from license_expression import *
>>> index = get_license_index()
>>> apache = [l for l in index if l["license_key"] == "apache-2.0"][0]
>>> apache["other_spdx_license_keys"].append("Apache-2.0+")
>>> spdx_licensing = build_spdx_licensing(index) 
>>> spdx_licensing.parse("Apache-2.0+", validate=True, strict=True)
LicenseSymbol('Apache-2.0', aliases=('Apache-2.0+',), is_exception=False)

or with a new symbol:

>>> index = get_license_index()
>>> a2plus = {'license_key': 'apache-2.0-plus', 'spdx_license_key': 'Apache-2.0+', 'other_spdx_license_keys': [], 'is_exception': False, 'is_deprecated': False, 'json': 'TBD', 'yml': 'TBD', 'html': 'TBD', 'text': 'TBD'}
>>> index.append(a2plus)
>>> spdx_licensing = build_spdx_licensing(index) 
>>> spdx_licensing.parse("Apache-2.0+", validate=True, strict=True)
LicenseSymbol('Apache-2.0+', is_exception=False)

[RazerM]

@pombredanne thank you for the thorough response!

I didn't see Apache-2.0+ in the wild with my dependencies. I am evaluating the use of this library to parse licenses and I had noticed in the SPDX spec that this syntax exists.

Given the workarounds that are available if this does come up, it should be fine.

[pombredanne]

@RazerM you wrote:

I am evaluating the use of this library to parse licenses

I can state --without a hint of bias 👼 -- that this is likely the finest license-expression parsing library available in Python (and I think it is also the only one :] ) and likely one of the nicest anywhere! ... more seriously keep us posted with your eval. This is used in a few other projects such as AboutCode toolkit, ScanCode toolkit and ScanCode.io, FSFE REUSE and other places. It is supposed to be decently good at the small things it knows to do. And it is backed by a not too shabby boolean engine by @bastikr that I co-maintain.

If there is something that does not work as it should, I would like it fixed, so your feedback is mucho welcomed!

[mgorny]

I've used license-expression as part of pycargoebuild to parse license strings from Rust packages. Unfortunately, there are quite some people who unpredictably use + with licenses such as MPL-2.0 and I'd really prefer if I hadn't to work around this, at least long term.

You will note that even at SPDX, the use of trailing + is generally out of favor

Could you explain? I find that specification very hard to read but I couldn't find anything saying that.

[pombredanne]

@mgorny do you have a list of all these cases?
For MPL-2.0 in particular MPL-2.0+ means nothing special, the MPL-2.0 already has baked in terms that allow to use any other versions. See section 10.2:

10.2. Effect of New Versions

You may distribute the Covered Software under the terms of the version
of the License under which You originally received the Covered Software,
or under the terms of any subsequent version published by the license
steward.

So here, we could effectively parse MPL-2.0+ to MPL-2.0 with a proper alias in the licensing symbols IMHO.

[mjherzog]

@mgorny The deprecation of "+" in favor "or-later" may not be in the spec, but it is in the (long) history of SPDX discussions. The "+" symbol was kept for backward compatibility.

[mgorny]

@mgorny do you have a list of all these cases? For MPL-2.0 in particular MPL-2.0+ means nothing special, the MPL-2.0 already has baked in terms that allow to use any other versions.

I don't have a list (this is the first time it was reported to me, with MPL-2.0+), and I know. The problem roughly is, it's valid and Rust accepts it, so people can use it, and so we have to handle it somehow. In my code I already include the automatic fallback to plain license if we don't have an explicit expansion, it's verification that's gating it. That said, I removed verify=True for the time being, I suppose that's good enough for us.

[mgorny]

@mgorny The deprecation of "+" in favor "or-later" may not be in the spec, but it is in the (long) history of SPDX discussions. The "+" symbol was kept for backward compatibility.

Well, then, perhaps it should be in the spec and propagated more widely because right now new projects are using it (whether it's meaningless or not).