Can we remember and encode the python-version flag

[peterbe]

Imagine...

Day 1
A developer creates the project and knows with confidence that this project is only going to use Python 3.6. So he installs the latest psycopg2 when creating the project:

touch requirements.txt
hashin --python-version cp36 psycopg2
cat requirements.txt

the output from that cat is as follows:

psycopg2==2.7.4 \
    --hash=sha256:0b9851e798bae024ed1a2a6377a8dab4b8a128a56ed406f572f9f06194e4b275 \
    --hash=sha256:733166464598c239323142c071fa4c9b91c14359176e5ae7e202db6bcc1d2eb5 \
    --hash=sha256:8966829cb0d21a08a3c5ac971a2eb67c3927ae27c247300a8476554cc0ce2ae8 \
    --hash=sha256:ad75fe10bea19ad2188c5cb5fc4cdf53ee808d9b44578c94a3cd1e9fc2beb656 \
    --hash=sha256:b68e89bb086a9476fa85298caab43f92d0a6af135a5f433d1f6b6d82cafa7b55

Cool. It only has binary wheels for CPython 3.6 and it includes Windows wheels if another developer wants to join and dev on her Windows computer.

Day 2
Apparently there's a new version out! Let's upgrade to that. This time, another entity attempts to upgrade the version and the requirements.txt. (I say "entity" because it could be a human or some sort of bot that uses hashin). They run:

hashin psycopg2
cat requirements.txt

The output of that cat is:

psycopg2==2.7.5 \
    --hash=sha256:0b9e48a1c1505699a64ac58815ca99104aacace8321e455072cee4f7fe7b2698 \
    --hash=sha256:0f4c784e1b5a320efb434c66a50b8dd7e30a7dc047e8f45c0a8d2694bfe72781 \
    --hash=sha256:0fdbaa32c9eb09ef09d425dc154628fca6fa69d2f7c1a33f889abb7e0efb3909 \
    --hash=sha256:11fbf688d5c953c0a5ba625cc42dea9aeb2321942c7c5ed9341a68f865dc8cb1 \
    --hash=sha256:19eaac4eb25ab078bd0f28304a0cb08702d120caadfe76bb1e6846ed1f68635e \
    --hash=sha256:3232ec1a3bf4dba97fbf9b03ce12e4b6c1d01ea3c85773903a67ced725728232 \
    --hash=sha256:36f8f9c216fcca048006f6dd60e4d3e6f406afde26cfb99e063f137070139eaf \
    --hash=sha256:59c1a0e4f9abe970062ed35d0720935197800a7ef7a62b3a9e3a70588d9ca40b \
    --hash=sha256:6506c5ff88750948c28d41852c09c5d2a49f51f28c6d90cbf1b6808e18c64e88 \
    --hash=sha256:6bc3e68ee16f571681b8c0b6d5c0a77bef3c589012352b3f0cf5520e674e9d01 \
    --hash=sha256:6dbbd7aabbc861eec6b910522534894d9dbb507d5819bc982032c3ea2e974f51 \
    --hash=sha256:6e737915de826650d1a5f7ff4ac6cf888a26f021a647390ca7bafdba0e85462b \
    --hash=sha256:6ed9b2cfe85abc720e8943c1808eeffd41daa73e18b7c1e1a228b0b91f768ccc \
    --hash=sha256:711ec617ba453fdfc66616db2520db3a6d9a891e3bf62ef9aba4c95bb4e61230 \
    --hash=sha256:844dacdf7530c5c612718cf12bc001f59b2d9329d35b495f1ff25045161aa6af \
    --hash=sha256:86b52e146da13c896e50c5a3341a9448151f1092b1a4153e425d1e8b62fec508 \
    --hash=sha256:985c06c2a0f227131733ae58d6a541a5bc8b665e7305494782bebdb74202b793 \
    --hash=sha256:a86dfe45f4f9c55b1a2312ff20a59b30da8d39c0e8821d00018372a2a177098f \
    --hash=sha256:aa3cd07f7f7e3183b63d48300666f920828a9dbd7d7ec53d450df2c4953687a9 \
    --hash=sha256:b1964ed645ef8317806d615d9ff006c0dadc09dfc54b99ae67f9ba7a1ec9d5d2 \
    --hash=sha256:b2abbff9e4141484bb89b96eb8eae186d77bc6d5ffbec6b01783ee5c3c467351 \
    --hash=sha256:cc33c3a90492e21713260095f02b12bee02b8d1f2c03a221d763ce04fa90e2e9 \
    --hash=sha256:d7de3bf0986d777807611c36e809b77a13bf1888f5c8db0ebf24b47a52d10726 \
    --hash=sha256:db5e3c52576cc5b93a959a03ccc3b02cb8f0af1fbbdc80645f7a215f0b864f3a \
    --hash=sha256:e168aa795ffbb11379c942cf95bf813c7db9aa55538eb61de8c6815e092416f5 \
    --hash=sha256:e9ca911f8e2d3117e5241d5fa9aaa991cb22fb0792627eeada47425d706b5ec8 \
    --hash=sha256:eccf962d41ca46e6326b97c8fe0a6687b58dfc1a5f6540ed071ff1474cea749e \
    --hash=sha256:efa19deae6b9e504a74347fe5e25c2cb9343766c489c2ae921b05f37338b18d1 \
    --hash=sha256:f4b0460a21f784abe17b496f66e74157a6c36116fa86da8bf6aa028b9e8ad5fe \
    --hash=sha256:f93d508ca64d924d478fb11e272e09524698f0c581d9032e68958cfbdd41faef

Now we get all those wheels for all other versions of Python. Gross!

Can we avoid that and "continue" to stick to CPython 3.6 like the original developer did?

[peterbe]

PEP 496 -- Environment Markers definitely has an example for this:

unittest2 >=2.0,<3.0 ; python_version == '2.4' or python_version == '2.5'

(PEP 496 is not all about python versions but also about other stuff such as sys.platform)

So if the hashin user used hashin Django --python-version cp35 --python-version cp36 we just convert that to Django==2.1; python_version == '3.5' or python_version == '3.6' Right? Probably a bad idea.

The PEP 496 lingo is that of setup.py and setuptools. It's not an obvious serialization and deserialization format. Lots of parsing hacking.

Another option that @mythmon mentioned is to use comments. E.g. comments specifically from hashin. If we "own" the serialization of comments we can more easily parse it and rewrite it. For example, the output could become something like:

# hashin python-versions: cp35, cp36
# hashin algorithm: sha256
psycopg2==2.7.4 \
    --hash=sha256:0b9851e798bae024ed1a2a6377a8dab4b8a128a56ed406f572f9f06194e4b275 \
    --hash=sha256:733166464598c239323142c071fa4c9b91c14359176e5ae7e202db6bcc1d2eb5 \
    --hash=sha256:8966829cb0d21a08a3c5ac971a2eb67c3927ae27c247300a8476554cc0ce2ae8 \
    --hash=sha256:ad75fe10bea19ad2188c5cb5fc4cdf53ee808d9b44578c94a3cd1e9fc2beb656 \
    --hash=sha256:b68e89bb086a9476fa85298caab43f92d0a6af135a5f433d1f6b6d82cafa7b55

So if the user used --algorithm (or -a) or --python-version we just cram it into the file above the package name and when someone runs hashin again in the future we just look for those things and basically make it as if this next user had also used those cli flags.

(Note, the default algorithm is sha256 so the example above might not be applicable or useful)

[peterbe]

Here's an example that Pyup uses

Note that that example has a comment inside the special Pyup comment. Smart feature to have.