Sort array of dependencies in environment.yml

[rcasero]

I have an environment.yml file for a conda environment:

name: foo
channels:
- pytorch
- conda-forge
- bioconda
dependencies:
- pybigwig=0.3.18
- cpuonly=2.0
- mkl_random=1.2.0
- mkl_fft=1.2.0
- ucsc-bigbedtobed=377
- certifi=2021.5.30
- pip=21.2.4
- torchvision=0.11.3
- cudnn=7.6.5.32
- pytorch-cpu=1.1.0
- python=3.6.15
- pip:
  - -r requirements.txt

I'd like to sort the dependencies: section alphabetically.

I'm running yq version 4.26.1 installed with snap on linux.

The best I've managed is

cat environment.yml | yq '.dependencies | sort_keys(.)'

but this doesn't seem to sort the array

- pybigwig=0.3.18
- cpuonly=2.0
- mkl_random=1.2.0
- mkl_fft=1.2.0
- ucsc-bigbedtobed=377
- certifi=2021.5.30
- pip=21.2.4
- torchvision=0.11.3
- cudnn=7.6.5.32
- pytorch-cpu=1.1.0
- python=3.6.15
- pip:
    - -r requirements.txt

(In addition, not sure how I'd put the sorted array back into the file).

Any help would be appreciated, thanks!

Kind regards,

Ramon.

[rcasero]

Other things I've tried, but give syntax errors:

$ cat environment.yml | yq '.dependencies | sort(.)'
Error: Bad expression, please check expression syntax
$ cat environment.yml | yq '.dependencies | sort'
Error: sort only works for scalars, got !!map
$ cat environment.yml | yq '.dependencies | sort(..)'
Error: Bad expression, please check expression syntax

[mikefarah]

This is a little trickier than examples I have in the docs, because in that array of strings there is a single map entry pip: ... - so the vanilla sort will complain about not knowing how to sort a map ( sort only works for scalars, got !!map)

What needs to be done is to use the sort_by operator to specify how to sort this array - particularly when it encounters a map.

Solving this actually revealed a bug in the alternative (//) operator in yq! I've made a new release v4.27.4 - so you will need that version for this:

yq '.dependencies |= sort_by( select(tag == "!!str") //  (keys | .[0]) )' file.yaml

Explanation:

• we want to update .dependencies based on its current value |=
• we're going to use a custom sort: sort_by
• sort_by works by giving it an expression that will return a value for each item in the array, the array entries will be sorted by those calculated values.
• if it's a string, we can return it as is select(tag == "!!str")
• alternatively // it must be a map, so lets get the first key keys | .[0]
  Note that the alternative opwill return the RHS if the LHS is falsy (undefined, empty, false)

[rcasero]

Thanks, @mikefarah .

I have upgraded yq with

sudo snap switch yq --channel=latest/edge
sudo snap refresh yq 

I get a weird error when I try to apply yq to the file in place from a conda environment in CentOS Stream 8

$ cat environment.yml 
name: 20220901_chopbert_data_wrangling
channels:
- pytorch
- conda-forge
- bioconda
dependencies:
- pybigwig=0.3.18
- cpuonly=2.0
- mkl_random=1.2.0
- mkl_fft=1.2.0
- ucsc-bigbedtobed=377
- certifi=2021.5.30
- pip=21.2.4
- torchvision=0.11.3
- cudnn=7.6.5.32
- pytorch-cpu=1.1.0
- python=3.6.15
- pip:
  - -r requirements.txt

$ yq -i '.dependencies |= sort_by( select(tag == "!!str") //  (keys | .[0]) )' environment.yml 
Error: stat environment.yml: no such file or directory

The sorting works with the following, that just sends the output to stdout

$ yq '.dependencies |= sort_by( select(tag == "!!str") //  (keys | .[0]) )' < environment.yml 
name: 20220901_chopbert_data_wrangling
channels:
  - pytorch
  - bioconda
  - conda-forge
dependencies:
  - certifi=2021.5.30
  - cpuonly=2.0
  - cudnn=7.6.5.32
  - mkl_fft=1.2.0
  - mkl_random=1.2.0
  - pip:
      - -r requirements.txt
  - pip=21.2.4
  - pybigwig=0.3.18
  - python=3.6.15
  - pytorch-cpu=1.1.0
  - torchvision=0.11.3
  - ucsc-bigbedtobed=377

As a workaround to rewrite the file, this works

yq '.dependencies |= sort_by( select(tag == "!!str") //  (keys | .[0]) )' < environment.yml | sponge environment.yml

[mikefarah]

yeah that'll be snap and its strict confinement

[rcasero]

Got it, thanks!