Skip to content

io_properties: fix --update-aws-params and add RunByUser tagging#973

Open
fruch wants to merge 3 commits into
scylladb:masterfrom
fruch:fix_SMI-297
Open

io_properties: fix --update-aws-params and add RunByUser tagging#973
fruch wants to merge 3 commits into
scylladb:masterfrom
fruch:fix_SMI-297

Conversation

@fruch

@fruch fruch commented Jun 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fix the --update-aws-params flag in get_scylla_io_properties.py to produce minimal, correct diffs when updating instance IO parameters.

Changes

Sort aws_io_params.yaml for stable diffs

  • Sort instance types by family (letter prefix, generation number, suffix) then by size within each family
  • Standardize sub-key ordering to alphabetical (read_bandwidth, read_iops, write_bandwidth, write_iops)
  • Ensures updating a single instance type won't cause unrelated diff noise

Add RunByUser tag to EC2 instances

  • Add get_username() to identify who launched the instance (matches scylla-cluster-tests pattern)
  • Tag all launched instances with RunByUser for cost tracking and ownership
  • Add missing import getpass and import subprocess

Fix --update-aws-params to write correct format

  • update_single_instance_params was storing the raw io_properties.yaml structure (with disks list wrapper and mountpoint) instead of the flat 4-value format
  • update_aws_params_yaml was re-reading/re-writing the entire file per instance in a loop with sort_keys=False
  • Both functions produced diffs on unrelated entries when updating a single instance type
  • Add extract_io_params() to consistently extract just the 4 IO values
  • Add write_aws_params_yaml() using sort_aws_instance_types for consistent ordering
  • Remove unused sort_instances_by_size function

fruch added 3 commits June 21, 2026 20:32
Sort instance types by family (letter prefix, generation number, suffix)
then by size within each family. Standardize sub-key ordering to
alphabetical (read_bandwidth, read_iops, write_bandwidth, write_iops).

This ensures that updating a single instance type won't cause unrelated
diff noise from re-ordering.
Add get_username() to identify who launched the instance, matching
the pattern used in scylla-cluster-tests. Tags all launched instances
with RunByUser for cost tracking and ownership identification.

Also adds the missing 'import getpass' and 'import subprocess' that
the function requires.
Problems fixed:
- update_single_instance_params stored the raw io_properties.yaml
  structure (with 'disks' list wrapper and 'mountpoint') instead of
  the flat 4-value format used in aws_io_params.yaml
- update_aws_params_yaml re-read and re-wrote the entire file on each
  instance in a loop, and used sort_keys=False causing unstable output
- Both functions produced diffs on unrelated entries when updating
  a single instance type

Changes:
- Add extract_io_params() to consistently extract the 4 IO values
  from raw io_properties structure
- Add write_aws_params_yaml() that uses sort_aws_instance_types for
  consistent top-level ordering and sorted sub-keys
- update_aws_params_yaml now reads once, updates all instances, writes
  once
- update_single_instance_params now extracts flat values and uses the
  same write function
- Remove unused sort_instances_by_size function
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant