Validating recipes

[nirs]

Problem

What problem are you trying to solve? Why do we need this feature? What is the
current behavior and why is it not sufficient?

Invalid recipes resources

New recipe with errors can be applied, and fail backup/restore time. We want an easy way to validate new recipes before we apply them.

Recipes created by:

• User created recipes manually based on the docs
• AI generated recipes

Possible issues:

• syntax error - invalid yaml
• semantics errors:
  • out of range numeric values (minimum, maximum)
    • maximum value not define in the spec
    • not validated in ramen
  • wrong enums "onError: continue|fail"
    • defined in CRD
  • referring to non exisitng group
  • referring to exiting hooks
• Values that need replacement using DRPC.spec.recipeParameters (e.g. $timeout")
  • We can add warning or note about such values

Validating protected application recipes

Installed recipes can have the same issues and are not displayed in validate application reports.

• DRPCSummary does not show or validate recipeRef

  • show value, validate that recipe exists

• Recipes are not displayed

  • Show recipe name, namespace
  • Show Valid: true/false

Issues in installed recipes:

• invalid yaml (impossible)
• semantics errors
  • how do show the errors?
• pvc selector can match no pvc
  • can be a warning - this is probably an error
• kube object selector can match no object
  • can be a warning - this is probably an error
• DRPP.spec.receipeParameters (map)
  • In the recipe we can have value like "$name"
  • ramen replaces with values from DRPP.spec.receipeParameters.
  • ramen use zero value when parameter is missing and log a warning?
  • We can have values without a value in the map

Proposed solution

How do you want to solve the problem? Describe the approach and why this is the
right solution.

• Support offline validation of new recipes
• Validate installed recipes in validate application command

Future work:

• Dry-run/test a recipe: backup an application using a recipe and restore it
  • Probably something that ramen can provide
  • But can be very useful if we can test a recipe on an application without ramen

Alternatives considered

What other approaches did you consider? Why is the proposed solution better?

• Manual validation - too hard to catch issues
• Standalone command - how to deliver the tool? we want to deliver using odf cli, it will have to duplicate the same code used for online validation

User experience

How will the user experience change? If we add new commands or flags, describe
how they will be used in different scenarios. How do the new commands and flags
work with existing commands and flags? Show example usage and expected output.

validate application

• Show and validate DRPC.spec.RceipeRf
• Show and validate the recipes in the managed clusters

Open issue:

• What fields we show in the ValidatedRecipe

validate recipe (offline)

$ ramenctl validate recipe my-recipe.yaml [--verbose]
error: workflow "restore" is ... (stderr)
name: my-recipe
namespace: my-namespace
workflows:
- name: backup
  state: ok ✅ 
  sequence:
    - group1
    - hook1  
- name: restore
  state: problem ❌
  description: why this is a problem

Open questions:

• What do we want to show during the operation? (it should be very quick)
• What it output? (yaml, text)
  • same as yaml report for validation application ValidatedRceipe
  • or maybe list of errors and warnings
• Do we need a log file? probably not needed
• Debug mode can help to log stderr during valdation
• No html output

Configuration

Do we need new configuration options? Specify the new configuration and how it
affects existing options.

No change needed

Documentation

What documentation changes are needed? Describe the content of new or updated
documents. We want to start with documentation, not code.

• Update docs/validate.md with example report for validate application
• Add validate recipe section

Security and privacy

Does this feature handle secrets or sensitive data? Do we need to collect,
store, or display data that may contain sensitive information? How will the data
be sanitized?

No effect.

Backward compatibility

Does this feature change existing commands, flags, configuration, or output
formats? Will it break existing users or projects that depend on ramenctl as a
module?

No effect.

Performance

How does this feature affect performance? Does it add slow operations or
operations that require a lot of memory?

No effect.

Testing

How will the new feature be tested? Describe the new tests to add and changes to
existing tests.

validate application

report:

• ValidatedRecipe (Equal)
• ValidatedWorkflow (Equal)
• ValidatedGroup (Equal)
• ValidatedHook (Equal)

application:

• application/command_validated_test.go - units tests for single Validated values
  • DRPCSummmary.Recipe (ValidatedReceipe)
    • Feed recipe and check the DRPCSummary.Recipe
• application/command_test.go
  • add testdata/disapp-receipe-check-exec-deploy-rbd (including the recipes)
  • change TestValidateApplicationPassed to read expected status from testdata/appset-deploy-rbd-status.yaml
  • raname TestValidateApplicationPassed to TestValidateApplicationAppsetPassed
  • add new test TestValidateApplicationDisappPassed
  • add testdata/disapp-deploy-rbd-status.yaml

validate recipe

• Similar test to validate application - test the entire command
• Add testdata/ with recipe for each test