profile-generate does not populate profile-param-value-origin in markdown using value from profile.json

[ericslandry]

Describe the bug

When I run trestle author profile-generate... I expect the value for "profile-param-value-origin" to be copied from my profile.json to the markdown file. It is being copied from the markdown to my profile.json when I run trestle author profile-assemble...

To Reproduce

Steps to reproduce the behavior:

1. Update the value for param-value-origin in my profile.json file

    "modify": {
      "set-parameters": [
        {
          "param-id": "ac-01_odp.01",
          "props": [
            {
              "name": "param-value-origin",
              "value": "This is the new parameter origin value"
            }
          ],
          "values": [
            "This is the parameter value"
          ]
        },

2. Generate the markdown with:

trestle author profile-generate --output markdown/profile/foo --name foo --overwrite-header-values

3. See that the ac-1.md still has the old value

  ac-01_odp.01:
    guidelines:
      - prose: This is the markdown prose
    profile-values:
      - 'This is the parameter value'
    profile-param-value-origin: This is the OLD parameter origin value

The other value get updated correctly during profile-generate, but not this origin field. The origin fields gets correctly copied from markdown to the profile.json when using "trestle author profile-assemble..."

Expected behavior

I expect ac-1.md to have the new value

  ac-01_odp.01:
    guidelines:
      - prose: This is the markdown prose
    profile-values:
      - 'This is the parameter value'
    profile-param-value-origin: This is the new parameter origin value

Environment

• OS: Ubuntu lastest
• Python version: 3.13.6
• Installed packages:

[degenaro]

@AleJo2995 Could you please have a look at this for an assessment of working as designed or broken, soon-ish? If you cannot, please let me know and I'll take a look. Thanks!

[degenaro]

@ericslandry Some questions:

1. If you erase the markdown and then do the profile-generate with the changed json, do the expected changes appear in markdown?
2. If you then make another change to the json and do profile-generate, the expected change does not appear in the markdown, correct?

The markdown is the "source of truth" if it exists, so by design the change to the json has no effect.

To live within the current design, simply erase the markdown or...please refer to the profile-generate portion of this document: https://oscal-compass.dev/compliance-trestle/latest/tutorials/Trestle_authoring/ssp_profile_catalog_authoring/. In particular, see --force-overwrite and see if that works for you.

[ericslandry]

@degenaro

1. Yes
2. Yes, but I thought the --overwrite-header-values flag would overwrite all the front-matter fields.

[degenaro]

Hi @ericslandry, thanks for your reply. In your original remarks, you changed the json but that did not cause the markdown to be updated. The "fix" is to erase the markdown or use --force-overwrite, which seemed to work for you if I'm understanding correctly.

With respect to --overwrite-header-values, it seems (according to my reading of the documentation) that only the optional YAML header will be used to overwrite.

When generating markdown a YAML header may be optionally provided, and if so, the option --overwrite-header-values will cause the values in the provided YAML header to overwrite the value in the markdown file's header for any items that are common. Otherwise the provided YAML header will simply insert any values not already in the markdown header.

What additional feature/fix is desired?

[ericslandry]

Hey @degenaro ! Thanks for the feedback!

I'm trying to create a system profile that combines multiple catalogs/baseline profiles. Due to various factors, I expect my system profile to change over time. The markdown is the source of truth for certain things, but if for example I want to add an import of a new profile, then that can't happen in the markdown, so I'm forced to overwrite and copy prose from previous.

Ideally, I would like to be able to evolve my OSCAL artifacts without any overwrites and re-copying.

(I hope my explanation makes sense)

[degenaro]

@vikas-agarwal76 Any thoughts here?

[vikas-agarwal76]

@ericslandry I believe the --force-overwrite option will handle your situation. As per documentation -
"--force-overwrite option can be used to overwrite markdowns with content from JSON. Note that this option will completely delete all existing markdowns (in the given output folder) without saving any changes. To save the changes, run assemble first."

So, before modifying the JSON file manually just run assemble to have all markdown changes into JSON. Then you can modify JSON as you like and run generate again with --force-overwrite option. This will generate the markdowns fresh, but since previous markdown content was already in JSON, the new Markdowns will have all the previous content.

Also remember -
"Similarly, when assembling to JSON, the --set-parameters option will cause any changes in the header to take effect and change values in the assembled JSON for the control. The changes can including setting parameter and property values. The --set-parameters option is available only for catalog-assemble and profile-assemble."

So while running profile-assemble, use the --set-parameters option also to copy the changes to parameter values in Markdown (profile-values field) into JSON.

[degenaro]

@ericslandry Anything further on this issue? We hope to make a new release soon...

[ericslandry]

@vikas-agarwal76 I understand that --force-overwrite will update my markdown's profile-param-value-origin field in the front matter. What I'm arguing is that all fields in the front matter should be updated when using --overwrite-header-values, including profile-param-value-origin.

[vikas-agarwal76]

@ericslandry --force-overwrite option should copy all fields from "SetParameters" in profile to the Markdowns. Is this not happening? Please tell which field is not being copied.

--overwrite-header-values option is to be used with the --yaml-header option when you are providing a separate yaml header file. --yaml-header will copy the header values from the separate yaml file provided into the Markdowns during profile-generate, but will not overwrite existing header values (if that property already exists in Markdowns) unless --overwrite-header-values option is also provided.

[degenaro]

@ericslandry do you concur with @vikas-agarwal76 that trestle is working as designed? Or do you still think we have a bug here?