Make autocomplete_metadata.py emit schema-compliant molecule metadata and improve CI tests

[Copilot]

autocomplete_metadata.py could emit invalid metadata.yaml entries (notably malformed strings/identifiers and invalid NMRlipids field types), causing schema validation failures for newly created molecule metadata. This update hardens output normalization so autogenerated metadata conforms to metadata_schema.json under partial/dirty upstream API payloads.

• Output sanitization for schema-facing fields

  • Added text normalization (clean_text) to strip HTML tags/entities from API-derived values before writing bioschema_properties and alternate names.
  • Added safe numeric parsing (safe_float) for molecularWeight to avoid invalid numeric output.

• sameAs identifier hardening

  • Added sanitize_sameas to keep only schema-valid identifier values/patterns (e.g., ChEBI, ChEMBL, cas, slm, pubchem.compound as int).
  • Prevents invalid external IDs from being persisted into generated YAML.

• NMRlipids schema compliance

  • Ensured NMRlipids.name is always populated via fallback chain (existing value → API names → molecule id).
  • Stopped emitting empty-string charge (schema expects integer when present).

• Focused regression coverage

  • Added tests/test_autocomplete_metadata.py to execute the autocomplete flow with mocked API responses and assert generated YAML validates against metadata_schema.json, including cleaned smiles/alternateName.

def clean_text(value):
    if not isinstance(value, str):
        return value
    return re.sub(r"<[^>]+>", "", unescape(value)).strip()

sameas = sanitize_sameas(extract_sameas(sources))
alternate_names = [clean_text(name) for name in alternate_names if clean_text(name)]

-Clena up test environments and CI
During fixing, it was discovered that the tox test environments should be restructured.
See #506 (review) below

[Copilot]

Pull request overview

This PR hardens developer/autocomplete_metadata.py so it generates metadata.yaml that conforms to src/fairmd/lipids/schema_validation/schema/metadata_schema.json, even when upstream API payloads contain dirty/HTML-ish strings or non-numeric weights. It also adds a regression test to validate the generated YAML against the JSON schema.

Changes:

• Added normalization helpers (clean_text, safe_float) and applied them to schema-facing fields (names, SMILES, InChI/InChIKey, molecular weight).
• Added sanitize_sameas to keep sameAs identifiers schema-pattern compliant.
• Added tests/test_autocomplete_metadata.py to mock API responses and assert schema validation passes and cleaned fields match expectations.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
developer/autocomplete_metadata.py	Adds sanitization/parsing helpers and applies them to emitted metadata fields to improve schema compliance.
tests/test_autocomplete_metadata.py	Adds a mocked end-to-end autocomplete test that validates generated YAML against metadata_schema.json.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mdondrup]

Have tested and debugged this. Now the output is formatted correctly. Also added a test.

[mdondrup]

Currently test test_calc_ff_quality_sin_exp_curve is failling (on MacOS 15). Nothing to do with my changes.
Edit: this must have been a temp glitch, seems to work now.

[korbinib]

looks good to me. Do we want to have the additional cas API call?

[comcon1]

It's good that you decided to add out-of-package tests, but then they must be clearly isolated from the package tests.

1. We need to make sure they are not running on release or other stories when we distribute the package. Because develop folder is not distributed with the package. It's conditioning stuff in the repo.
2. We need to name this kind of tests develop, and not network. There are other tests that use direct internet access and that are run currently inside tests-all tox label.
3. Tests develop tests should be accessible via tox. Currently, there is no tox label for them. And we should probably separate tests-all.

What is my suggestion.

• 1. Let's remove tests-all environment and replace it with tests-package environment.
• 2. Let's add tests-develop environment.
• 3. Let's call proper tox labels from proper events.

@MagnusSletten was working a lot with this part. I would like to also ask his opinion.

[mdondrup]

I added tests-package and replaced tests-all by it. Also added tests-develop. I am not sure if all events:call assignments are proper yet. Possibly need to discuss that.

[comcon1]

@mdondrup thanks. Now I cleaned CI a bit. I also would like to separate package and development tests and hang trigger to changes of different folders in PR. This is important I think..

[comcon1]

// I don't like that we had different coverage reporting from integration and unit-tests. I would like to have only coverage from unit tests because integration tests doesn't help do develop, so I'm not very interested in having this value despite it's large.

[MagnusSletten]

Nice pr!

Separating developer tests also means we can run them specifically when the developer/ folder is changed in CI, while keeping them isolated from the normal package tests.

Small nitpick: I slightly prefer keeping Codecov/coverage reporting in CI rather than baking it into tests-min, but I do not think this is a blocker. The generated files are gitignored, and this will not affect normal development at all.

[comcon1]

Now, we are ready! Thanks, all!

[mdondrup]

This PR has diverged from its original title and task. We should change the name and description.

[mdondrup]

I think this was merged prematurely. Now the tests are not working