Skip to content

# Feat: Add get_metadata_schema Helper for Sample Metadata Introspection - #1124

Open
rajat552 wants to merge 4 commits into
malariagen:masterfrom
rajat552:feature/schema-introspection
Open

# Feat: Add get_metadata_schema Helper for Sample Metadata Introspection#1124
rajat552 wants to merge 4 commits into
malariagen:masterfrom
rajat552:feature/schema-introspection

Conversation

@rajat552

Copy link
Copy Markdown

Description

The Problem

Currently, when constructing a sample_query to filter dataset metadata or analysis methods (like PCA or Diversity calculations), users (or programmatic clients like an LLM) have no straightforward programmatic way to know what columns are available or what the possible valid values are (e.g., discovering all unique country or taxon values).

The Solution

This PR introduces a small, non-breaking helper method get_metadata_schema() to the AnophelesSampleMetadata mixin class. It programmatically introspects the sample metadata dataframe and exposes a dictionary of the schema.

It automatically categorizes columns by parsing df.columns and df.dtypes to build an intelligent dictionary mapping:

  • Low-cardinality Categoricals (Strings/Objects with ≤ 100 unique values): Returns the type as "categorical" and enumerates the possible "values" (e.g., for taxon, country).
  • High-cardinality Strings: Returns "type": "string".
  • Numeric columns (Int/Float): Returns "type": "numeric" and calculates the "min" and "max" bounds (e.g., for year).
  • Booleans: Returns "type": "boolean" and "values": [True, False].

Closes #1123

Example Usage

import malariagen_data

ag3 = malariagen_data.Ag3()
schema = ag3.get_metadata_schema(sample_sets="AG1000G-UG")

print(schema["country"])
# Output: {'type': 'categorical', 'values': ['Uganda']}

print(schema["year"])
# Output: {'type': 'numeric', 'min': 2012, 'max': 2012}

@rajat552

Copy link
Copy Markdown
Author

@jonbrenas Could you please review this PR?
Please let me know if any changes or improvements are required.

@jonbrenas

Copy link
Copy Markdown
Collaborator

Thanks @rajat552. I think you have an interesting idea. However, this PR doesn't cover the first line of your stated problem at all (i.e., PCA and diversity stats) as it only applies to the sample metadata. I am pretty sure there are pandas functions that do most of what you are doing already.

@rajat552

rajat552 commented Mar 15, 2026

Copy link
Copy Markdown
Author

@jonbrenas
Thanks for the feedback!

You're right that the current implementation focuses on sample metadata rather than directly addressing PCA or diversity statistics. My intention with this PR was to introduce a small helper that exposes metadata schema information programmatically, which could serve as a building block for safer query construction.

While pandas does provide similar functionality, exposing this through the API could help users and programmatic clients discover valid metadata fields and values before constructing queries.

If you think this functionality should instead be implemented differently, or placed elsewhere in the API, I would be happy to adapt the implementation based on your suggestions.

@rajat552

Copy link
Copy Markdown
Author

@jonbrenas I have pushed a commit to resolve the linting issues reported by CI.
Please let me know if any further changes are required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

# Feature Request: Introduce a Structured Query Abstraction Layer to Support API Extensions and Natural Language Interfaces

2 participants