Add Result.extra for arbitrary attributes + flexible objective classes (MinimizeAttribute/MaximizeAttribute)

[jakepu]

Summary

• Introduces a mutable Result.extra dict for attaching custom per-result attributes without schema changes.
• Adds MinimizeAttribute and MaximizeAttribute objectives that can optimize on any built-in Result field or a key in Result.extra.
• Updates docs and adds an example and tests.
• Bumps DB version to 0.1 to avoid silently running on outdated schemas.

Motivation

Today, adding a new metric requires modifying the Result model or using a sidecar table. This PR makes it simple and first-class to attach arbitrary attributes to results and use them directly as optimization metrics.

What’s changed

• resultsdb/models
  • Result.extra: MutableDict-backed PickleType to store arbitrary attributes.
  • Helper methods: set_attribute(), get_attribute(), update_attributes().
• search/objective
  • MinimizeAttribute(attribute_name, missing_value=None)
  • MaximizeAttribute(attribute_name, missing_value=None)
  • If attribute_name is a concrete column (e.g., time, accuracy), ordering is done in SQL; otherwise, comparison uses values from Result.extra.
• docs
  • README: “Attaching custom attributes to Result” and “Using custom attributes as metrics.”
• examples
  • examples/py_api/custom_metric_example.py: Demonstrates optimizing a custom qps metric via MaximizeAttribute.
• tests
  • tests/test_custom_metric.py: Verifies extra helpers and attribute objectives.
• DB
  • DB_VERSION bumped to 0.1.

Usage

Attach metrics:

from opentuner import Result

# During measurement
return Result(time=elapsed_s).update_attributes({
  'qps': throughput,
  'p95_ms': p95_latency,
})

Optimize built-ins or custom metrics:

from opentuner.search.objective import MinimizeAttribute, MaximizeAttribute

# Built-ins (uses SQL ordering)
objective = MinimizeAttribute('time')          # like MinimizeTime
objective = MaximizeAttribute('accuracy')      # like MaximizeAccuracy

# Custom (uses Result.extra)
objective = MaximizeAttribute('qps', missing_value=float('-inf'))
objective = MinimizeAttribute('p95_ms', missing_value=float('inf'))

Example

See examples/py_api/custom_metric_example.py for a runnable script optimizing a synthetic qps metric in Result.extra.

Backward compatibility

• Existing code is unaffected; built-in objectives still work.
• New classes are optional APIs.
• For custom metrics, ordering falls back to Python-level comparisons (no SQL ORDER BY); acceptable for most use cases.

Database migration

• DB version bumped to 0.1. Existing databases will be rejected by the connector to prevent mismatched schemas.
• Recommended upgrade path:
  • SQLite: back up and remove your old DB file, or reinitialize with a new path.
  • Other backends: reinitialize the schema as appropriate for your environment.

Testing

• Added tests/test_custom_metric.py.
• All tests pass locally:
  • 25 passed, 1 warning (SQLAlchemy 2.0 deprecation about declarative_base).

Quick start in a venv:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt -r optional-requirements.txt
pytest -q

Files changed

• opentuner/resultsdb/models.py: add Result.extra, helper methods
• opentuner/search/objective.py: add MinimizeAttribute, MaximizeAttribute
• opentuner/resultsdb/connect.py: bump DB_VERSION to 0.1
• examples/py_api/custom_metric_example.py: new example
• tests/test_custom_metric.py: new tests
• README.md: docs section and examples
• CHANGES.txt: changelog entry

Notes and robustness considerations

• MutableDict ensures in-place updates on Result.extra are tracked by SQLAlchemy without reassigning the dict.
• Attribute objectives implement result_relative() to integrate with existing relative comparisons.
• Using custom metrics does not add DB indexes (by design); if you need DB-level sorting/filtering on a custom attribute, consider a dedicated column or a sidecar table with its own index.
• The SQLAlchemy deprecation warning about declarative_base is pre-existing and unrelated to this change.

Checklist

• Feature implemented
• Tests added and passing
• Documentation updated
• Example added
• DB version bumped with clear migration note