SchemaView - light schema access layer

[rschili]

imodel-native: iTwin/imodel-native#1369

Some diagrams refer to this as "Runtime Schemas" which was my working title. I didn't update all the pictures, just be aware. The final name we're going with is SchemaView.

Motivation

Yet another ECSchema layer for typescript, so let me explain:

• ecschema-metadata (SchemaContext) is the full-fidelity schema toolkit. It models every detail of the EC specification, but loading it is expensive: one synchronous RPC per schema, large memory footprint (~150-200 MB for a large iModel), and it blocks the backend event loop during loading which is the primary concern here.
• Incremental schema loading was implemented as a lighter alternative but remains incomplete and does not cover all use cases.
• ECSQL against ECDbMeta works for targeted queries but is awkward for traversal patterns (e.g. walking all inherited properties of a class) and does not cache on the TS side. For one-shot questions this is perfectly viable!
• Older backend metadata layer was discontinued and pulled single classes - this is somewhat similar to that again but loads one bigger chunk at once.

This PR introduces an optimized schema bundle which is transported via a single async call and parsed into an in-memory cache that provides fast synchronous access. It is lossy by design - it drops what runtime consumers do not need (units, formats, phenomena, full custom attribute instances) to keep the payload small and parsing fast.

Why one single bundle?
I did extensive testing. Of course it could be broken into smaller chunks, lazy loaded, but we have done all of that before - every approach has tradeoffs. This is keeping it simple and in testing performs rather well, even in scnearios in which it is weak by design.

The most extreme example is a single "check if class A IS of class B" on an iModel with huge schemas, magnitudes larger than an average iModel.
In this example, SchemaView takes about a .8 seconds spinup delay, then it responds instantaneously.
Asking the question directly via ECSql takes about 0.002s. With smaller iModels SchemaView catches up faster - ECSql is always available as an alternative option.

Design

Transport: A new PRAGMA schema_view(N) in ECDb returns the binary blob via ConcurrentQuery. No new RPC methods - it flows through the existing queryRows path. The pragma accepts a format version parameter for forward compatibility.

Binary format: C++ writer reads ec_ tables directly and produces a compact blob with:

• String lookup table (deduplicates repeated schema/class/property names)
• Property definition deduplication (many class-property pairs share identical definitions)
• Per-item ecInstanceId (row ID from ec_ tables) for fallback to ECSQL when needed

Cache invalidation: Uses PRAGMA checksum(ecdb_schema) (SHA3-256) to detect schema changes. Schema imports or changeset pulls that modify schemas automatically invalidate the cache.

Performance characteristics

Initial investigation into how much metadata we ingest, tested on an iModel with enormous metadata and poor performance:

Performance comparison walking all properties:

Performance comparison on simulated frontend with limited network bandwidth:

Memory footprint at runtime:

[hl662]

When would users want to use this compared to the full fledged schema context?

[rschili]

When would users want to use this compared to the full fledged schema context?

I was hoping for the answer to be "always" unless they are trying to store/author schemas.

[grigasp]

Overall looks good.

A few comments:

• Not sure the prefix Runtime fits the purpose of the API. Based on the docs, it sounds like SchemaContext is reserved for editing use cases, but that happens during runtime as well.

• I think a better distinction between schemaContext and getSchemas() on IModelConnection and IModelDb is needed. Why one is a getter and the other is a method?

  The schemaContext getter is still marked @beta, so I think we could at least rename it. However, considering it becomes a niche API used only for specific situations, I would consider removing it completely and instead exposing a global factory function to create a SchemaContext when it's needed.

• Since RuntimeSchemaContext is "lossy", I think the docs should be very clear with recommendations on what to do when that lost information is needed.

  For example, in my case I used SchemaContext API to traverse from Property -> KindOfQuantity -> Format -> Unit. With the new API, KindOfQuantity only has presentation formats string. What do I do from there?

• I think we'll want to bump IModelReadRpcInterface.version to ensure frontends can use the new pragma.

[rschili]

@grigasp

* Not sure the prefix `Runtime` fits the purpose of the API. Based on the docs, it sounds like `SchemaContext` is reserved for editing use cases, but that happens during runtime as well.

Fair point - "Runtime" is ambiguous since SchemaContext is also used at runtime. I thought hard about this, considered alternatives like CompactSchemaContext, LightSchemaContext, SchemaSnapshot, etc., but none clearly communicated the intent better.

The distinction we're drawing is "optimized for runtime consumption" (pre-loaded, read-only, synchronous, compact) vs "optimized for schema introspection and editing" (lazy-loaded, mutable, full-fidelity). "Runtime" is imperfect, changing it now would be a substantial update.
The prefix does carry the connotation of "this is the thing you reach for in hot paths." Open to a better name if one comes to mind, but the alternatives seemed worse.

* I think a better distinction between `schemaContext` and `getSchemas()` on `IModelConnection` and `IModelDb` is needed. Why one is a getter and the other is a method?
  The `schemaContext` getter is still marked `@beta`, so I think we could at least rename it. However, considering it becomes a niche API used only for specific situations, I would consider removing it completely and instead exposing a global factory function to create a `SchemaContext` when it's needed.

I think keeping it as a getter is the right call here. schemaContext and getSchemas() serve fundamentally different access patterns - schemaContext returns a thin wrapper object instantly (the real work happens lazily on nested calls like getSchemaItem()), while getSchemas() does real async work up front to fetch and deserialize the binary blob. Making one a getter and the other a method reflects that difference: synchronous, cheap construction vs. async, non-trivial I/O. A factory function would obscure that distinction and make the simple case harder to reach. I agree the naming asymmetry looks inconsistent at first glance, but I think it's actually communicating something useful about the cost model.

  For example, in my case I used `SchemaContext` API to traverse from `Property` -> `KindOfQuantity` -> `Format` -> `Unit`. With the new API, `KindOfQuantity` only has presentation formats string. What do I do from there?

For now you would have to use ECSql to pull that remaining info on demand. Let's dive into that and better underatand what we need. Modelling the whole units object model into this seems overkill to me. But I agree, if you DO need it, the path should be simple. Maybe RuntimeProperty could expose a helper method that does the work for you?

* I think we'll want to bump `IModelReadRpcInterface.version` to ensure frontends can use the new pragma.

done

[grigasp]

@rschili

Fair point - "Runtime" is ambiguous since SchemaContext is also used at runtime. I thought hard about this, considered alternatives like CompactSchemaContext, LightSchemaContext, SchemaSnapshot, etc., but none clearly communicated the intent better.

The distinction we're drawing is "optimized for runtime consumption" (pre-loaded, read-only, synchronous, compact) vs "optimized for schema introspection and editing" (lazy-loaded, mutable, full-fidelity). "Runtime" is imperfect, changing it now would be a substantial update. The prefix does carry the connotation of "this is the thing you reach for in hot paths." Open to a better name if one comes to mind, but the alternatives seemed worse.

I would consider ReadonlySchemaContext.

I think keeping it as a getter is the right call here. schemaContext and getSchemas() serve fundamentally different access patterns - schemaContext returns a thin wrapper object instantly (the real work happens lazily on nested calls like getSchemaItem()), while getSchemas() does real async work up front to fetch and deserialize the binary blob. Making one a getter and the other a method reflects that difference: synchronous, cheap construction vs. async, non-trivial I/O. A factory function would obscure that distinction and make the simple case harder to reach. I agree the naming asymmetry looks inconsistent at first glance, but I think it's actually communicating something useful about the cost model.

Personally, a getter vs a method wouldn't make me realize that one is more expensive than the other. FWIW, I would probably think a getter is cheaper.

However, I'd like to reiterate on my suggestion to remove schemaContext getter completely. If we say that it's going to be needed in less than 10% of use cases, then why confuse all the consumers with two similar but different options? We could provide quick & simple schema access that suits 90% of consumers, and the rest could be forwarded to an API that lets them create SchemaContext. I think it's fair to say that more complex use cases require more complex setup.

But I agree, if you DO need it, the path should be simple. Maybe RuntimeProperty could expose a helper method that does the work for you?

I don't think we'll be able to stop looking at schema-based formats/units anytime soon. Even when the full replacement is ready, we'll probably want to look at what's in schema as a fallback. It doesn't necessarily have to be simple, but it has to be clear.

My preference would be for RuntimeKoQ.presentationFormats to return something like Array<{ formatName: string; precision: number; unitName: string }>. Example:

// koq format spec:
//   f:DefaultRealU(4)[u:M_PER_SEC_SQ];f:DefaultRealU(4)[u:CM_PER_SEC_SQ];f:DefaultRealU(4)[u:FT_PER_SEC_SQ]
// formats array:
[
  { formatName: "Formats.DefaultRealU", precision: 4, unitName: "Units.M_PER_SEC_SQ" },
  { formatName: "Formats.DefaultRealU", precision: 4, unitName: "Units.CM_PER_SEC_SQ" },
  { formatName: "Formats.DefaultRealU", precision: 4, unitName: "Units.FT_PER_SEC_SQ" },
]

That way we could use formatName and unitName to pull the details we need.

[hl662]

I don't think we'll be able to stop looking at schema-based formats/units anytime soon. Even when the full replacement is ready, we'll probably want to look at what's in schema as a fallback. It doesn't necessarily have to be simple, but it has to be clear.

I agree, format set fotmatsProvider already has the concept of a fallback provider to lean into, a schema formats provider naturally fits that. If need be, we can extend the formatsProvider interface to allow an optional fallback formatsProvider within, bringing that functionality out of FSFormatsProvider. And do something similar with unitsProvider and a fallback of some sort.

[grigasp]

I looked over the public API surface and how SchemaView is maintained by and exposed through IModelDb and IModelConnection - LGTM. The only thing I don't really like is that now we'll have schemaContext getter + getSchemaView() method, but it's just my opinion, I understand the unwillingness to break the API.

I didn't look at at the tests or internals of SchemaView implementation. Hopefully someone from the iModels / EC team can do that.