Not pure (but deterministic) lenses

[mitar]

So currently lenses have to be pure and cannot obtain additional data. This is an issue for cases like the Stripe API example from the doc. But in practice, such transformations are pretty common, at least in my case which is focusing on API versioning transformations. You move part of data to another API endpoint, for example. I was thinking that this can be nicely solved in the case of API versioning by having lens be able to call API at the version for which the lense is written. In this way the lense continues to work even many versions later and you use API versioning transformations to get data for the lense back-migrated from the latest API version to the version the lense expect.

The issue I am having is how would one describe such API requests in a declarative way, like current lenses are. I am finding that this is mostly impossible and it looks I would have to write imperative code for forward and backward migration. But then it is hard to also be able to automatically implement transformations of schemas, and so on.

So I am curious if you have any thoughts on this subject?

[teohhanhui]

Noticed this relevant section from their paper:

6.0.2 Augmenting data. A change to a schema may
require new data, or remove previously required data.
Although Cambria has support for providing default data in
the case of added or removed fields, it does not provide a
mechanism to look up missing data, and so cannot support
this kind of change today. To solve this problem, Cambria
needs a way to express dependencies on other data sources,
or perhaps some kind of callback to allow a lens author to
fetch data from other sources.

Similarly, moving data between documents poses
challenges. Splitting one document into many, or migrating data
from one document to another, are obvious use cases here.
Past systems like the XML-transforming language XSLT have
struggled with combining and splitting documents.

https://dl.acm.org/doi/epdf/10.1145/3447865.3457963

[michielbdejong]

Right! That quote from the ACM paper is shorter than the corresponding section in their blogpost, leaves out the example details from Stripe and GitHub, but adds the mention of XSLT.

I think this is also related to the 'Extract Entity' challenge