How would I go about making highlights interactive?

[tefkah]

Semi related to (#170)

Sorry if this seems a bit all over the place, I wanted to make clear what I had considered already!

I would like to be able to click on the highlights/render a button close to them. To do that, I will need to find where the current highlight is, which is quite tricky: what I'm doing at the moment is basically copying how Readium resolves a locator to a range, then finding where the range is on the page. Then I have toyed between rendering a button, or figuring out whether we are inside of a highlight from a point in the page. The latter is obviously not very good for accessibility.

So I have a couple of questions (to avoid an XY problem), from most specific to my goal (click on highlight to do something with it) to least

Are there any plans on providing interactable highlights in the toolkit?

I think the mobile toolkits have something like an onHighlightTapped. If something like this would be available in the ts-toolkit, it would solve my issue (obviously haha)

Are there plans to make it possible to retrieve highlights after creating them?

Currently you can do

nav.cFrames[0].msg.send("decorate", {
    group,
    action: "add",
    decoration: {
      locator: locator,
      style: {
        tint: 'green',
      },
      id,
    },
  })

But afterwards there is no real way of interacting with the highlights again (other than updating/removing them). If I could retrieve the highlight (the Highlight to be specific, or a reference to the created DOM nodes) by some ID, i would be able to resolve the position of them on the screen, and render buttons etc.

I know that the cFrames "api" will be deprecated at some point, so I doubt this will be the long term way of doing things anyway!

Is there an easy way to resolve a Locator to a Range?

Since I have a reference to a Locator for a highlight (it's necessary to make the highlight), I could figure out where to render a button if I could easily resolve it to a Range (or similar). I know Readium does this under the hood namely rangeFromLocator

ts-toolkit/navigator-html-injectables/src/helpers/locator.ts

Lines 10 to 69 in 587c788

	export function rangeFromLocator(doc: Document, locator: Locator) {
	try {
	const locations = locator.locations;
	const text = locator.text;
	if (text && text.highlight) {
	let root;
	if (locations && locations.getCssSelector()) {
	root = doc.querySelector(locations.getCssSelector()!);
	}
	if (!root) {
	root = doc.body;
	}
	const anchor = new TextQuoteAnchor(root, text.highlight, {
	prefix: text.before,
	suffix: text.after,
	});
	try {
	return anchor.toRange();
	} catch (error) {
	// We don't watch to "crash" when the quote is not found
	console.warn("Quote not found:", anchor);
	return null;
	}
	}
	if (locations) {
	let element = null;
	if (!element && locations.getCssSelector()) {
	element = doc.querySelector(locations.getCssSelector()!);
	}
	if (!element && locations.fragments) {
	for (const htmlId of locations.fragments) {
	element = doc.getElementById(htmlId);
	if (element) {
	break;
	}
	}
	}
	if (element) {
	const range = doc.createRange();
	if (element.childNodes.length === 0 || isReplacedLikeElement(element)) {
	range.selectNode(element);
	return range;
	}
	range.setStartBefore(element);
	range.setEndAfter(element);
	return range;
	}
	}
	} catch (e) {
	console.error(e);
	}
	return null;
	}

I basically ended up copying this, but this ends up duplicating a very large amount of code.

I think there might sort of be a way by storing an locator.locations.otherLocations.domRange, but from what I could find this

1. Does not seem to be used to actually navigate/highlight
2. I'm not really sure how to actually map a Range object to a DomRange, and could not find any examples of it!

Any obvious solution I missed?

I tried to look for alternatives but couldn't figure out anything else! May have missed it though!

[JayPanoz]

Cc'ing @chocolatkey as we know we need helpers (or some kind of API) for highlights and he has more experience than I currently do with this injectable/module.

Sorry I have some catching up to do myself as I have not had the opportunity to use it through Thorium Web yet, but as you could see, you are not the only one interested in this right now.

I will eventually come back at highlights through read-aloud so it sounds like a good time to collect feature requests and pain points.

[l-konrad]

Also interested!

[tefkah]

Coming back to this after a while of sitting on it.

I think 90% of my request could probably be solved with two changes

• Add a "on Decoration click" handler somehow. The implementation details i'll leave to you, I think the easiest would probably be to also add the divs to the page for the experimentalLayout, and just add onClicks to that. Then expose a decorationClick handler through the Navigator.
• Add some way to query the current Decorations. Just returning a list of all of them would be good enough IMO, so consumers can figure out how to interact with them ourselves. This is mostly an ergonomics issue: i could of course store all the highlights I make somewhere (which I do). It just feels a bit unnecessary because I know Readium is already keeping track of them!

I'm happy to have a go at implementing the above two changes if you would be up for that!

A last, optional change that would make it a lot easier to work with things:

• Expose something like rangeFromLocator

  ts-toolkit/navigator-html-injectables/src/helpers/locator.ts

  Lines 10 to 69 in 587c788

  	export function rangeFromLocator(doc: Document, locator: Locator) {
  	try {
  	const locations = locator.locations;
  	const text = locator.text;
  	if (text && text.highlight) {
  	let root;
  	if (locations && locations.getCssSelector()) {
  	root = doc.querySelector(locations.getCssSelector()!);
  	}
  	if (!root) {
  	root = doc.body;
  	}
  	const anchor = new TextQuoteAnchor(root, text.highlight, {
  	prefix: text.before,
  	suffix: text.after,
  	});
  	try {
  	return anchor.toRange();
  	} catch (error) {
  	// We don't watch to "crash" when the quote is not found
  	console.warn("Quote not found:", anchor);
  	return null;
  	}
  	}
  	if (locations) {
  	let element = null;
  	if (!element && locations.getCssSelector()) {
  	element = doc.querySelector(locations.getCssSelector()!);
  	}
  	if (!element && locations.fragments) {
  	for (const htmlId of locations.fragments) {
  	element = doc.getElementById(htmlId);
  	if (element) {
  	break;
  	}
  	}
  	}
  	if (element) {
  	const range = doc.createRange();
  	if (element.childNodes.length === 0 || isReplacedLikeElement(element)) {
  	range.selectNode(element);
  	return range;
  	}
  	range.setStartBefore(element);
  	range.setEndAfter(element);
  	return range;
  	}
  	}
  	} catch (e) {
  	console.error(e);
  	}
  	return null;
  	}

This would make so much more custom functionality possible. For instance, I wanted to add a little bookmark icon next to highlights. This is really quite impossible to do, because the Decoration api only allows you to add highlights. Any custom functionality is extremely limited because there's no real way to find where in the page a certain locator actually is. You can only go there using navigator.go.

I can see that this might be against the spirit of your idea to deprecate being able to access the cFrames of the navigator, as it kinda requires that access to be useful. Still: it would unlock a lot of extra functionality consumers of the library would be able to add

[JayPanoz]

Thanks for the extensive thoughts and ideas!

I have not been able to catch up yet, I'm slowly returning to ts-toolkit and Thorium Web after an intensive development cycle on Readium Speech, that should lead to read-aloud and Decorator eventually.

CC @mickael-menu to raise his awareness as we'd preferably want to align all toolkits. That is to say I'll also need his input and feedback at some point and it'd be better to have him early in this process.

[mickael-menu]

Interactive decorations are already supported according to the original proposal: https://readium.org/architecture/proposals/008-decorator-api.html#handling-user-clicks-on-decorations.

As long as you adhere to the spirit of the proposal, it's fine. You can adjust the API to make it more idiomatic to TypeScript. Here are the Swift and Kotlin versions for reference:

• https://github.com/readium/swift-toolkit/blob/4a4b7f91573a91d689e7b2556ed2243438ff35d3/Sources/Navigator/Decorator/DecorableNavigator.swift#L28-L49
• https://github.com/readium/kotlin-toolkit/blob/c59c0ef036906466b53bb028c24d29ab21df75ae/readium/navigator/src/main/java/org/readium/r2/navigator/DecorableNavigator.kt#L52-L88

If there is anything wrong or insufficient in the proposal, we can discuss it in an issue on the architecture repo.

For instance, I wanted to add a little bookmark icon next to highlights. This is really quite impossible to do, because the Decoration api only allows you to add highlights. Any custom functionality is extremely limited because there's no real way to find where in the page a certain locator actually is.

I'm not sure about the current state of the TypeScript implementation, but the original proposal should make this possible. You could achieve it with a single decoration using a custom HTML template for both the icon and highlight. Alternatively, and perhaps more effectively, you could use two separate decorations: one for the highlight and another for the icon.

https://readium.org/architecture/proposals/008-decorator-api.html#registering-decoration-styles

Add some way to query the current Decorations. Just returning a list of all of them would be good enough IMO, so consumers can figure out how to interact with them ourselves. This is mostly an ergonomics issue: i could of course store all the highlights I make somewhere (which I do). It just feels a bit unnecessary because I know Readium is already keeping track of them!

This shouldn't be necessary. Your host application owns the source of truth for highlights. Readium renders what you provide and manages highlight diffing for optimization, similar to how React works. Why do you need this, to be able to get the onClick event on each decoration?

[tefkah]

Thanks for the extensive reply @mickael-menu !

I don't think either click events or custom HTML templates are possible in the ts-toolkit currently! The way to interact with decorators appears to be quite low level at the moment, and only supports adding highlights (unless I'm missing something!).

I think the proposal should do almost all of what I would want!

This shouldn't be necessary. Your host application owns the source of truth for highlights. Readium renders what you provide and manages highlight diffing for optimization, similar to how React works. Why do you need this, to be able to get the onClick event on each decoration?

I think my suggestion here was a bit misguided and too specific to my situation.
It's a slightly different use-case I need which in a roundabout way finding the decorators could help with.
What I really want is to be able to find where a Locator which only has a { text: { }} component is on the page. That's also why I mentioned exposing the rangeFromLocator method.

One usecase is that I would like to implement custom scrolling methods. For instance, when a user clicks on an annotation they've made in a sidepanel, and it's in the same frame that's open right now, I would like to be able to scroll to that location when the layout mode is scrollable. The ts-toolkits built in navigation methods are too abrupt to provide a nice experience imo. Of course that could also be solved by the ts-toolkit providing many specific navigation options, but I think that would get messy very quickly.

Another is that I want to show a little modal above the decoration whenever the user either clicks on it directly, or clicks on it from the annotation panel. The former could be solved by providing some information about the boundaries of the highlight on click, but for the latter I would either need to be able to resolve the position of a text-based Locator to a screen/page-space one, or (which was my initial intention) I would figure out the location of the associated Decorator.
I can't remember exactly, but the thinking was something to go like

Stored Annotation w/Decoration id -> Navigator Decoration -> CSS.Highlight highlight -> Range -> Page/Screen coordinates

This can ofc be skipped if I can go from the stored annotation (which would have a Locatordirectly to aRange`.

I have considered using the domRange property for locations. I would then manually be able to convert that to a Range, and find the position on the page that way.
However I couldn't really find any examples of this being used anywhere, and AFAICT the ts-toolkit Navigator does not use it to actually resolve to places. I also just found it hard to figure out how it worked, given the lack of examples i could find! It also felt a little "unportable": not sure if highlights made in our apps (which use the swift/kotlin toolkits) would be able to provide the same kind of information.

Sorry this turned out to be rather long again! I just wanted to make I didn't actually miss a very obvious way of achieving what I wanted! I do think the way forward is to align the Decorator api of the ts-toolkit with those of the swift/kotlin ones.