Ingestion can miss adverts when multiple providers are used in a chain.

[alanshaw]

When ingesting adverts (after a sync), we traverse the chain from the sync'd head until we find an advert that has already been processed ingest.go#L951.

If an advert has not already been processed, we add it to a map of adverts to process keyed by advert provider.

We later iterate over the adverts grouped by provider ingest.go#L1015. Per usual Go map iterations, the order is random.

Each iteration calls ingestWorkerLogic (ingest.go#L1034) where it iterates over the adverts from oldest to newest. As it ingests, it marks each advert "processed" ingest.go#L1171 and as a side effect, marks each advert as the latest advert ingested ingest.go#L555 per publisher.

Marking adverts as processed is problematic in failure cases - adverts for other providers in the map may appear earlier in the chain, and yet a later advert has been marked as processed and the latest advert.

In short, this can cause the ingestion process to skip chunks of a chain permanently. For example, when ingesting adverts for one provider fails (e.g. network error), and a subsequent provider's adverts that exist later in the chain are ingested successfully. When this happens, the next time ingest runs, it'll terminate at the latest advert recorded, which is after adverts that have not yet been ingested.

I hope I am reading the code correctly. It is not easy to follow. I'd appreciate if someone could confirm this is an issue.

[gammazero]

The chain published by a publisher can only have adverts for a single provider. If a publisher wants to publish adverts for another provider, then they need to use a different chain to publish those.

If a single chain ever publishes adverts for multiple providers than that need to be prevented.

[willscott]

i think this is the distinction between 'publisher' and 'provider', right?

in that with extended providers one can have an endpoint for cids that is scoped to a single context ID, and have multiple of those such that what's returned is different for different CIDs within one advertisement chain. I believe this is what @alanshaw is referring to. I think that would still be considered a single 'provider' under your definition, right, @gammazero ?

[gammazero]

Yes, by my definition that would still be considered a "single provider", even if that provider had multiple content endpoints available as "extended providers".

I do not think @alanshaw is considering "extended providers" as those are not seen as separate providers, but rather just additional endpoints and optionally IDs that get returned with queries.

The case that @alanshaw describes is that:

when ingesting adverts for one provider fails (e.g. network error), and a subsequent provider's adverts that exist later in the chain are ingested successfully. When this happens, the next time ingest runs, it'll terminate at the latest advert recorded, which is after adverts that have not yet been ingested.

The indexer tracks progress for a publisher (of index data), not for the provider(s) (of stored data) described by the advertisements. So, if an error occurs and ingestion is stopped, when it is resumed later the indexer will get the latest advert that has been processed ingest.go#771 for the publisher, and walk the chain from the head back to the latest advert ingested ingest.go#L775-778 for the publisher. The walk of the chain stops at the chain publisher's last processed advert, seen here, subscriber.go#L444. The dagsync logic guarantees that there is only one sync at a time being done for a publisher (see here).

So, there should not be anything missed on the chain because of different providers on the chain, although that is not supported. I think the display of current progress in the /providers/ information is misleading, because it shows the most recent advertisement on the chain for a publisher of that chain, not for the specific provider. This is because there is the assumption that a chain only publishes adverts for one provider.

[alanshaw]

I do not think @alanshaw is considering "extended providers" as those are not seen as separate providers, but rather just additional endpoints and optionally IDs that get returned with queries.

Correct, I'm not referring to extended providers.

So, there should not be anything missed on the chain because of different providers on the chain, although that is not supported.

Ok, so then this comment is confusing to me, and I assume incorrect?

storetheindex/internal/ingest/ingest.go

Lines 947 to 949 in d6a889e

	// Group the CIDs by the provider. Most of the time a publisher will
	// only publish Ads for one provider, but it's possible that an ad
	// chain can include multiple providers.

I don't understand why the code groups adverts by provider, if it is not supported for there to be multiple providers on a single chain?

storetheindex/internal/ingest/ingest.go

Line 941 in d6a889e

adsGroupedByProvider := map[peer.ID][]adInfo{}

The entire comment above that line is confusing - I don't believe that function will receive adverts from multiple chains as the comment suggests. Could you help me understand?

[alanshaw]

So, if an error occurs and ingestion is stopped, when it is resumed later the indexer will get the latest advert that has been processed ingest.go#771 for the publisher, and walk the chain from the head back to the latest advert ingested ingest.go#L775-778 for the publisher. The walk of the chain stops at the chain publisher's last processed advert, seen here, subscriber.go#L444.

Yes, but the problem is that adverts are grouped by provider.

e.g.

map{
  Provider A => advert 1, 2, 3
  Provider B => advert 4
}

We iterate over the map, provider A first, provider B second.

1. Publisher ingests advert 1 from Provider A and then fails on advert 2.
2. Latest advert for Publisher is now advert 1. Note error handling does not prevent iteration continuing.
3. Publisher ingests advert 4 from Provider B
4. Latest advert for Publisher is now advert 4

Advert 2 and 3 are never ingested.

[alanshaw]

@gammazero any thoughts?

[gammazero]

I don't understand why the code groups adverts by provider, if it is not supported for there to be multiple providers on a single chain?

This comment does seem to imply that multiple providers on the same chain is possible... I guess maybe a publisher can publish for the different publishers on the same chain. I did not remember that being allowed... but I guess so.

Either way, a reason to group ads by provider is that that work pertaining to a specific provider can be processed by a single worker. In other words, no concurrent work for the same provider. This ensures that a provider's ads are processed in order.

Processing ads for a provider in order is needed to ensure that only the most recent updates to address and extended providers info is applied to that provider. It allows for optimizations like this so that only the latest provider information needs to be looked at.

[gammazero]

The entire comment above that line is confusing - I don't believe that function will receive adverts from multiple chains as the comment suggests.

This function is being called concurrently for chains published by different publishers. Suppose there are two different publishers serving ads on behalf of the same provider. The indexer will skip ads that it is reading from one chain if that ad has already been read from another chain. It can determine that ads are identical if their CIDs are the same.

Furthermore, if the indexer detects that is is reading an ad that has previously been read from another chain, then the indexer also knows that all of the previous ads on chain being read have also been previously read. This is because the current ad CID is calculated over the ad data plus the CID of the previous ad in the chain (in blockchain-esque fasion). So, if two ads on two different chains have the same CID, this means that all previous ads in both chains are identical.

In short, if the indexer sees an ad that has already been processed, then it can stop processing the chain that it is reading.

[gammazero]

Yes, but the problem is that adverts are grouped by provider. ... Advert 2 and 3 are never ingested.

Right, and the indexer is not saving the place it left off in the chain by provider, it is saving it by publisher, as seen here and here.

So, yes, you are correct. Those ads would be lost. Which leads me back to my initial belief, that publishing ads for multiple providers, on the same ad chain, is not supported.

So, where to go from here...

• We can track chain ingestion per provider. However that will cause other complications, like needing to skip already processed portions of chains for other providers. So, that seems least preferable.
• We can stop grouping ads by provider and process the ads from the chain in earliest to latest order. Optimizations like ignoring all but the most recent address updates for a provider will still work. Since ad processing will not be serialized by provider, this means that no two ad chains can have ads for the same provider. ...which they probably should not anyway since there is no way to determine the correct order across chains unless the chains are identical. So this seems reasonable, and would likely simplify the indexer code significantly.
• Or, we can say that publishing ads for multiple providers on the same chain is not supported, and then go back and do the second option at some later point in time.