Explicit sync hanging indefinitely

[masih]

Explicit sync was triggered without --resync or --depth flag.
Observed for /dns/staging-indexer-ipni-store-bucket.s3.us-east-2.amazonaws.com/https. hanging at:

storetheindex/server/admin/handler.go

Line 367 in ee8dd34

_, err := h.ingester.Sync(h.ctx, peerInfo, int(depth), resync)

[masih]

This then prevents further sync since the provider remains in the list of "already in progress" syncs until indexer is restarted. On restart we then get:

Cannot sync with peer	{"peerID": "12D3KooWPMQTKSMA3eFUxc23gBfMHEgzfk7W1TBezKNsBwPMRLQ7", "address": "/dns/staging-indexer-ipni-store-bucket.s3.us-east-2.amazonaws.com/https", "err": "context canceled"}

[masih]

--resync flag makes no difference

[masih]

Increasing the worker count to a very high number unblocked the ingest. This looks like a locking issue where if there are many concurrent syncs happening, a pending sync might never get scheduled (or take longer than it takes to ingest at least one ad from all other providers). Fair Usage is the feature that's required here.

The specific area in code where wait happens:

storetheindex/internal/ingest/ingest.go

Lines 594 to 626 in 826e603

	// the mutex from being unlocked.
	go func() {
	for range cq.Out() {
	}
	}()
	ing.outEventsMutex.Lock()
	defer ing.outEventsMutex.Unlock()
	pubEventsChans, ok := ing.outEventsChans[peerID]
	if !ok {
	log.Warnw("Advertisement processed notification already canceled", "peer", peerID)
	return
	}
	for i, ca := range pubEventsChans {
	if ca == events {
	if len(pubEventsChans) == 1 {
	if len(ing.outEventsChans) == 1 {
	ing.outEventsChans = nil
	} else {
	delete(ing.outEventsChans, peerID)
	}
	} else {
	pubEventsChans[i] = pubEventsChans[len(pubEventsChans)-1]
	pubEventsChans[len(pubEventsChans)-1] = nil
	ing.outEventsChans[peerID] = pubEventsChans[:len(pubEventsChans)-1]
	}
	close(events)
	break
	}
	}
	}
	ing.outEventsMutex.Lock()