Persist Peer Uptime

[carlaKC]

At present, we store the online and offline events of peers that we have channels open with to get an record of peer uptime. This information is lost on restart. Persisting this information provides a more long lived record of our peer’s uptime over the lifetime of a single channel, and a historical record of how a peer performed in past channels.

[peer-info-bucket]
    [peer-pubkey-bucket]
        [uptime-bucket]     
            [timestamp] online
            [timestamp] offline
            [timestamp] channel closed
    [peer-pubkey-bucket]
        [uptime-bucket]     
            [timestamp] offline
            [timestamp] online

Uptime queries are made over a desired time period - what was uptime last month/ over the last 7 days, for example - so entries are keyed by timestamp. We also require the raw online/offline events; there is no simple way to aggregate events and still allow queries over time ranges. Since our peer has no incentive to remain connected to us on channel close, we also need to differentiate between disconnections due to peer flakes and due to channel closes this can be done by adding a channel closed record for each peer.

Another consideration would be rate limiting the number of writes we allow each peer; we don’t want a flaky peer to fill up our disk. This DoS vector is an existing issue in our in-memory uptime tracking; we will add in memory events for every online/offline event, this issue should address this DoS vector as well. Rate limiting allows us to reduce the number of events we store (each event will require 9 bytes on disk, timestamp + uint8 enum).

Ideally we would like to maintain the accuracy of our uptime data, so we can scale our rate limiting with the degree to which our peer flaps, trading off accuracy for disk space as a peer requires more space. This flap rate can be tracked in the peer's bucket under its own key. We won't write each flap to disk, but rather flush our total count periodically so that this write can't be used to DoS us.

    [peer-pubkey-bucket]
        [uptime-bucket]     
        [flap-count-key]     

Thresholds tend to be magic numbers, so I would suggest that we pick a max on disk size we're ok with dedicating to these events for one peer for one month and work backwards form there.

We need to keep a record of our own downtime, if we do not, we will mistakenly interpret our own downtime as an online/offline period for each of our peers, depending on the state they were in when we went down.

• We record Alice as online, Bob as offline
• LND does down for 3 days
• On restart, we will interpret our on disk data as Alice having had 3 days of online time, and Bob having 3 days offline time

This could be done with an on-shutdown timestamp written to disk, but if lnd does not shutdown cleanly we will not have this event, and will have the same issue as before. Instead, we can track our own uptime in a special bucket in peer-info on a best effort basis. This can be done by writing a last-online timestamp to disk periodically (say every minute). On startup, we can record our own downtime by last liveliness timestamp we have as the time we went offline, and our current timestamp as the time we came back online.

As part of this change, I also suggest the addition of a dedicated uptime rpc endpoint, since adding a start/end time parameter to ListChannels, where uptime currently resides, does not make sense in the context of the call.

Rpc PeerUptime(PeerUptimeRequest) returns PeerUptimeResponse

Message PeerUptimeRequest{
  String peer_pubkey = 1;
  Uint64 start_time = 2;
  Uint64 end_time = 3;
}

Message PeerUptimeResponse{
  // The amount of time that we monitored our peer's uptime over the period queried.
  uint64 monitored_time = 1;

  // The amount of time that our peer was online for while we monitored it. 
  uint64 online_time = 2;

  // The amount of time that our node was down, and thus unable to monitor peer online status, for over the period queried.
  uint64 local_downtime = 3;
}

[carlaKC]

Done some rough calculations for how we could rate limit peers. Decided to keep it quite simple, and just upgrade rate limiting linearly. We can always move to a more complex algo if desired.

	// Rate limits indicates the rate limiting we apply according to
	// peer flap rate. We decide on the bucket we place our peer in by
	// calculating an index in this list = flapRate %20
	rateLimits := []time.Duration{
		time.Second,      // 0-199 flaps
		time.Second * 5,  // 200-399 flaps
		time.Second * 30, // 400-599 flaps
		time.Minute,      // 600-799 flaps
		time.Minute * 30, // 800-999 flaps
		time.Hour,        // 1000+ flaps
	}

	for _, rate := range rateLimits {
		fmt.Printf("%v reached after %v\n", rate, rate*200)
	}

	// Open question, how do we downgrade people when they have
	// a devops shitshow and bomb-out.
	// Perhaps write all time flap rate and

	// month is the amount of time that we have in a month
	month := time.Hour * 24 * 30
	entryCount := 0

	// magicNumber is the point at which we upgrade our peer to a more
	// rate limited tier, forged in the depth of handwave-mordor. Increasing
	// this number makes us less strict (they can store more events),
	// decreasing this number makes us more strict (we upgrade more quickly).
	magicNumber := 200

	// We start with elapsed time =0, then accumulate entries according to
	// our current tier. This has the effect of calculating the number of
	// events our current rate limited schedule will produce within a month.
	var elapsedTime time.Duration
	for elapsedTime <= month {
		entryCount++

		// We figure out the tier we currently belong in using magic
		tier := entryCount / magicNumber

		// If we are beyond our last tier, we just leave them at this
		// level. The worst case for a deadly spammy peer will then
		// be 7 bytes * number of hours in a month (assuming 1 H top
		// tier)
		if tier >= len(rateLimits) {
			tier = len(rateLimits) - 1
		}

		increment := rateLimits[tier]
		elapsedTime += increment
	}

	// Each DB entry will be 9 bytes (timestamp and enum).
	var bytes int64 = 9

	fmt.Printf("We allowed: %v entries in a month, %v bytes\n",
		entryCount, int64(entryCount)*bytes)

	// Last, we do a quick sanity check of how many entries a worst case
	// peer would generate if they operated for a month on our most rate
	// limited level.
	worstCase := int64(month / time.Hour)
	fmt.Printf("Worst case peer creates: %v entries, %v bytes\n",
		worstCase, worstCase*bytes)

Output with current values:

1s reached after 3m20s
5s reached after 16m40s
30s reached after 1h40m0s
1m0s reached after 3h20m0s
30m0s reached after 100h0m0s
1h0m0s reached after 200h0m0s
We allowed: 1614 entries in a month, 14526 bytes
Worst case peer creates: 720 entries, 6480 bytes

Open question as to how we downgrade peers that had a devops-disaster and flapped chaotically for 20 minutes then recovered. Perhaps we want to downgrade peers like that to get granularity back, since this rate limiting would be applied over the whole peer lifetime.

[carlaKC]

Rather than trying to store individual events on disk, which can lead to long scans, we can also just "ping to disk" like we currently do with flap rate. Every {interval - 30 mins?} we simply record the amount of time that we have observed the peer as online over that period, the amount of time we've actually been monitoring them (since we don't monitor when we have no channels) and our last write time. On restart, we can just pick up from where we left off, because we know how long we've been down for (based on last write timestamp), with an accuracy loss never > the interval that we write on.

[peer]
  [jan]
       [last write]
       [online seconds]
       [monitored seconds]
  [feb]
       [last write]
       [online seconds]
       [monitored seconds]

The downside of this approach is that we are limited to the granularity that we pick - eg, if we store information per-month, you can't get the uptime for a specific day in that month. However, since the space requirements of this approach are pretty small, we can pick a sufficiently small interval to record (eg, per day) so that we can still provide useful historical information.

We'll still track exact uptime events in memory, and will be able to provide more exact information in ListChannels. This new information can be surfaced in a dedicated Uptime api, which limits querying to the granularity that we set.