Feedback on trust graph calculations

[ed-kung]

After having done a deep dive on SN's Web of Trust Algorithm, I have some feedback. Please don't view this as criticism: I think the current algorithm does what it's supposed to do and is directionally correct. These thoughts are shared in the spirit of discussing how the existing algorithm can be improved to make it more robust. It's also possible I made some mistakes so let me know if you see anything you disagree with or that I got wrong!

These comments all apply to the calculation of the trust graph, because everything else is a fairly straightforward application of the TrustRank algorithm.

1. The current system is gameable.

I didn't mention this in my SN post, but the current system is actually pretty gameable. That's because other users' trust of me is calculated based on how much they zap posts that I zapped first.

But the issue is that the before/after calculation is based on the timestamp of the first zap. I can easily game the system by zapping 1 sat on an item early, waiting to see if it becomes popular, and if it does, zap it a lot more. The timestamp associated with my action on the item will be very early, leading all the trust from other peoples' zaps to accumulate to myself.

2. Using the ratio of zap amounts leads to unintended scaling behavior.

Imagine three users, A, B, and C. They all agree with each other 100% of the time. However, users A and B always zap 21 sats, but user C always zaps 42 sats. Based on how "before" and "after" are calculated, the trust between C and the other users will be one half of the trust between A and B.

I understand the intent behind using the ratio of the zap amounts, but it seems like a person's baseline zapping behavior should be taken into consideration. I'm not sure what the solution is, but I think specifying an underlying behavioral model would help (see below).

3. It's unclear how to interpret the confidence interval calculation.

I understand the intent behind using the binomial proportion confidence interval, but I don't think it's well specified. The issue here is that the binomial model assumes an integer number of successes and trials, both of which should be non-negative.

But before - disagree can be negative, and b_total - after may not be an integer. The calculation still works and is directionally correct, but it's hard to interpret what the resulting number actually means and whether or not it's well scaled.

4. For users with long histories, it'll be hard to overcome historical behavior.

I believe a user's entire history of zaps is currently used to calculate trust graph. This makes it hard to overcome a long history of zapping behavior. It might make more sense to take the last $$N$$ item actions in a territory for each user when calculating the trust graph.

5. It's unclear what the underlying behavioral model is.

I think a lot of these problems can be solved by explicitly writing down an underlying behavioral model. It wouldn't have to be super realistic, it would just have to capture some basic features of user behavior. I'm not entirely sure what the model would look like, but I think it would involve at least the following ideas:

• An underlying similarity parameter $$\theta_{ij}$$ between users $$i$$ and $$j$$ that we're trying to learn about.
• A parameter describing the probability that user $$i$$ becomes aware of an item $$k$$.
• Conditional on two users $$i$$ and $$j$$ both being aware of an item, the conditional probability that they both zap/downzap/disagree on item $$k$$, as a function of their similarity parameter, with some adjustment based on their usual zapping amounts.
• Given the above, there should be a mapping from $$i$$ and $$j$$'s zapping behavior to an estimate of $$\theta_{ij}$$, and we should be able to calculate a confidence interval for it.

[huumn]

Man this is soooooo awesome thanks! To date, I've never had anyone review this code.

But the issue is that the before/after calculation is based on the timestamp of the first zap.

I'll switch it to max now that it's out of the closet. I remember noting this to myself 2 years ago when I wrote it, wanting to do a weighted average, but I totally forgot about it until now. Thanks!

Using the ratio of zap amounts leads to unintended scaling behavior.

Ideally we'd compare deviations from mean zapping amounts I think. Before a week ago, we only used zap counts and this was a lazy half step. Surprisingly it didn't seem to make much of a difference empirically.

At least for the toy example, C's trust of A and B is turned into a probability distribution, so if I only trust A and B, then I trust each 50% regardless of how much they're zapping. Still, your point stands, the effect of this is that I trust people who zap the same amount as me more - which is maybe not what we want. (The probability distribution thing gives me more confidence in this fudgeness than I probably deserve to give it.)

But before - disagree can be negative

There's a control statement that prevents this from being negative. If it's negative, we just score the user at 0. Still, ideally that wouldn't be necessary.

I believe a user's entire history of zaps is currently used to calculate trust graph.
It might make more sense to take the last N item actions in a territory for each user when calculating the trust graph.

Great suggestion. I've only considered creating a sliding window of time but not number of zaps.

I'm not entirely sure what the model would look like, but I think it would involve at least the following ideas

Wooo this is great!

[ed-kung]

Will take a look at coming up with a sensible behavioral model.