Skip to content

Commit 41157eb

Browse files
committed
📝 Address juror reliability concerns
1 parent 5f9190b commit 41157eb

File tree

1 file changed

+2
-1
lines changed

1 file changed

+2
-1
lines changed

‎Deep Funding.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ After participating in the ML competition and Prediction Market, and doing a few
4646
- **Juror Reliability**
4747
- So far, expert juror's pairwise comparisons have been inconsistent, noisy, and low in statistical power
4848
- Getting comparisons has been quite expensive in time and resources
49+
- The jury (secret) pool diversity is not guaranteed
4950
- Asking jurors "how much better" introduces order‑dependence and scale mismatch
5051
- Messy jurors have [disproportionate impact on the weights](https://davidgasquez.github.io/deepfunding-trial-data-analysis/#-robustness-checks)
5152
- Weights are not consistent due to the limited amount of data collected and the variance on it
@@ -104,7 +105,7 @@ Once the competition ends, extra comparisons could be gathered for projects that
104105
- Recommendation systems
105106
- Sports (elo)
106107
- RLHF
107-
- Pairwise make thins a decision (yes / no, this or that). No one knows what 3.4x better means
108+
- Pairwise make choices a simple decision (yes / no, this or that). No one knows what 3.4x better means
108109
- Occam's razor works here too: simple things generalize better
109110
- Intensity makes the distribution curve arbitrary
110111
- We should test the assumption experts jurors give good results. Jurors are messy and not well calibrated. Collecting more information from "expert" jurors will probably add more noise. We should instead assume noisy jurors and use techniques to deal with that.

0 commit comments

Comments
 (0)