Skip to content

test: move example reference values into the scripts themselves #92

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
endolith wants to merge 2 commits into from
Open

test: move example reference values into the scripts themselves #92

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c1099ec
test: move example reference values into the scripts themselves
endolith Aug 5, 2026
28cb15e
test: require exact method/figure keys in example checks
endolith Aug 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions examples/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,15 @@

These are some example scripts to demonstrate the various simulations that can be done, and to verify the simulator by reproducing results of already-published works.

Scripts double as tests: each one that has expected output defines a
`reference_table` (the values its computed `table` is checked against, in the
same order as the script's columns) and a `tolerance` (the absolute
comparison tolerance). `tests/test_examples.py` runs each script and checks
`table` against `reference_table`. Where a script reproduces the published
results, the reference values come from the paper; where it does not (see
issues #88 and #91), the reference is the script's "Typical result" as a
regression guard until the discrepancy is fixed.

## Wikipedia

### Likelihood of a Condorcet cycle
Expand Down
38 changes: 35 additions & 3 deletions examples/merrill_1984_fig_2c_2d.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,8 @@
'Plurality': {2: 100.0, 3: 51.3, 4: 36.2, 5: 21.0, 7: 7.8},
}

table = {}

for fig, disp, ymin, orig in (('2.c', 1.0, 50, merrill_fig_2c),
('2.d', 0.5, 0, merrill_fig_2d)):

Expand Down Expand Up @@ -128,24 +130,25 @@

for method in ('Black', 'Coombs', 'Borda', 'Approval', 'Hare', 'Runoff',
'Plurality'):
x, y = zip(*sorted(orig[method].items()))

Check failure on line 133 in examples/merrill_1984_fig_2c_2d.py

View workflow job for this annotation

GitHub Actions / lint

ruff (B905) 

examples/merrill_1984_fig_2c_2d.py:133:16: B905 `zip()` without an explicit `strict=` parameter
  help: Add explicit value for parameter `strict=`

Check failure on line 133 in examples/merrill_1984_fig_2c_2d.py

View workflow job for this annotation

GitHub Actions / lint

ruff (B905) 

examples/merrill_1984_fig_2c_2d.py:133:16: B905 `zip()` without an explicit `strict=` parameter
  help: Add explicit value for parameter `strict=`
plt.plot(x, y, ':', lw=0.8)

# Restart color cycle, so result colors match
plt.gca().set_prop_cycle(None)

table = []
table[fig] = []

# Of those elections with CW, likelihood that method chooses CW
x_cw, y_cw = zip(*sorted(condorcet_winner_count['CW'].items()))

Check failure on line 142 in examples/merrill_1984_fig_2c_2d.py

View workflow job for this annotation

GitHub Actions / lint

ruff (B905) 

examples/merrill_1984_fig_2c_2d.py:142:18: B905 `zip()` without an explicit `strict=` parameter
  help: Add explicit value for parameter `strict=`

Check failure on line 142 in examples/merrill_1984_fig_2c_2d.py

View workflow job for this annotation

GitHub Actions / lint

ruff (B905) 

examples/merrill_1984_fig_2c_2d.py:142:18: B905 `zip()` without an explicit `strict=` parameter
  help: Add explicit value for parameter `strict=`
for method in ('Black', 'Coombs', 'Borda', 'Approval', 'Hare', 'Runoff',
'Plurality'):
x, y = zip(*sorted(condorcet_winner_count[method].items()))

Check failure on line 145 in examples/merrill_1984_fig_2c_2d.py

View workflow job for this annotation

GitHub Actions / lint

ruff (B905) 

examples/merrill_1984_fig_2c_2d.py:145:16: B905 `zip()` without an explicit `strict=` parameter
  help: Add explicit value for parameter `strict=`

Check failure on line 145 in examples/merrill_1984_fig_2c_2d.py

View workflow job for this annotation

GitHub Actions / lint

ruff (B905) 

examples/merrill_1984_fig_2c_2d.py:145:16: B905 `zip()` without an explicit `strict=` parameter
  help: Add explicit value for parameter `strict=`
CE = np.array(y)/y_cw
plt.plot(x, CE*100, '-', label=method)
table.append([method, *CE*100])
table[fig].append([method, *CE*100])

print(tabulate(table, ["Method", *x], tablefmt="pipe", floatfmt='.1f'))
print(tabulate(table[fig], ["Method", *x], tablefmt="pipe",
floatfmt='.1f'))
print()

plt.plot([], [], 'k:', lw=0.8, label='Merrill') # Dummy plot for label
Expand All @@ -155,3 +158,32 @@
plt.ylim(ymin, 102)
plt.xlim(1.8, 7.2)
plt.show()

# Regression reference from the "Results with 500_000 elections" tables in the
# docstring, ordered by n_cands_list. These sims do not reproduce Merrill's
# published figures exactly (discrepancies up to ~7%, see issue #88), so this
# is only a regression guard against breaking the current close-enough output
# until that discrepancy is fixed.
reference_table = {
'2.c': {
'Black': (100.0, 100.0, 100.0, 100.0, 100.0, 100.0),
'Coombs': (100.0, 99.4, 98.6, 97.8, 96.9, 96.0),
'Borda': (100.0, 91.4, 89.2, 87.1, 85.7, 84.6),
'Approval': (100.0, 85.9, 79.8, 73.9, 70.1, 66.8),
'Hare': (100.0, 94.1, 86.6, 78.9, 71.7, 65.2),
'Runoff': (100.0, 94.1, 87.1, 79.7, 72.8, 66.1),
'Plurality': (100.0, 80.6, 67.6, 57.4, 49.3, 42.6),
},
'2.d': {
'Black': (100.0, 100.0, 100.0, 100.0, 100.0, 100.0),
'Coombs': (100.0, 98.2, 95.9, 93.4, 90.9, 88.4),
'Borda': (100.0, 89.2, 86.3, 83.8, 82.1, 80.8),
'Approval': (100.0, 84.0, 76.9, 71.5, 67.8, 64.7),
'Hare': (100.0, 72.2, 50.3, 35.8, 26.0, 19.7),
'Runoff': (100.0, 72.2, 50.6, 35.3, 24.4, 16.9),
'Plurality': (100.0, 55.9, 34.7, 21.5, 13.5, 8.5),
},
}

# Absolute tolerance (percentage points) for test_examples.py
tolerance = 5.0
42 changes: 39 additions & 3 deletions examples/merrill_1984_fig_2c_2d_updated.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,8 @@
star(honest_normed_scores(utilities, 5), tiebreaker),
}

table = {}

for fig, disp, ymin in (('2.c', 1.0, 50),
('2.d', 0.5, 0)):

Expand Down Expand Up @@ -113,18 +115,19 @@
plt.title(f'Figure {fig}: Condorcet Efficiency under Spatial-Model '
f'Assumptions [Disp: {disp}]')

table = []
table[fig] = []

# Of those elections with CW, likelihood that method chooses CW
x_cw, y_cw = zip(*sorted(condorcet_winner_count['CW'].items()))

Check failure on line 121 in examples/merrill_1984_fig_2c_2d_updated.py

View workflow job for this annotation

GitHub Actions / lint

ruff (B905) 

examples/merrill_1984_fig_2c_2d_updated.py:121:18: B905 `zip()` without an explicit `strict=` parameter
  help: Add explicit value for parameter `strict=`

Check failure on line 121 in examples/merrill_1984_fig_2c_2d_updated.py

View workflow job for this annotation

GitHub Actions / lint

ruff (B905) 

examples/merrill_1984_fig_2c_2d_updated.py:121:18: B905 `zip()` without an explicit `strict=` parameter
  help: Add explicit value for parameter `strict=`
for method in ('Condorcet RCV', 'Coombs', 'STAR', 'Borda', 'Score',
'Approval (opt.)', 'Hare RCV', 'Top-2 Runoff', 'Plurality'):
x, y = zip(*sorted(condorcet_winner_count[method].items()))

Check failure on line 124 in examples/merrill_1984_fig_2c_2d_updated.py

View workflow job for this annotation

GitHub Actions / lint

ruff (B905) 

examples/merrill_1984_fig_2c_2d_updated.py:124:16: B905 `zip()` without an explicit `strict=` parameter
  help: Add explicit value for parameter `strict=`

Check failure on line 124 in examples/merrill_1984_fig_2c_2d_updated.py

View workflow job for this annotation

GitHub Actions / lint

ruff (B905) 

examples/merrill_1984_fig_2c_2d_updated.py:124:16: B905 `zip()` without an explicit `strict=` parameter
  help: Add explicit value for parameter `strict=`
CE = np.array(y)/y_cw
plt.plot(x, CE*100, '-', label=method)
table.append([method, *CE*100])
table[fig].append([method, *CE*100])

print(tabulate(table, ["Method", *x], tablefmt="pipe", floatfmt='.1f'))
print(tabulate(table[fig], ["Method", *x], tablefmt="pipe",
floatfmt='.1f'))
print()

plt.legend()
Expand All @@ -133,3 +136,36 @@
plt.ylim(ymin, 102)
plt.xlim(1.8, 7.2)
plt.show()

# Regression reference from the "Results with 100_000 elections" tables in the
# docstring, ordered by n_cands_list. These sims do not reproduce Merrill's
# published figures exactly (discrepancies up to ~7%, see issue #88), so this
# is only a regression guard against breaking the current close-enough output
# until that discrepancy is fixed.
reference_table = {
'2.c': {
'Condorcet RCV': (100.0, 100.0, 100.0, 100.0, 100.0, 100.0),
'Coombs': (100.0, 99.4, 98.6, 97.8, 96.9, 96.0),
'STAR': (100.0, 97.8, 94.7, 92.1, 89.9, 88.0),
'Borda': (100.0, 91.4, 89.2, 87.1, 85.8, 84.8),
'Score': (100.0, 88.7, 84.6, 82.7, 81.3, 79.9),
'Approval (opt.)': (100.0, 86.0, 79.7, 73.9, 70.5, 67.0),
'Hare RCV': (100.0, 94.1, 86.6, 79.0, 71.3, 65.1),
'Top-2 Runoff': (100.0, 94.1, 87.1, 79.9, 72.8, 65.9),
'Plurality': (100.0, 80.6, 67.8, 57.3, 49.1, 42.7),
},
'2.d': {
'Condorcet RCV': (100.0, 100.0, 100.0, 100.0, 100.0, 100.0),
'Coombs': (100.0, 98.3, 95.7, 93.4, 90.8, 88.5),
'STAR': (100.0, 96.5, 91.5, 87.2, 83.1, 79.8),
'Borda': (100.0, 89.2, 86.4, 83.9, 82.2, 80.7),
'Score': (100.0, 86.2, 80.5, 77.4, 74.5, 72.1),
'Approval (opt.)': (100.0, 83.7, 76.9, 71.6, 67.6, 64.6),
'Hare RCV': (100.0, 72.4, 50.4, 35.6, 26.2, 19.8),
'Top-2 Runoff': (100.0, 72.4, 50.5, 35.3, 24.5, 17.1),
'Plurality': (100.0, 56.3, 34.8, 21.5, 13.5, 8.6),
},
}

# Absolute tolerance (percentage points) for test_examples.py
tolerance = 5.0
38 changes: 35 additions & 3 deletions examples/merrill_1984_fig_4a_4b.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,8 @@
'Plurality': {2: 100.0, 3: 41.1, 4: 27.0, 5: -1.0, 7: -9},
}

table = {}

for fig, disp, ymin, orig in (('4.a', 1.0, 55, merrill_fig_4a),
('4.b', 0.5, 0, merrill_fig_4b)):

Expand Down Expand Up @@ -127,25 +129,26 @@

for method in ('Black', 'Coombs', 'Borda', 'Approval', 'Hare', 'Runoff',
'Plurality'):
x, y = zip(*sorted(orig[method].items()))

Check failure on line 132 in examples/merrill_1984_fig_4a_4b.py

View workflow job for this annotation

GitHub Actions / lint

ruff (B905) 

examples/merrill_1984_fig_4a_4b.py:132:16: B905 `zip()` without an explicit `strict=` parameter
  help: Add explicit value for parameter `strict=`

Check failure on line 132 in examples/merrill_1984_fig_4a_4b.py

View workflow job for this annotation

GitHub Actions / lint

ruff (B905) 

examples/merrill_1984_fig_4a_4b.py:132:16: B905 `zip()` without an explicit `strict=` parameter
  help: Add explicit value for parameter `strict=`
plt.plot(x, y, ':', lw=0.8)

# Restart color cycle, so result colors match
plt.gca().set_prop_cycle(None)

table = []
table[fig] = []

# Calculate Social Utility Efficiency from summed utilities
x_uw, y_uw = zip(*sorted(utility_sums['SU max'].items()))

Check failure on line 141 in examples/merrill_1984_fig_4a_4b.py

View workflow job for this annotation

GitHub Actions / lint

ruff (B905) 

examples/merrill_1984_fig_4a_4b.py:141:18: B905 `zip()` without an explicit `strict=` parameter
  help: Add explicit value for parameter `strict=`

Check failure on line 141 in examples/merrill_1984_fig_4a_4b.py

View workflow job for this annotation

GitHub Actions / lint

ruff (B905) 

examples/merrill_1984_fig_4a_4b.py:141:18: B905 `zip()` without an explicit `strict=` parameter
  help: Add explicit value for parameter `strict=`
x_rw, y_rw = zip(*sorted(utility_sums['RW'].items()))
for method in ('Black', 'Coombs', 'Borda', 'Approval', 'Hare', 'Runoff',
'Plurality'):
x, y = zip(*sorted(utility_sums[method].items()))
SUE = (np.array(y) - y_rw) / (np.array(y_uw) - y_rw)
plt.plot(x, SUE*100, '-', label=method)
table.append([method, *SUE*100])
table[fig].append([method, *SUE*100])

print(tabulate(table, ["Method", *x], tablefmt="pipe", floatfmt='.1f'))
print(tabulate(table[fig], ["Method", *x], tablefmt="pipe",
floatfmt='.1f'))
print()

plt.plot([], [], 'k:', lw=0.8, label='Merrill') # Dummy plot for label
Expand All @@ -155,3 +158,32 @@
plt.ylim(ymin, 102)
plt.xlim(1.8, 7.2)
plt.show()

# Regression reference from the "Results with 500_000 elections" tables in the
# docstring, ordered by n_cands_list. These sims do not reproduce Merrill's
# published figures exactly (discrepancies up to ~7%, see issue #88), so this
# is only a regression guard against breaking the current close-enough output
# until that discrepancy is fixed.
reference_table = {
'4.a': {
'Black': (100.0, 97.2, 97.1, 97.3, 97.6, 97.8),
'Coombs': (100.0, 97.1, 96.8, 97.0, 97.2, 97.4),
'Borda': (100.0, 98.7, 98.2, 97.9, 97.7, 97.6),
'Approval': (100.0, 98.7, 97.3, 96.2, 95.6, 95.2),
'Hare': (100.0, 94.2, 92.6, 91.7, 91.0, 90.3),
'Runoff': (100.0, 94.2, 92.0, 90.4, 88.9, 87.4),
'Plurality': (100.0, 84.7, 77.1, 72.1, 68.1, 64.8),
},
'4.b': {
'Black': (100.0, 95.5, 95.2, 95.5, 95.8, 96.2),
'Coombs': (100.0, 94.9, 94.1, 94.0, 94.0, 94.1),
'Borda': (100.0, 97.9, 97.1, 96.6, 96.4, 96.3),
'Approval': (100.0, 98.6, 96.7, 95.6, 94.9, 94.5),
'Hare': (100.0, 70.2, 55.9, 46.7, 39.7, 34.6),
'Runoff': (100.0, 70.2, 51.7, 36.9, 24.3, 13.5),
'Plurality': (100.0, 50.1, 23.7, 4.3, -11.8, -25.1),
},
}

# Absolute tolerance (percentage points) for test_examples.py
tolerance = 5.0
44 changes: 40 additions & 4 deletions examples/merrill_1984_fig_4a_4b_updated.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,8 +78,10 @@
tiebreaker),
}

table = {}

for fig, disp, _ymin in (('4.a', 1.0, 55),
('4.b', 0.5, 0)):
('4.b', 0.5, 0)):

utility_sums = {key: Counter() for key in (ranked_methods.keys() |
rated_methods.keys() |
Expand Down Expand Up @@ -114,7 +116,7 @@
plt.title(f'Figure {fig}: Social Utility Efficiency under Spatial-Model '
f'Assumptions [Disp: {disp}]')

table = []
table[fig] = []

# Calculate Social Utility Efficiency from summed utilities
x_uw, y_uw = zip(*sorted(utility_sums['SU max'].items()))
Expand All @@ -124,9 +126,10 @@
x, y = zip(*sorted(utility_sums[method].items()))
SUE = (np.array(y) - y_rw) / (np.array(y_uw) - y_rw)
plt.plot(x, SUE*100, '-', label=method)
table.append([method, *SUE*100])
table[fig].append([method, *SUE*100])

print(tabulate(table, ["Method", *x], tablefmt="pipe", floatfmt='.1f'))
print(tabulate(table[fig], ["Method", *x], tablefmt="pipe",
floatfmt='.1f'))
print()

plt.legend()
Expand All @@ -135,3 +138,36 @@
plt.ylim(85, 100.5) # or ymin
plt.xlim(1.8, 7.2)
plt.show()

# Regression reference from the "Results with 100_000 elections" tables in the
# docstring, ordered by n_cands_list. These sims do not reproduce Merrill's
# published figures exactly (discrepancies up to ~7%, see issue #88), so this
# is only a regression guard against breaking the current close-enough output
# until that discrepancy is fixed.
reference_table = {
'4.a': {
'Score': (100.0, 100.0, 99.9, 99.9, 99.9, 99.8),
'STAR': (100.0, 97.5, 97.8, 98.3, 98.6, 98.8),
'Borda': (100.0, 98.8, 98.2, 97.9, 97.6, 97.6),
'Condorcet RCV': (100.0, 97.1, 97.0, 97.3, 97.6, 97.8),
'Coombs': (100.0, 97.0, 96.8, 97.0, 97.2, 97.4),
'Approval (opt.)': (100.0, 98.7, 97.3, 96.2, 95.5, 95.2),
'Hare RCV': (100.0, 94.2, 92.5, 91.6, 91.0, 90.4),
'Top-2 Runoff': (100.0, 94.2, 91.9, 90.4, 88.9, 87.5),
'Plurality': (100.0, 84.8, 77.4, 72.0, 68.2, 64.9),
},
'4.b': {
'Score': (100.0, 100.0, 99.9, 99.7, 99.5, 99.3),
'STAR': (100.0, 96.2, 96.7, 97.2, 97.6, 97.8),
'Borda': (100.0, 97.9, 97.1, 96.6, 96.4, 96.3),
'Condorcet RCV': (100.0, 95.5, 95.2, 95.5, 95.8, 96.2),
'Coombs': (100.0, 95.0, 94.2, 94.1, 94.0, 94.1),
'Approval (opt.)': (100.0, 98.6, 96.7, 95.5, 94.9, 94.6),
'Hare RCV': (100.0, 70.4, 55.9, 46.7, 39.5, 35.0),
'Top-2 Runoff': (100.0, 70.4, 51.8, 37.3, 23.9, 13.6),
'Plurality': (100.0, 50.5, 23.9, 4.7, -12.2, -24.7),
},
}

# Absolute tolerance (percentage points) for test_examples.py
tolerance = 5.0
11 changes: 8 additions & 3 deletions examples/merrill_1984_table_1_fig_1.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,8 +84,10 @@
elapsed_time = time.monotonic() - start_time
print('Elapsed:', time.strftime("%H:%M:%S", time.gmtime(elapsed_time)), '\n')

# Plot Merrill's results as dotted lines for comparison
merrill_table_1 = {
# Reference values from Merrill's published Table 1, used to plot his results
# as dotted lines for comparison. This script reproduces them within ~2 pp,
# so test_examples.py checks the computed table against these.
reference_table = {
'Plurality': {2: 100.0, 3: 79.1, 4: 69.4, 5: 62.1, 7: 52.0, 10: 42.6},
'Runoff': {2: 100.0, 3: 96.2, 4: 90.1, 5: 83.6, 7: 73.5, 10: 61.3},
'Hare': {2: 100.0, 3: 96.2, 4: 92.7, 5: 89.1, 7: 84.8, 10: 77.9},
Expand All @@ -97,11 +99,14 @@
'CW': {2: 100.0, 3: 91.6, 4: 83.4, 5: 75.8, 7: 64.3, 10: 52.5},
}

# Absolute tolerance (percentage points) for test_examples.py
tolerance = 4.0

plt.figure(f'Figure 1. {n_voters} voters, {n_elections} elections')
plt.title('Figure 1: Condorcet Efficiencies for a Random Society')
for method in ('Plurality', 'Runoff', 'Hare', 'Approval', 'Borda', 'Coombs',
'Black'):
x, y = zip(*sorted(merrill_table_1[method].items()))
x, y = zip(*sorted(reference_table[method].items()))
plt.plot(x, y, ':', lw=0.8)

# Restart color cycle, so result colors match
Expand Down
20 changes: 20 additions & 0 deletions examples/merrill_1984_table_2.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,3 +124,23 @@
table.append(['CW', *(y_cw / n_elections * 100)])

print(tabulate(table, header, tablefmt="pipe", floatfmt='.1f'))

# Regression reference from the "Typical result" table in the docstring,
# ordered by `conditions`. The published Merrill (1984) Table 2 is not
# reproduced (some values are off by up to ~5 pp; see issue #88), so this is
# only a regression guard against breaking the current close-enough output
# until that discrepancy is fixed.
reference_table = {
'Plurality': (57.5, 65.8, 62.2, 78.4, 21.7, 24.4, 27.2, 41.3),
'Runoff': (80.1, 87.3, 81.6, 93.6, 35.4, 42.2, 41.5, 61.5),
'Hare': (79.2, 86.7, 84.0, 95.4, 35.9, 46.8, 41.0, 69.9),
'Approval': (73.8, 77.8, 76.9, 85.4, 71.5, 76.4, 73.8, 82.7),
'Borda': (87.1, 89.3, 88.2, 92.3, 83.7, 86.3, 85.2, 89.4),
'Coombs': (97.8, 97.3, 97.9, 98.2, 93.5, 92.3, 93.8, 94.5),
'Black': (100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0),
'SU max': (82.9, 85.8, 85.3, 90.8, 78.1, 81.5, 80.8, 87.1),
'CW': (99.7, 99.7, 99.7, 99.6, 98.9, 98.6, 98.7, 98.5),
}

# Absolute tolerance (percentage points) for test_examples.py
tolerance = 3.5
11 changes: 8 additions & 3 deletions examples/merrill_1984_table_3_fig_3.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,8 +79,10 @@
elapsed_time = time.monotonic() - start_time
print('Elapsed:', time.strftime("%H:%M:%S", time.gmtime(elapsed_time)), '\n')

# Plot Merrill's results as dotted lines for comparison
merrill_table_1 = {
# Reference values from Merrill's published Table 3, used to plot his results
# as dotted lines for comparison. This script reproduces them within ~2 pp,
# so test_examples.py checks the computed table against these.
reference_table = {
'Plurality': {2: 100.0, 3: 83.0, 4: 75.0, 5: 69.2, 7: 62.8, 10: 53.3},
'Runoff': {2: 100.0, 3: 89.5, 4: 83.8, 5: 80.5, 7: 75.6, 10: 67.6},
'Hare': {2: 100.0, 3: 89.5, 4: 84.7, 5: 82.4, 7: 80.5, 10: 74.9},
Expand All @@ -90,11 +92,14 @@
'Black': {2: 100.0, 3: 93.1, 4: 91.9, 5: 92.0, 7: 93.1, 10: 94.3},
}

# Absolute tolerance (percentage points) for test_examples.py
tolerance = 3.5

plt.figure(f'Figure 3. {n_voters} voters, {n_elections} elections')
plt.title('Figure 3: Efficiencies for Social Utility for a Random Society')
for method in ('Plurality', 'Runoff', 'Hare', 'Approval', 'Borda', 'Coombs',
'Black'):
x, y = zip(*sorted(merrill_table_1[method].items()))
x, y = zip(*sorted(reference_table[method].items()))
plt.plot(x, y, ':', lw=0.8)

# Restart color cycle, so result colors match
Expand Down
18 changes: 18 additions & 0 deletions examples/merrill_1984_table_4.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,3 +121,21 @@
SUE = (y - y_rw)/(y_uw - y_rw)
table.append([method, *(SUE*100)])
print(tabulate(table, header, tablefmt="pipe", floatfmt='.1f'))

# Regression reference from the "Typical result" table in the docstring,
# ordered by `conditions`. The published Merrill (1984) Table 4 is not
# reproduced (some values are off by up to ~9 pp; see issue #88), so this is
# only a regression guard against breaking the current close-enough output
# until that discrepancy is fixed.
reference_table = {
'Plurality': (72.1, 79.1, 80.4, 92.4, 4.0, 6.3, 25.2, 52.9),
'Runoff': (90.5, 94.2, 92.0, 97.5, 36.6, 43.6, 53.3, 75.3),
'Hare': (91.7, 94.7, 94.3, 98.4, 46.4, 57.7, 58.7, 83.6),
'Approval': (96.2, 97.0, 96.8, 98.5, 95.6, 96.8, 95.8, 98.0),
'Borda': (97.8, 98.6, 98.3, 99.4, 96.6, 97.7, 97.4, 99.0),
'Coombs': (97.0, 97.5, 97.7, 98.7, 94.0, 94.3, 95.0, 96.7),
'Black': (97.3, 97.8, 98.0, 99.0, 95.5, 96.1, 96.5, 98.0),
}

# Absolute tolerance (percentage points) for test_examples.py
tolerance = 3.5
Loading
Loading