Perturb/Alter Ballots within Profile

[graceg571]

Closes #340
Opening to discuss API and architecture decisions before polishing. See "Design discussion" below.

Summary

Adds a module-level utility that applies a user-defined function to every ballot in a RankProfile.

new_profile = transform_ballots_profile(profile, my_transform_func)

The user's function receives either a full RankBallot or a plain list[Candidate] (the ranking, unwrapped), controlled by an explicit on parameter, and returns the same to produce the transformed profile.

API

def transform_ballots_profile(
    profile: RankProfile,
    transform_ballot_func: Callable[[RankBallot | list[Candidate]], RankBallot | list[Candidate]],
    on: Literal["candidates", "ballot"] = "ballot",
    group_ballots: bool = True,
) -> RankProfile

• on="ballot" (default): the function receives a RankBallot. Fully general — handles ties, weights, and anything else on the ballot.
• on="candidates": the function receives the ranking as a flat list[Candidate] — the friendliest form for common transforms like swapping two candidates. Requires untied rankings; raises ValueError (identifying the offending ballot) if the profile contains a tie.
• group_ballots=True: condenses identical ballot types before applying the function, so the function is called once per unique ballot type rather than once per ballot. On real CVR data, unique types are typically a small fraction of total ballots, so this is the main performance lever. Must be set to False for stochastic/sampled transforms — see "Performance and correctness notes."

Design choices worth flagging for review:

• Module-level function, not an instance method. Follows the cleaning module's organization and keeps PreferenceProfile's surface focused. The issue sketches pref_profile.transform(...) — happy to add a thin wrapper method on the class if that's preferred; all logic would stay in the module either way.
• No in_place parameter. The issue sketches in_place=False as the default; this draft omits the parameter entirely. True in-place mutation requires invalidating cached derived state on the profile and complicates every future mutating path. Can be added later if there's demand.
• No provenance tracking. Unlike CleanedProfile, the returned profile does not record its parent or which ballots changed. Transform and clean are related, so worth deciding whether they should converge.

Behavior details

• Validation is per-ballot and fail-fast. Tied cells in candidates mode and over-length returned rankings raise immediately with the ballot index and offending content in the message, before further work is done.
• Weights are preserved per ballot type and ride through the transform untouched. Only rankings are modified.
• Return normalization (candidates mode): returned candidates are re-wrapped into singleton frozensets and padded to max_ranking_length.
• Open: None/empty returns. Current draft treats them as an empty ranking; an alternative is drop the ballot.
• Open: regrouping after transform. A transform can map two previously distinct ballot types onto the same ranking, leaving the output with duplicate rows whose weights are not combined. Current draft does not regroup and documents this; the alternative is to condense the result by default.. Leaning toward regrouping by default.

Performance and correctness notes

Making an arbitrary user-defined transform fast is fundamentally limited: the function's logic is opaque to the library, so it must be called from a Python loop — no vectorization is possible. The design accepts this and tries to pull remaining levers:

1. Deduplication (group_ballots=True): the only way to make an opaque function cheaper is to call it fewer times. Grouping first means cost scales with unique ballot types, not total ballots.
2. Cheap materialization: candidates mode skips RankBallot construction and hands the function a plain list, iterating over the underlying array rather than pandas rows.

Stochastic transforms and grouping do not mix. With group_ballots=True, a probabilistic function (e.g. "swap A and B with probability 0.3") makes one draw per ballot type and applies it to the entire aggregated weight — 100 identical ballots all flip or none do, which is a different (wrong) distribution, not just a slower/faster trade-off. Sampled transforms must use group_ballots=False on a profile with unit weights. This is documented in the docstring; a deterministic flag that makes the contract explicit is an option if reviewers prefer.

Testing (to-do)

• Identity transform returns an equal profile (both modes)
• Swap transform produces the expected rankings (both modes)
• Tied profile + candidates mode raises with the ballot index
• Over-length returned ranking raises with the ballot index
• Weights preserved through transform; grouping combines weights as expected
• (pending resolution of the open questions: None-return behavior, regroup behavior)

---

Future work: predefined (non-opaque) transform functions

This section is the design discussion the draft is really for.

The general callable above is the escape hatch — it can express anything, at Python-loop speed. But most real transforms are drawn from a small vocabulary: swap two candidates, swap two candidates within some rank distance, truncate to top-k, remove a candidate, replace one ballot type with another. For these, we know the intent, which changes what's possible:

1. Wrapper functions (near-term, cheap). Predefined transforms shipped as ordinary functions that call transform_ballots_profile internally:

swap_candidates(profile, "A", "B", max_distance=2, strict_order=True)
swap_rank_positions(profile, 1, 2)
replace_ballot_type(profile, original_ballot= ["A", "B", "C"], new_ballot= ["A", "C", "B"])
remove_candidate_from_ballots(profile, "C")
truncate_ballots_at_position(profile, 5)
truncate_ballots_at_candidate(profile, "A")

Can utilize the internal df representation to perform operations with integers and constrain transform to ballots with matching operation.

[fetachino]

The design discussion lists the core behavior tests as TODO (identity, both modes, tied rankings, over-length returns, weight preservation, and regrouping). Before this API is finalized, could the PR add at least the deterministic identity/swap/error/weight cases? Those tests would also pin down the currently open None-return and regrouping semantics.

[graceg571]

The design discussion lists the core behavior tests as TODO (identity, both modes, tied rankings, over-length returns, weight preservation, and regrouping). Before this API is finalized, could the PR add at least the deterministic identity/swap/error/weight cases? Those tests would also pin down the currently open None-return and regrouping semantics.

Good call! I will be sure to add tests around these cases. We are still in the midst of discussing the design for the API, and I'm using this draft PR as notes and to-dos to keep track of where we end up. I'm implementing a new design this week and would love your feedback once it's up!