Fix scatterplot with a subset hue/size/style order (#3601)

[Leonard013]

Fixes #3601

Problem

scatterplot (and relplot(kind="scatter")) mishandle a hue_order / size_order / style_order that is a strict subset of the values present in the data:

• style_order subset → hard crash (KeyError).
• hue_order subset → silent wrong output: rows whose hue value is omitted from the order are still drawn, but as fully transparent points.

lineplot, relplot(kind="line"), and the objects interface all handle a subset order correctly by simply not drawing the omitted rows. Only the scatter-type path is broken.

Minimal reproduction:

import seaborn as sns
df = sns.load_dataset("diamonds").dropna()

# 1) style_order subset -> KeyError
sns.scatterplot(data=df, x="price", y="carat", style="clarity",
                style_order=["I1", "IF"])
# KeyError: 'SI2'

# 2) hue_order subset -> rows with an unlisted hue are drawn transparent
sns.scatterplot(data=df, x="price", y="carat", hue="clarity",
                hue_order=["I1", "IF"])

Root cause

_ScatterPlotter.plot (seaborn/relational.py) draws every row of self.comp_data.dropna() and then maps each row's semantic value to a visual attribute:

data = self.comp_data.dropna()
...
points.set_facecolors(self._hue_map(data["hue"]))     # hue
...
p = [self._style_map(val, "path") for val in data["style"]]   # style

When the order is a subset, the mapping lookup tables only contain the ordered levels, so values outside the order are unmapped:

• StyleMapping._lookup_single (seaborn/_base.py:587) does self.lookup_table[key][attr] and raises KeyError for an unmapped style value.
• HueMapping._lookup_single (seaborn/_base.py:172-183) explicitly returns (0, 0, 0, 0) (transparent) for an unmapped categorical hue value — its own comment notes this only happens "in scatterplot with hue_order, because scatterplot does not consider hue a grouping variable".

By contrast, _LinePlotter.plot iterates self.iter_data(("hue", "size", "style")), which only yields the mapped levels, so line plots naturally drop the unmapped rows.

Fix

Reduce the scatter data to the rows whose table-mapped hue/size/style value is actually mapped, right after dropna() — matching the maintainer's endorsed direction (reduce the dataframe to the rows with relevant hue/size/style values, in the scatter plotting method) and making scatter behave like lineplot / the objects interface:

data = self.comp_data.dropna()

if "hue" in self.variables and self._hue_map.map_type != "numeric":
    data = data[data["hue"].isin(self._hue_map.levels)]
if "size" in self.variables and self._size_map.map_type != "numeric":
    data = data[data["size"].isin(self._size_map.levels)]
if "style" in self.variables and self._style_map.map_type != "numeric":
    data = data[data["style"].isin(self._style_map.levels)]

Notes on the design:

• The guard is map_type != "numeric", i.e. it covers the table-based mappings — both categorical and datetime. A datetime hue/size is built through the same categorical_mapping lookup table (seaborn/_base.py), so a subset order exhibits the identical bug; keying on "not numeric" rather than "is categorical" fixes datetime too. Numeric hue/size mappings are intentionally left untouched: they interpolate out-of-table values through the norm/colormap (e.g. a numeric dict palette that does not cover every value), which lineplot also preserves. For a numeric mapping the levels are the full set of unique data values anyway, so filtering would at best be a no-op and at worst break intended interpolation.
• Each mapping attribute (self._hue_map etc.) is accessed only inside if "<var>" in self.variables, exactly as the existing code below it already does, so instances built without the corresponding map_*() call are unaffected.

Tests (tests/test_relational.py, TestScatterPlotter)

• test_hue_order (updated): previously asserted that omitted-hue rows were drawn as transparent points (i.e. it encoded the bug). It now asserts those rows are dropped, that no drawn point is transparent, and that the legend shows only the ordered levels.
• test_style_order (new): a style_order subset no longer raises; the omitted rows are dropped and the legend shows only the ordered levels.
• test_hue_order_datetime (new): a datetime hue with a subset hue_order drops the omitted rows instead of drawing them transparent — covering the datetime map type the != "numeric" guard now handles. It fails on the narrower == "categorical" guard and passes with != "numeric".

All fail on main and pass with the fix.

Verification

Environment: seaborn 0.14.0.dev0 (this branch), matplotlib 3.11.0, pandas 3.0.3, numpy 2.5.1, Python 3.12.

Regression tests — before the source fix (tests present, fix absent):

$ python -m pytest tests/test_relational.py::TestScatterPlotter::test_hue_order \
                   tests/test_relational.py::TestScatterPlotter::test_style_order -v
test_hue_order  FAILED   # AssertionError: 100 == 72 (all rows still drawn, omitted rows transparent)
test_style_order FAILED  # KeyError: 'c'  (seaborn/_base.py:587, self.lookup_table[key][attr])
2 failed

Regression tests — after the source fix:

$ python -m pytest tests/test_relational.py::TestScatterPlotter::test_hue_order \
                   tests/test_relational.py::TestScatterPlotter::test_style_order -v
2 passed

Behavioral before/after on a 30-row frame with levels a/b/c and a subset order ['a','b'] (10 'c' rows):

Case	Before	After	lineplot (reference)
style_order=['a','b']	KeyError: 'c'	20 points, no crash	20 points
hue_order=['a','b']	30 points, 'c' transparent	20 points, none transparent	20 points, legend ['a','b']

Common (non-subset) case is unchanged — 30/30 points drawn:

hue, no order            -> 30 points
style_order = all levels -> 30 points
numeric hue              -> 30 points   (the != "numeric" guard leaves numeric mapping untouched)
datetime hue, subset order -> omitted rows dropped, none transparent (matches categorical)

Full modules:

$ python -m pytest tests/test_relational.py -q
122 passed

$ python -m pytest tests/test_axisgrid.py -q    # pairplot / relplot exercise scatter
123 passed

$ ruff check seaborn/relational.py tests/test_relational.py
All checks passed!

Compatibility

The only observable change is for the previously-broken subset case, which either crashed (style_order) or produced wrong output (hue_order/transparent points). Passing no order, or an order equal to the full set of levels, is unchanged, as are numeric hue/size mappings. The single updated existing test (test_hue_order) had asserted the buggy transparent-point behavior. This brings scatter into line with lineplot and the objects interface.