Skip to content

Add functionality to create centroids and centroid connector to model network - #478

Open
lmz wants to merge 256 commits into
network-wrangler:mainfrom
BayAreaMetro:centroids
Open

Add functionality to create centroids and centroid connector to model network#478
lmz wants to merge 256 commits into
network-wrangler:mainfrom
BayAreaMetro:centroids

Conversation

@lmz

@lmz lmz commented Jun 19, 2026

Copy link
Copy Markdown
Member

This PR adds end-to-end support for creating centroids and centroid connectors in a model network, including the core APIs, geometry/link generation logic, and tests/documentation updates needed to use and validate the workflow.

What this PR does

This PR introduces functionality to build model-zone access structures by generating:

  • Centroid nodes (zone representation in the network)
  • Centroid connector links between centroids and nearby network nodes
  • The feature is designed to support common modeling workflows where zones/TAZs need to be integrated into the routable network with generated access/egress links.

Key changes

  • Added APIs/utilities to create centroid nodes from input zone geometry/data.
  • Added logic to create centroid connector links from each centroid to eligible nearby network nodes.
  • Added validation and guardrails around connector creation inputs and outputs.
  • Added/updated tests covering centroid and connector creation behavior.
  • Updated docs/examples to demonstrate expected usage and outputs.

Why this change is needed

Many demand and assignment workflows require zone-level connectors to represent how trips enter/exit the physical network. This PR provides a native, repeatable way to generate those structures directly in network_wrangler, reducing manual preprocessing and ensuring consistent connector generation behavior.

Implementation notes

Centroid generation and connector generation are implemented as explicit network-building operations so they can be run as part of a reproducible wrangling pipeline.
Connector creation is based on proximity/eligibility rules to attach centroids to nearby nodes.
The new functionality is intended to be configurable for model-specific connector policies.

Testing

This PR includes tests for:

  • Correct centroid creation from valid inputs
  • Connector generation for expected nearest/eligible nodes
  • Error/edge handling for invalid or missing inputs
  • Basic integrity checks on produced nodes/links
  • Backward compatibility

Existing workflows are unaffected unless the new centroid/connector functionality is explicitly invoked. No breaking changes are intended for users not using these new APIs.

Limitations / follow-ups

Connector selection policies may still need tuning for specific model conventions (e.g., max connectors per zone, facility/type filtering, directional behavior).
Future enhancements can expand configurability and performance for very large zone sets.


lmz added 30 commits July 1, 2025 17:02
Moving API detail to API_*.md
lmz and others added 19 commits February 20, 2026 10:38
- Fix TypeError 'Series object is not callable' in match_bus_stops_to_roadway_nodes
  (missing & operator, extra parens treating boolean Series as a function call)
- Fix unhashable type error in add_additional_data_to_stops by converting
  Categorical route_type column to int before groupby aggregation
- Relax TripsTable.shape_id to nullable=True (shape_id is optional per GTFS spec)
- Add Union import to io_table.py

Not centroid-branch related; applies to any use of transit network creation.
@lmz
lmz marked this pull request as draft June 19, 2026 15:26
lmz added 5 commits June 23, 2026 17:05
And remove some documentation from networks.md since it's already in api_roadway.md
…tup compatibility

- io_table.py: call _prepare_df_for_json() before writing parquet and json
  files (previously only geojson); extend _restore_*() calls to parquet on
  read. Fixes ArrowInvalid errors when ScopedLinkValueItem Pydantic objects
  are present in sc_* columns.

- links/create.py: call order_fields_from_data_model() after
  validate_df_to_model() in data_to_links_df(). When write_links() drops
  geometry columns (include_geometry=False), ML_geometry is absent from the
  written file and gets appended at the end of the DataFrame on read-back.
  The reorder step restores the schema-defined column order, fixing the
  test_roadway_geojson_read_write_read column-ordering assertion.

- tests/test_setup.py: replace hardcoded 'python3' and Unix-style venv
  paths (bin/pip, bin/python) with sys.executable and a platform-aware
  helper _venv_exe() that uses Scripts/ on Windows. Fixes CalledProcessError
  on Windows where 'python3' does not exist.
@lmz
lmz marked this pull request as ready for review July 1, 2026 23:59
@lmz
lmz requested a review from e-lo July 1, 2026 23:59
@lmz

lmz commented Jul 13, 2026

Copy link
Copy Markdown
Member Author

cc @i-am-sijia

@e-lo e-lo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some things to address :-)

Comment thread docs/design.md

**4.[`ProjectCard`](https://network-wrangler.github.io/projectcard/main/api/#projectcard.projectcard.ProjectCard)** objects store information (including metadata) about changes to the network. Network Wrangler uses the [`projectcard`](https://network-wrangler.github.io/projectcard) package to read project cards from .yaml-like files and validate them.

## Creating Networks

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

# pandera >= 0.26 rejects a plain object column of None as not geometry dtype,
# even with nullable=True and coerce=True. Pre-casting to GeoSeries avoids this.
# Values loaded from GeoJSON files may be dicts or JSON strings; convert to Shapely first.
if "ML_geometry" not in links_df.columns:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Do we want ML_geometry to be mandatory?
  2. This coercing should be its own function (e.g. _fill_missing_link_geometries_from_nodes())
  3. All the apply() is very SLOW...especially when we are using it just for counting.

Suggest
(a) just doing the counting to make sure things work for a test on a tiny network (not here).
(b) i think you can do a select just on ones that don't work (if any) and then apply? If its null should be skipping entirely.

nodes_df["X"] = nodes_df.geometry.x
nodes_df["Y"] = nodes_df.geometry.y

# Handle duplicate model_node_id values by averaging coordinates

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. this should be an option rather than default - in many (most?) cases this indicates an error which shouldn't be obfuscated.
  2. should encapsulate in a function "merge nodes"
  3. should probably error/warn if distances between are really big (i.e. almost definitely indicates an error)

@@ -0,0 +1,473 @@
"""Functions to create centroid connectors."""

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add usage

@@ -0,0 +1,473 @@
"""Functions to create centroid connectors."""

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be in a sub-module similar to /nodes /links?

mode_node_df.reset_index(drop=True, inplace=True)
mode_node_df["connector_num"] = 0

def calculate_min_angle_separation(candidate_angle, selected_angles):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This helper can be removed...see below

f"Before choosing centroid connector nodes, mode_node_df:\n{mode_node_df}"
)

# Process each zone and select connectors

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was pretty sure there was a way to do this without the extra applies and this is what Claude came up with which I think is ok. Its basically just keeping a running tab of the min angle of already selected connectors for each zone and updating as they are selected. I only 68% understand this (but didn't burn a ton of time on it either) but it should be faster.

fit_col = f"{mode}_centroid_fit"
# rows are already sorted by [zone_id, fit_col, distance_from_centroid], so within a
# zone the first row is the best fit + closest.
for zone_num in mode_node_df[zone_id].unique():
    idx = mode_node_df.index[mode_node_df[zone_id] == zone_num]
    if len(idx) == 0:
        continue

    angles = mode_node_df.loc[idx, "centroid_angle"].to_numpy()
    fit = mode_node_df.loc[idx, fit_col].to_numpy()
    conn = np.zeros(len(idx), dtype=int)

    def circ_sep(a):  # wraparound angular distance of every angle from scalar a
        d = np.abs(angles - a)
        return np.minimum(d, 360 - d)

    conn[0] = 1                       # connector 1: best fit + closest
    min_sep = circ_sep(angles[0])     # running min separation to the selected set

    for k in range(2, num_centroid_connectors + 1):
        unsel = conn == 0
        if not unsel.any():
            break
        best_fit = fit[unsel].min()               # best remaining fit level
        eligible = unsel & (fit == best_fit)
        pick = np.argmax(np.where(eligible, min_sep, -np.inf))
        conn[pick] = k
        min_sep = np.minimum(min_sep, circ_sep(angles[pick]))   # incremental, O(n)

    mode_node_df.loc[idx, "connector_num"] = conn

raise ManagedLaneAccessEgressError(msg)

# access link should go from A_GP to A_ML
# Use larger offsets (1M and 2M) to avoid collisions when GP link IDs differ by small amounts

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should probably be a parameter and not hard coded...right? there are different limits in various softwares.


return is_connected

def write(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if do this here - should do it to transit!?

return bearing_deg


def bearing_to_cardinal_direction(bearing: float, cardinal_only: bool = False) -> Optional[str]:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do vector operations applied to series...

def bearings_to_cardinal_directions(bearings: pd.Series, cardinal_only: bool = False) -> pd.Series:
    """Vectorized bearing (deg) -> cardinal/intercardinal label. NaN bearings map to None."""

    _CARDINAL_LABELS = np.array(
    [d.value for d in (CardinalDirection.N, CardinalDirection.E, CardinalDirection.S, CardinalDirection.W)]
) #also do intercardinal if want that....

    labels, sector_deg = (
        (_CARDINAL_LABELS, 90.0) if cardinal_only else (_INTERCARDINAL_LABELS, 45.0)
    )
    # Shift by half a sector so North straddles 0°, then floor-divide into sector indices.
    sector = ((bearings + sector_deg / 2) % 360) // sector_deg

    directions = pd.Series(pd.NA, index=bearings.index, dtype="object")
    valid = sector.notna()
    directions[valid] = labels[sector[valid].astype(int)]
    return directions

@lmz

lmz commented Jul 28, 2026

Copy link
Copy Markdown
Member Author

Thank you @e-lo ! I'll go through these ASAP, hopefully by end of week!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants