Add functionality to create centroids and centroid connector to model network - #478
Add functionality to create centroids and centroid connector to model network#478lmz wants to merge 256 commits into
Conversation
Moving API detail to API_*.md
- Fix TypeError 'Series object is not callable' in match_bus_stops_to_roadway_nodes (missing & operator, extra parens treating boolean Series as a function call) - Fix unhashable type error in add_additional_data_to_stops by converting Categorical route_type column to int before groupby aggregation - Relax TripsTable.shape_id to nullable=True (shape_id is optional per GTFS spec) - Add Union import to io_table.py Not centroid-branch related; applies to any use of transit network creation.
And remove some documentation from networks.md since it's already in api_roadway.md
…tup compatibility - io_table.py: call _prepare_df_for_json() before writing parquet and json files (previously only geojson); extend _restore_*() calls to parquet on read. Fixes ArrowInvalid errors when ScopedLinkValueItem Pydantic objects are present in sc_* columns. - links/create.py: call order_fields_from_data_model() after validate_df_to_model() in data_to_links_df(). When write_links() drops geometry columns (include_geometry=False), ML_geometry is absent from the written file and gets appended at the end of the DataFrame on read-back. The reorder step restores the schema-defined column order, fixing the test_roadway_geojson_read_write_read column-ordering assertion. - tests/test_setup.py: replace hardcoded 'python3' and Unix-style venv paths (bin/pip, bin/python) with sys.executable and a platform-aware helper _venv_exe() that uses Scripts/ on Windows. Fixes CalledProcessError on Windows where 'python3' does not exist.
|
cc @i-am-sijia |
e-lo
left a comment
There was a problem hiding this comment.
Some things to address :-)
|
|
||
| **4.[`ProjectCard`](https://network-wrangler.github.io/projectcard/main/api/#projectcard.projectcard.ProjectCard)** objects store information (including metadata) about changes to the network. Network Wrangler uses the [`projectcard`](https://network-wrangler.github.io/projectcard) package to read project cards from .yaml-like files and validate them. | ||
|
|
||
| ## Creating Networks |
There was a problem hiding this comment.
most of this belongs in: https://github.com/network-wrangler/network_wrangler/blob/main/docs/how_to.md
| # pandera >= 0.26 rejects a plain object column of None as not geometry dtype, | ||
| # even with nullable=True and coerce=True. Pre-casting to GeoSeries avoids this. | ||
| # Values loaded from GeoJSON files may be dicts or JSON strings; convert to Shapely first. | ||
| if "ML_geometry" not in links_df.columns: |
There was a problem hiding this comment.
- Do we want ML_geometry to be mandatory?
- This coercing should be its own function (e.g.
_fill_missing_link_geometries_from_nodes()) - All the
apply()is very SLOW...especially when we are using it just for counting.
Suggest
(a) just doing the counting to make sure things work for a test on a tiny network (not here).
(b) i think you can do a select just on ones that don't work (if any) and then apply? If its null should be skipping entirely.
| nodes_df["X"] = nodes_df.geometry.x | ||
| nodes_df["Y"] = nodes_df.geometry.y | ||
|
|
||
| # Handle duplicate model_node_id values by averaging coordinates |
There was a problem hiding this comment.
- this should be an option rather than default - in many (most?) cases this indicates an error which shouldn't be obfuscated.
- should encapsulate in a function "merge nodes"
- should probably error/warn if distances between are really big (i.e. almost definitely indicates an error)
| @@ -0,0 +1,473 @@ | |||
| """Functions to create centroid connectors.""" | |||
|
|
|||
| @@ -0,0 +1,473 @@ | |||
| """Functions to create centroid connectors.""" | |||
There was a problem hiding this comment.
Should this be in a sub-module similar to /nodes /links?
| mode_node_df.reset_index(drop=True, inplace=True) | ||
| mode_node_df["connector_num"] = 0 | ||
|
|
||
| def calculate_min_angle_separation(candidate_angle, selected_angles): |
There was a problem hiding this comment.
This helper can be removed...see below
| f"Before choosing centroid connector nodes, mode_node_df:\n{mode_node_df}" | ||
| ) | ||
|
|
||
| # Process each zone and select connectors |
There was a problem hiding this comment.
I was pretty sure there was a way to do this without the extra applies and this is what Claude came up with which I think is ok. Its basically just keeping a running tab of the min angle of already selected connectors for each zone and updating as they are selected. I only 68% understand this (but didn't burn a ton of time on it either) but it should be faster.
fit_col = f"{mode}_centroid_fit"
# rows are already sorted by [zone_id, fit_col, distance_from_centroid], so within a
# zone the first row is the best fit + closest.
for zone_num in mode_node_df[zone_id].unique():
idx = mode_node_df.index[mode_node_df[zone_id] == zone_num]
if len(idx) == 0:
continue
angles = mode_node_df.loc[idx, "centroid_angle"].to_numpy()
fit = mode_node_df.loc[idx, fit_col].to_numpy()
conn = np.zeros(len(idx), dtype=int)
def circ_sep(a): # wraparound angular distance of every angle from scalar a
d = np.abs(angles - a)
return np.minimum(d, 360 - d)
conn[0] = 1 # connector 1: best fit + closest
min_sep = circ_sep(angles[0]) # running min separation to the selected set
for k in range(2, num_centroid_connectors + 1):
unsel = conn == 0
if not unsel.any():
break
best_fit = fit[unsel].min() # best remaining fit level
eligible = unsel & (fit == best_fit)
pick = np.argmax(np.where(eligible, min_sep, -np.inf))
conn[pick] = k
min_sep = np.minimum(min_sep, circ_sep(angles[pick])) # incremental, O(n)
mode_node_df.loc[idx, "connector_num"] = conn| raise ManagedLaneAccessEgressError(msg) | ||
|
|
||
| # access link should go from A_GP to A_ML | ||
| # Use larger offsets (1M and 2M) to avoid collisions when GP link IDs differ by small amounts |
There was a problem hiding this comment.
this should probably be a parameter and not hard coded...right? there are different limits in various softwares.
|
|
||
| return is_connected | ||
|
|
||
| def write( |
There was a problem hiding this comment.
if do this here - should do it to transit!?
| return bearing_deg | ||
|
|
||
|
|
||
| def bearing_to_cardinal_direction(bearing: float, cardinal_only: bool = False) -> Optional[str]: |
There was a problem hiding this comment.
do vector operations applied to series...
def bearings_to_cardinal_directions(bearings: pd.Series, cardinal_only: bool = False) -> pd.Series:
"""Vectorized bearing (deg) -> cardinal/intercardinal label. NaN bearings map to None."""
_CARDINAL_LABELS = np.array(
[d.value for d in (CardinalDirection.N, CardinalDirection.E, CardinalDirection.S, CardinalDirection.W)]
) #also do intercardinal if want that....
labels, sector_deg = (
(_CARDINAL_LABELS, 90.0) if cardinal_only else (_INTERCARDINAL_LABELS, 45.0)
)
# Shift by half a sector so North straddles 0°, then floor-divide into sector indices.
sector = ((bearings + sector_deg / 2) % 360) // sector_deg
directions = pd.Series(pd.NA, index=bearings.index, dtype="object")
valid = sector.notna()
directions[valid] = labels[sector[valid].astype(int)]
return directions|
Thank you @e-lo ! I'll go through these ASAP, hopefully by end of week! |
This PR adds end-to-end support for creating centroids and centroid connectors in a model network, including the core APIs, geometry/link generation logic, and tests/documentation updates needed to use and validate the workflow.
What this PR does
This PR introduces functionality to build model-zone access structures by generating:
Key changes
Why this change is needed
Many demand and assignment workflows require zone-level connectors to represent how trips enter/exit the physical network. This PR provides a native, repeatable way to generate those structures directly in network_wrangler, reducing manual preprocessing and ensuring consistent connector generation behavior.
Implementation notes
Centroid generation and connector generation are implemented as explicit network-building operations so they can be run as part of a reproducible wrangling pipeline.
Connector creation is based on proximity/eligibility rules to attach centroids to nearby nodes.
The new functionality is intended to be configurable for model-specific connector policies.
Testing
This PR includes tests for:
Existing workflows are unaffected unless the new centroid/connector functionality is explicitly invoked. No breaking changes are intended for users not using these new APIs.
Limitations / follow-ups
Connector selection policies may still need tuning for specific model conventions (e.g., max connectors per zone, facility/type filtering, directional behavior).
Future enhancements can expand configurability and performance for very large zone sets.