Add Filecoin DX toolkit, examples, and architecture docs

[GautamBytes]

Summary

This PR adds a dedicated Filecoin developer-experience layer to py-libp2p, including pinned protocol/topic/bootstrap constants, practical bootstrap/runtime helpers, reusable pubsub presets, CLI tooling, runnable examples, architecture-positioning docs, and traceability artifacts.

##Closes: #1279

Why

Filecoin developers need a low-friction Python path for diagnostics, tooling, analytics, testnet work, and research workflows without implementing full node behavior. This PR provides that DX layer while documenting clear boundaries for where Lotus/Forest remain required.

What changed

• Added new libp2p.filecoin package:
  • constants and protocol/topic helpers
  • network presets for mainnet/calibnet
  • canonical + runtime bootstrap helpers
  • pubsub/gossipsub preset builders
  • CLI module and python -m libp2p.filecoin support
• Added console scripts:
  • filecoin-dx
  • filecoin-connect-demo
  • filecoin-ping-identify-demo
  • filecoin-pubsub-demo
• Added Filecoin examples:
  • connect demo
  • ping/identify demo
  • read-only pubsub observer demo
• Added architecture positioning doc:
  • where py-libp2p fits today
  • where full Lotus/Forest implementations are still required
  • suggested use-cases and explicit anti-goals
• Added dependency-tree and parity artifacts:
  • Lotus/Forest libp2p lineage mapping
  • parity matrix for implemented vs deferred concerns
• Added Filecoin-focused tests:
  • constants, networks, bootstrap, presets, CLI
  • examples/doc guard tests
  • dependency-tree artifact checks
• Stability fix in pubsub control-path:
  • malformed/non-UTF8 message IDs are skipped safely instead of crashing observer flows

Validation

Ran locally:

• python -m tox run -e py312-lint (includes pre-commit, ruff, mypy, pyrefly) -> pass
• python -m pytest -p no:rerunfailures tests/core/filecoin -q -> 37 passed
• sphinx-build -b html docs docs/_build/html -> succeeded
• Filecoin CLI smoke checks (topics/bootstrap/preset) -> expected outputs
• Demo smoke:
  • connect (mainnet/calibnet) -> successful
  • ping/identify -> successful
  • pubsub observer -> subscribes and runs in read-only mode

Notes

• Scope is DX/tooling only. No full chain-sync, messagepool, or consensus behavior is introduced.
• The DHT string behavior follows the pinned implementation baseline used for this branch and is documented explicitly in architecture/parity docs.

[yashksaini-coder]

PR #1260 Review — Filecoin DX Toolkit

The changes and module architecture is clean, tests are solid, and the parity matrix + dependency tree documentation is really impressive work.

---

Few Issues I found during review

1. Duck-typing in handle_ihave is unnecessary and fragile

• Location: gossipsub.py:1328-1336
• Issue: LastSeenCache always has .has(). The getattr fallback that reaches into .cache couples production code to internal state just to accommodate test mocks.
• Action: Call seen_messages.has(msg_id_bytes) directly. Fix the mocks if needed, not the production path.

2. Dead code: safe_parse_message_id and parse_message_id_safe

• Location: utils.py:57-97
• Issue: Neither function is called anywhere. safe_parse_message_id pulls in import ast and uses ast.literal_eval to parse message IDs, which is odd for a networking protocol. The old parse_message_id_safe was removed from its only import site in this same PR.
• Action: Drop both functions and the ast import.

3. Redundant assert after raise

• Location: gossipsub.py:1303-1307
• Issue: The if self.pubsub is None: raise already narrows the type. The assert self.pubsub is not None on the next line is dead code.
• Action: Remove it.

4. Malformed message IDs downgraded from WARNING to DEBUG

• Location: gossipsub.py:1320, 1370
• Issue: Peers sending bad message IDs is worth knowing about in production. Keeping this at WARNING (or at least INFO) gives operators visibility into misbehaving or malicious peers.
• Action: Revert log level to WARNING or INFO.

5. Blocking DNS resolution in an async codebase

• Location: bootstrap.py:101-107
• Issue: socket.getaddrinfo blocks the Trio event loop. Acceptable at startup, but worth a comment or a future switch to trio.socket.getaddrinfo.
• Action: Add comment or create follow-up ticket.

6. filecoin_message_id typed as Any

• Location: constants.py:34
• Issue: It accesses msg.data and validates bytes. A Protocol type or a short docstring describing the expected interface would be better than Any.
• Action: Add proper type annotation or documentation.

7. Mainnet genesis name "testnetnet" is confusing without context

• Location: networks.py:37
• Issue: Correct per upstream, but a one-line comment explaining the historical naming would save future readers a trip to the Lotus source.
• Action: Add explanatory comment.

---

@GautamBytes Please have a look at these, and reply back on Discord once you have cleared them.

[GautamBytes]

sure , will go through this and do the suggested changes soon!

[yashksaini-coder]

sure , will go through this and do the suggested changes soon!

Dm over discord once, couldn't find the ID

[GautamBytes]

sure , will go through this and do the suggested changes soon!

Dm over discord once, couldn't find the ID

done , just sent the dm!

[acul71]

Hello @GautamBytes, thanks for this PR
Primary blockers are: (1) malformed IWANT handling still raises; (2) merge conflict with origin/main; (3) issue/newsfragment linkage mismatch per policy.

Full review here:

PR #1260 Review — Filecoin DX toolkit, examples, and architecture docs

1. Summary of Changes

• Adds a new libp2p.filecoin DX layer (constants, network presets, bootstrap/runtime helpers, pubsub presets, CLI entrypoint/module) plus Filecoin examples and docs.
• Updates pubsub internals for control-message ID handling (libp2p/pubsub/gossipsub.py, libp2p/pubsub/mcache.py, libp2p/pubsub/utils.py) and expands tests around these paths.
• Introduces artifacts/docs for architecture positioning and dependency lineage (artifacts/filecoin/libp2p_dependency_tree.v1.json, docs/filecoin/*, docs/filecoin_architecture_positioning.rst).
• Adds a newsfragment: newsfragments/33.feature.rst.
• No explicit deprecations in this PR text; functional scope is additive DX + pubsub robustness behavior changes.

---

2. Branch Sync Status and Merge Conflicts

Branch Sync Status

• Status: ⚠️ Diverged from origin/main.
• Details: branch_sync_status.txt contains 1 7 (branch is 1 commit behind and 7 commits ahead of origin/main).

Merge Conflict Analysis

• Conflicts Detected: ❌ Conflicts detected.
• Conflicting Files: tests/core/pubsub/test_gossipsub.py
• Conflict Summary:
  • File: tests/core/pubsub/test_gossipsub.py
  • Conflict Type: Both branches modified overlapping test region.
  • Location: Conflict markers reported near lines 829, 835, 846 in merge_conflict_check.log.
  • Nature: Likely overlapping pubsub message-ID behavior tests from both branches.
  • Severity: Medium.
  • Resolution Complexity: Moderate (test semantics must be preserved while reconciling both edits).
• Impact Assessment: PR is not mergeable cleanly until rebased/merged and conflicts resolved.
• Resolution Recommendations:
  • Rebase onto latest origin/main (preferred), then re-run pubsub test suite.
  • Resolve tests/core/pubsub/test_gossipsub.py conflicts preserving both intended behavior checks.

---

3. Strengths

• Strong breadth of implementation: Filecoin package, docs, CLI, and runnable examples are delivered cohesively.
• Good test coverage expansion across new Filecoin modules and updated pubsub control-ID behavior.
• Prior reviewer feedback appears largely addressed:
  • filecoin_message_id now has clearer typing/doc expectations.
  • Added context comment for mainnet "testnetnet" historical name.
  • Added startup-only comment around blocking DNS lookup.
  • Removed unnecessary duck-typing fallback in IHAVE seen checks.
• Validation runs completed successfully in this environment:
  • make lint: pass
  • make typecheck: pass
  • make test: 2486 passed, 16 skipped
  • make linux-docs: build succeeded; doctests succeeded

---

4. Issues Found

Critical

• File: libp2p/pubsub/gossipsub.py

• Line(s): 1349–1354

• Issue: handle_iwant() catches malformed control IDs, logs warning, then re-raises ValueError. This allows malformed peer input to bubble out of control-message handling (handle_rpc), risking task failure and connection instability instead of safely skipping bad IDs.

• Suggestion: Do not re-raise for malformed remote input. Log at warning/info and continue to next message ID; add regression test that malformed IWANT IDs are ignored without exception.

• File: tests/core/pubsub/test_gossipsub.py

• Line(s): 866–903

• Issue: Tests currently enforce raising ValueError for malformed IWANT IDs, which codifies crash-prone behavior for untrusted network data.

• Suggestion: Update expected behavior to "ignore malformed IDs and keep processing", then assert no write and no exception.

Major

• File: PR metadata / process compliance

• Line(s): N/A

• Issue: gh pr view 1260 --json closingIssuesReferences returns no linked issues. Under the provided review policy, every PR must explicitly refer to an issue from this repository.

• Suggestion: Add explicit issue linkage in PR description (Fixes #... / Implements #...) using the correct repo issue.

• File: newsfragments/33.feature.rst

• Line(s): 1–3

• Issue: Newsfragment uses issue number 33, but in libp2p/py-libp2p issue #33 is "Add logo" and unrelated to this Filecoin DX change.

• Suggestion: Rename/recreate newsfragment with the correct issue number for this work (same issue should be explicitly referenced in PR body).

Minor

• No additional minor issues identified beyond the above blockers.

---

5. Security Review

• Risk: Malformed IWANT control IDs can trigger exceptions in control-path processing.
• Impact: Medium (remote peer can induce avoidable control-path failures/disruptions; potential DoS vector against pubsub stability).
• Mitigation: Treat malformed control IDs as invalid peer input: log and skip; never raise for parse failures on untrusted network payloads.

No other clear key-management/crypto/input-validation regressions observed in the new Filecoin modules.

---

6. Documentation and Examples

• Documentation additions are strong and comprehensive (architecture positioning, parity/dependency docs, filecoin examples index integration).
• Example coverage is good (connect, ping/identify, pubsub).
• Docs build succeeded with no blocking warnings/errors in this run.
• No missing public-API docs spotted for the new Filecoin package at high level.

---

7. Newsfragment Requirement

• ✅ Newsfragment file is present: newsfragments/33.feature.rst.
• ❌ BLOCKER: Issue association appears incorrect for this repository (#33 is unrelated legacy issue) and PR body has no explicit issue linkage.
• Action Required for approval:
  1. Link the PR explicitly to the correct repository issue.
  2. Rename/create the newsfragment as <CORRECT_ISSUE_NUMBER>.<TYPE>.rst with user-facing text.

---

8. Tests and Validation

Artifacts saved under downloads/AI-PR-REVIEWS/1260/:

• lint_output.log: all checks passed.
• typecheck_output.log: mypy + pyrefly pre-commit hooks passed.
• test_output.log: full suite passed (2486 passed, 16 skipped in 98.99s), no failures/errors.
• docs_output.log: make linux-docs succeeded; doctests report 109 tests, 0 failures.

Observation: current tests pass because they encode the raise-on-malformed-IWANT behavior; this is exactly where robustness should be changed.

---

9. Recommendations for Improvement

• Make IWANT malformed ID handling non-throwing in GossipSub.handle_iwant.
• Align tests with robust network-input handling (invalid control IDs are ignored, not propagated).
• Rebase/merge from latest origin/main and resolve the reported conflict in tests/core/pubsub/test_gossipsub.py.
• Fix issue linkage + newsfragment numbering consistency before merge.

---

10. Questions for the Author

• Was the intent to keep raising on malformed IWANT IDs, or to "skip safely" as described in PR narrative?
• Which repository issue is the canonical tracker for this Filecoin DX work? (Please link it in PR body and align newsfragment filename.)
• After conflict resolution, can you confirm pubsub control-path behavior under malformed IDs with an explicit non-throwing regression test?

---

11. Overall Assessment

• Quality Rating: Needs Work
• Security Impact: Medium
• Merge Readiness: Needs fixes
• Confidence: High

Primary blockers are: (1) malformed IWANT handling still raises; (2) merge conflict with origin/main; (3) issue/newsfragment linkage mismatch per policy.

[GautamBytes]

will do the suggested changes soon @acul71

[acul71]

thanks @GautamBytes for the mods.
Can you try the test fix (both failing and warnings) in the review ?
About the examples can you add prove that they are working with a message in this PR with a simple explanation and logs?

PR #1260 Review — Filecoin DX toolkit, examples, and architecture docs

1. Summary of Changes

• Adds a Filecoin-focused DX layer under libp2p.filecoin (constants, network presets, bootstrap/runtime helpers, pubsub presets, CLI entrypoint/module support).
• Adds Filecoin examples (connect, ping/identify, pubsub observer) and expands docs (docs/filecoin/*, docs/filecoin_architecture_positioning.rst, docs index wiring).
• Adds dependency/parity artifacts (artifacts/filecoin/libp2p_dependency_tree.v1.json) and Filecoin-targeted tests in tests/core/filecoin/*.
• Includes pubsub control-path hardening in libp2p/pubsub/gossipsub.py for malformed IHAVE/IWANT message ID handling.
• PR references and closes issue #1279 (issue body aligns with the implemented scope and acceptance criteria).
• No explicit breaking/deprecation changes noted for this PR’s Filecoin DX surface.

---

2. Branch Sync Status and Merge Conflicts

Branch Sync Status

• Status: ℹ️ Ahead of main (no behind commits).
• Details: branch_sync_status.txt = 0 11 (0 behind, 11 ahead vs libp2p-https/main).

Merge Conflict Analysis

• Conflicts Detected: ✅ No conflicts.
• merge_conflict_check.log reports Already up to date. and === NO MERGE CONFLICTS DETECTED ===.

---

3. Strengths

• Maintainer-requested process blockers from prior thread appear resolved:
  • PR now links a canonical repo issue (#1279).
  • Newsfragment now uses matching issue ID (newsfragments/1279.feature.rst).
  • Branch currently merges cleanly with current main.
• Pubsub malformed IWANT path now skips invalid hex IDs rather than raising in handle_iwant, reducing untrusted-input crash risk in control handling.
• Filecoin DX package, examples, and docs are cohesive and easy to discover (CLI scripts + docs index integration).
• Lint, typecheck, and docs build all succeed in this environment.

---

4. Issues Found

Critical

• File: tests/core/pubsub/test_gossipsub.py
• Line(s): 954–964
• Issue: The test asserts the wrong outbound callsite. handle_iwant() builds an RPC packet and dispatches it through router queueing (send_rpc), while the test expects a direct call to pubsub.write_msg.
• Why this fails consistently: The valid message ID is looked up correctly, but no write_msg call is made by this path, so assert_called_once() on mock_write_msg always fails.
• Implementation snippet (actual behavior):
  Source: libp2p/pubsub/gossipsub.py (lines 1618-1659)

msg_ids: list[bytes] = []
for msg_id_str in iwant_msg.messageIDs:
    mid_bytes = safe_bytes_from_hex(msg_id_str)
    if mid_bytes is None:
        logger.warning(...)
        continue
    msg_ids.append(mid_bytes)

# ... build packet ...
packet.publish.extend(msgs_to_forward)
self.send_rpc(sender_peer_id, packet)

• Test snippet (current expectation):
  Source: tests/core/pubsub/test_gossipsub.py (lines 954-964)

mock_write_msg = AsyncMock()
monkeypatch.setattr(gossipsubs[index_bob].pubsub, "write_msg", mock_write_msg)

iwant_msg = rpc_pb2.ControlIWant(messageIDs=["not_a_valid_msg_id", valid_msg_id])
await gossipsubs[index_bob].handle_iwant(iwant_msg, id_alice)

mock_mcache_get.assert_called_once_with(valid_mid)
mock_write_msg.assert_called_once()

• Failure evidence from test log:
  Source: downloads/AI-PR-REVIEWS/1260/test_output.log (lines 5420-5428)

File ".../tests/core/pubsub/test_gossipsub.py", line 963, in test_handle_iwant_mixed_valid_and_invalid_msg_ids
    mock_write_msg.assert_called_once()
AssertionError: Expected 'mock' to have been called once. Called 0 times.

• Cross-implementation parity note (JS + Go):
  • js-libp2p: control handling computes IWANT response payload, then sends via router-level sendRpc(...) (not direct stream write in handleIWant).
    • Source: /home/luca/PNL_Launchpad_Curriculum/Libp2p/js-libp2p/packages/gossipsub/src/gossipsub.ts (around lines 1215-1231, 1361-1407).
  • go-libp2p-pubsub (used by go-libp2p): same pattern — handleIWant(...) returns messages, outer control handler emits via sendRPC(...).
    • Source: /home/luca/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.15.0/gossipsub.go (around lines 885-896, 973-1019).
  • Conclusion: py-libp2p implementation is aligned with JS/Go architecture; this is test expectation drift.
• Suggested test fix (drop-in):
  Source: tests/core/pubsub/test_gossipsub.py (test_handle_iwant_mixed_valid_and_invalid_msg_ids)

mock_send_rpc = MagicMock()
monkeypatch.setattr(gossipsubs[index_bob], "send_rpc", mock_send_rpc)

iwant_msg = rpc_pb2.ControlIWant(messageIDs=["not_a_valid_msg_id", valid_msg_id])
await gossipsubs[index_bob].handle_iwant(iwant_msg, id_alice)

# invalid ID skipped; only valid ID looked up
mock_mcache_get.assert_called_once_with(valid_mid)

# response goes through router RPC path
mock_send_rpc.assert_called_once()
called_peer_id, packet = mock_send_rpc.call_args[0]
assert called_peer_id == id_alice
assert isinstance(packet, rpc_pb2.RPC)
assert len(packet.publish) == 1
assert packet.publish[0] == test_message

Major

• None beyond the critical test failure.

Minor

• Existing non-blocking test warning remains in suite output:
  • PytestCollectionWarning for TestVector in tests/core/records/test_ipns_validator.py (24 warnings aggregate).
  • Source: downloads/AI-PR-REVIEWS/1260/test_output.log (lines 5432-5435)

=============================== warnings summary ===============================
tests/core/records/test_ipns_validator.py:489: 24 warnings
  /home/luca/Informatica/Learning/PNL_Launchpad_Curriculum/Libp2p/py-libp2p/tests/core/records/test_ipns_validator.py:489: PytestCollectionWarning: cannot collect test class 'TestVector' because it has a __init__ constructor (from: tests/core/records/test_ipns_validator.py::TestIPNSSpecTestVectors)
    class TestVector(TypedDict, total=False):

• Suggested fix:
  • Preferred: rename the TypedDict so it does not start with Test (pytest test-discovery prefix), e.g. SpecVector.
  • Alternative: keep the name and set __test__ = False on the class.
• Source: tests/core/records/test_ipns_validator.py (around line 489)

# preferred
class SpecVector(TypedDict, total=False):
    file: str
    name: str
    valid: bool
    error: str
    value: str

TEST_VECTORS: dict[str, SpecVector] = {...}

# alternative
class TestVector(TypedDict, total=False):
    __test__ = False
    file: str
    name: str
    valid: bool
    error: str
    value: str

• This appears pre-existing and not specific to this PR, but worth eventual cleanup.

---

5. Security Review

• Risk: Previously reported malformed IWANT control IDs raising exceptions appears addressed in current head (handle_iwant now logs warning and continues on invalid hex IDs).
• Impact: Low-to-medium residual for this PR scope; main security-sensitive path improved.
• Mitigation: Keep malformed control IDs non-throwing and retain regression coverage for mixed valid/invalid ID batches.

No new obvious key-management, serialization, or transport-security regressions were identified in the added Filecoin DX modules.

---

6. Documentation and Examples

• Documentation and examples are comprehensive for a first DX release:
  • architecture positioning doc
  • parity/dependency artifact docs
  • Filecoin examples and docs index integration
• make linux-docs succeeds with -W (no Sphinx warnings/errors observed in this run).

---

7. Newsfragment Requirement

• ✅ Present and correctly named: newsfragments/1279.feature.rst.
• ✅ PR body explicitly links issue #1279 (Closes: #1279), consistent with policy and filename.
• ✅ Content is user-facing and suitable for release notes inclusion.

---

8. Tests and Validation

• Lint (make lint): ✅ pass.
• Typecheck (make typecheck): ✅ pass.
• Tests (make test): ❌ fail.
  • Summary: 1 failed, 2587 passed, 15 skipped, 24 warnings in 104.97s.
  • Failing test: tests/core/pubsub/test_gossipsub.py::test_handle_iwant_mixed_valid_and_invalid_msg_ids.
  • Failure mode: expected mocked write path not called (Called 0 times), while invalid-ID warning is emitted.
• Docs (make linux-docs): ✅ pass.

---

9. Recommendations for Improvement

• Fix test_handle_iwant_mixed_valid_and_invalid_msg_ids to assert the current send path (send_rpc) rather than write_msg.
• Keep the malformed-ID regression intent explicit: invalid IDs are skipped, valid IDs in same IWANT still forwarded.
• After test fix, rerun make test and ensure full suite passes before merge.

---

10. Questions for the Author

• Was the router send-path change in IWANT handling intentionally standardized on send_rpc (vs direct write_msg) for queueing consistency?
• Do you want this test to validate only cache lookup + forwarding semantics (path-agnostic), or specifically the transport callsite?

---

11. Overall Assessment

• Quality Rating: Needs Work
• Security Impact: Low
• Merge Readiness: Needs fixes
• Confidence: High

Primary blocker is the current failing test in tests/core/pubsub/test_gossipsub.py; previous policy/process blockers from earlier review comments are resolved in this PR head.

[acul71]

@GautamBytes @yashksaini-coder

Quick update on the pubsub demo failures and the IHAVE/IWANT message ID handling.

I reproduced the issue in Filecoin pubsub demos: they were crashing in handle_ihave with:
TypeError: fromhex() argument must be str, not bytes.

Root cause

Our handle_ihave/handle_iwant parsing path assumed hex string message IDs only, but in real network traffic we can encounter byte-form IDs as well. That mismatch caused the crash.

What was fixed

• Hardened safe_bytes_from_hex to accept:
  • str (hex string -> decode),
  • bytes/bytearray that are ASCII hex (decode),
  • raw bytes/bytearray (use as-is),
  • invalid input (return None, skip safely).
• Kept control-path behavior defensive: malformed IDs are skipped/logged, no crash.
• handle_ihave now canonicalizes requested IWANT IDs via normalized bytes (mid_bytes.hex()), so outgoing requests stay consistent.
• Added focused tests around parser behavior and revalidated existing IHAVE/IWANT handler tests.

Validation

• make pr passed
• make docs passed
• Re-ran all Filecoin demo/CLI commands with timeout 25s; all exited 0 and pubsub demos no longer crash.

Protobuf/spec note

There is still a cross-implementation/provenance nuance:

• py-libp2p/go-libp2p-pubsub currently use repeated string messageIDs in RPC proto.
• js-libp2p uses repeated bytes messageIDs.
• Spec docs have both historical and newer variants across files; this likely explains mixed payloads in the wild.

So this fix is an interoperability hardening at the parser boundary (safe now), while a longer-term discussion could be whether to align schema more strictly with current spec direction.

[acul71]

Is this ready to be merged ?

[GautamBytes]

Is this ready to be merged ?

Hey sorry for late reply , It seems good to me now. tagging @seetadev for final review