Generating piece url based on curio multiaddr and contract mapper by lukasz-wal · Pull Request #22 · fidlabs/provider-sample-url-finder

lukasz-wal · 2025-10-23T13:01:15Z

No description provided.

kacperzuk-neti · 2025-10-23T13:57:00Z

url_finder/src/api/find_client.rs

        let (_, retrievability_percent) = match timeout(
            Duration::from_secs(RETRIEVABILITY_TIMEOUT_SEC),
-            url_tester::get_retrivability_with_head(urls),
+            url_tester::get_retrivability_with_get(urls),


Suggested change

url_tester::get_retrivability_with_get(urls),

url_tester::get_retrievability_with_get(urls),

kacperzuk-neti · 2025-10-23T13:58:10Z

url_finder/src/cid_contact.rs

+                let final_addr = if trimmed.ends_with("/https") {
+                    trimmed.replace("/https", "/tcp/443/https")
+                } else if trimmed.ends_with("/http") {
+                    trimmed.replace("/http", "/tcp/80/http")
+                } else {
+                    trimmed.to_string()


We should work on this as a string, we should parse this with a proper multiaddr lib.

@CodeWarriorr did we try any? Were there any issues?

This is an example of "multiaddr" from curio:
/dns/isabella.hegsx.com/https/http-path/%2Fipni-provider%2F12D3KooWSbAPqdYw1MGrQSV8LgjZDYBvha8bQ2YtHUCzRYyqLCzy
When we try to parse the original address by lib => UnknownProtocolString("http-path")
After removing unknown parts, our internal parser needs a port:
Failed to convert multiaddr: "/dns/isabella.hegsx.com/https" to URL: "Missing port"

kacperzuk-neti · 2025-10-23T13:59:09Z

url_finder/src/provider_endpoints.rs

+}
+
+pub async fn valid_curio_provider(address: &str) -> Result<Option<String>> {
+    let rpc_url = "https://api.node.glif.io/rpc/v1";


Should be an env variable.

kacperzuk-neti · 2025-10-23T14:00:09Z

url_finder/src/provider_endpoints.rs

+    let contract_address: &str = "0x14183aD016Ddc83D638425D6328009aa390339Ce";
+
+    let miner_peer_id_contract = Address::parse_checksummed(contract_address, None)
+        .map_err(|e| eyre!("Failed to parse miner id contact: {e}"))?;


Use https://docs.rs/alloy-primitives/latest/alloy_primitives/macro.address.html instead (that way it will verify that the address is correct at build time).

kacperzuk-neti · 2025-10-23T14:01:04Z

url_finder/src/provider_endpoints.rs

+    let rpc_provider = ProviderBuilder::new()
+        .connect(rpc_url)
+        .await
+        .map_err(|err| eyre!("Building provider failed: {}", err))?;


Is this map_err needed here? Doesnt seem to add anything valueble and err already implements std error?

kacperzuk-neti · 2025-10-23T14:02:56Z

url_finder/src/provider_endpoints.rs

+        .map_err(|e| eyre!("Transaction failed: {e}"))?;
+
+    let peer_data: PeerData =
+        PeerData::abi_decode(response.as_ref()).map_err(|e| eyre!("Decode failed: {e}"))?;


kacperzuk-neti · 2025-10-23T14:04:19Z

url_finder/src/url_tester.rs

 const RETRI_TIMEOUT_SEC: u64 = 15;

 /// return first working url through head requests
+/// let't keep both head and get versions for now


Suggested change

/// let't keep both head and get versions for now

/// let's keep both head and get versions for now

kacperzuk-neti · 2025-10-24T07:31:35Z

url_finder/src/url_tester.rs

+                            && matches!(
+                                content_type,
+                                Some("application/octet-stream") | Some("application/piece")
+                            )
+                            && etag.is_some()


Why check for these headers?

just verifying whether it is a file, I notice that the curio and boost contain etag
curio - application/octet-stream
boost - application/piece
but generally, yes, it is not necessary :-)

kacperzuk-neti · 2025-10-24T07:33:19Z

url_finder/src/url_tester.rs

+                match client.get(&url).send().await {
+                    Ok(resp) => {
+                        let content_type = resp
+                            .headers()
+                            .get("content-type")
+                            .and_then(|v| v.to_str().ok());
+                        let etag = resp.headers().get("etag");
+
+                        // check if the response has a content-type:
+                        // * application/octet-stream => curio
+                        // * application/piece        => boost
+                        // and an etag
+                        // in  header to indicating a file
+                        if resp.status().is_success()
+                            && matches!(
+                                content_type,
+                                Some("application/octet-stream") | Some("application/piece")
+                            )
+                            && etag.is_some()
+                        {
+                            tracing::info!("url WORKING: {:?}", url);
+                            success_clone.fetch_add(1, Ordering::SeqCst);
+                            Some(url)
+                        } else {
+                            debug!("Retrivability: URL::GET not working url: {:?}", url);
+                            None
+                        }
+                    }
+                    Err(err) => {
+                        debug!(
+                            "Retrivability: Get request for working url failed for {:?}: {:?}",
+                            url, err
+                        );
+                        None
+                    }
+                }


This is big and complicated enough to extract and deduplicate with filter_working_with_get

kacperzuk-neti · 2025-10-24T07:34:56Z

url_finder/src/url_tester.rs

 }

 /// return retrivable percent of the urls
+/// let't keep both head and get versions for now


Suggested change

/// let't keep both head and get versions for now

/// let's keep both head and get versions for now

CodeWarriorr · 2025-10-24T07:30:50Z

url_finder/src/cid_contact.rs

+                    None => &cleaned,
+                };
+
+                let final_addr = if trimmed.ends_with("/https") {


is it possible that tcp is already there ?

no idea, we didn't know anyone had 'tcp' in multiaddr from curio....

CodeWarriorr · 2025-10-24T07:37:12Z

url_finder/src/main.rs

        .with_state(app_state.clone());

-    let server_addr = SocketAddr::from(([0, 0, 0, 0], 3010));
+    let server_addr = SocketAddr::from(([0, 0, 0, 0], 3030));


this breaks docker-compose

CodeWarriorr · 2025-10-24T07:49:58Z

url_finder/Cargo.toml

 chrono = { version = "0.4.38", features = ["serde"] }
 regex = "1.11.1"
 moka = { version = "0.12.10", default-features = false, features = ["future"] } 
+alloy = "1.0.41"


alloy has a lot build into default feature set, this adds up to build time.
giving that we use only small part, we should narrow it down.
something like this should work:

alloy = { version = "1.0.41", default-features = false, features = [ "sol-types", "network", "providers", "rpc-types-eth" ] }

adjusted a little bit and done

CodeWarriorr · 2025-11-04T09:55:45Z

url_finder/src/multiaddr_parser.rs

            }
        }
        Err(e) => {
+            println!("Failed to parse multiaddr: {:?} due to {:?}", addr, e);


Why do we need second log, and most importantly why it's println! instead of tracing::{error,info,debug} ?

My mistake, it was left over from local testing, thanks ! fixed

CodeWarriorr · 2025-11-04T10:08:53Z

url_finder/src/api/find_client.rs

            result: ResultCode::Success,
            working_url: first_url,
-            retrievability_percent,
+            retrievability_percent: retrievability_percent.unwrap(),


why do we need unwrap() here ? seems unnecessary and will panic on None
maybe use unwrap_or() or always return default 0.0:f64

CodeWarriorr · 2025-11-04T10:10:40Z

url_finder/src/provider_endpoints.rs

+                break;
+            }
+            Err(e) => {
+                println!("Attempt {attempt}/3 failed: {e} for address: {address}");


lukasz-wal added 6 commits March 18, 2025 12:07

Fix ref for TAG

7f04bdc

Merge branch 'main' of github.com:fidlabs/provider-sample-url-finder

c0400a8

Merge branch 'main' of github.com:fidlabs/provider-sample-url-finder

a1262f0

Parse curio addrss from cid.contract

10d6bcf

Implement mapper about filecoin <=> curio peerId

81b3d59

multiaddr

040e4a4

lukasz-wal requested review from CodeWarriorr and kacperzuk-neti October 23, 2025 13:01

lukasz-wal added 3 commits October 23, 2025 15:29

Adjust logs

6e13403

Fix clippy issues

a385f8a

Run formatter

e47aea1

lukasz-wal had a problem deploying to production-fidl October 24, 2025 06:39 — with GitHub Actions Failure

lukasz-wal temporarily deployed to production-fidl October 24, 2025 06:47 — with GitHub Actions Inactive

Bump rust image

9419e25

lukasz-wal force-pushed the feature/curio-or-boost branch from fe05d78 to 9419e25 Compare October 24, 2025 06:50

lukasz-wal temporarily deployed to production-fidl October 24, 2025 06:51 — with GitHub Actions Inactive

kacperzuk-neti requested changes Oct 24, 2025

View reviewed changes

CodeWarriorr requested changes Oct 24, 2025

View reviewed changes

lukasz-wal temporarily deployed to production-fidl October 27, 2025 12:28 — with GitHub Actions Inactive

lukasz-wal force-pushed the feature/curio-or-boost branch 2 times, most recently from 68fea97 to a254055 Compare October 27, 2025 14:55

After review

a080668

lukasz-wal force-pushed the feature/curio-or-boost branch from a254055 to a080668 Compare October 28, 2025 07:17

lukasz-wal temporarily deployed to production-fidl October 28, 2025 07:17 — with GitHub Actions Inactive

lukasz-wal requested review from CodeWarriorr and kacperzuk-neti October 28, 2025 07:20

lukasz-wal temporarily deployed to production-fidl October 29, 2025 10:37 — with GitHub Actions Inactive

CodeWarriorr reviewed Nov 5, 2025

View reviewed changes

lukasz-wal force-pushed the feature/curio-or-boost branch from af8f01b to b6889f3 Compare November 6, 2025 08:32

lukasz-wal temporarily deployed to production-fidl November 6, 2025 08:34 — with GitHub Actions Inactive

Add error handler for curio peerId contract requests

6cf47df

lukasz-wal force-pushed the feature/curio-or-boost branch from b6889f3 to 6cf47df Compare November 12, 2025 07:01

lukasz-wal temporarily deployed to production-fidl November 12, 2025 07:04 — with GitHub Actions Inactive

CodeWarriorr approved these changes Nov 14, 2025

View reviewed changes

lukasz-wal merged commit c5163e3 into main Nov 17, 2025
5 checks passed

	url_tester::get_retrivability_with_get(urls),
	url_tester::get_retrievability_with_get(urls),

	/// let't keep both head and get versions for now
	/// let's keep both head and get versions for now

Conversation

lukasz-wal commented Oct 23, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lukasz-wal Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lukasz-wal Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lukasz-wal Oct 27, 2025 •

edited

Loading

lukasz-wal Oct 27, 2025 •

edited

Loading