Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix azure source fetch logic + Remove stackpath #403

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

ShlomieLiberow
Copy link

Azure responds with with text/html;charset=utf-8 which fails when checking for an exact match to "text/html" + stackpath has closed down

…hecking for en exact match to "text/html" + stackpath has closed down
@ShlomieLiberow
Copy link
Author

The tests fail partially because the Actions build is broken. Partly fixed with this PR but also because it is not pulling in Cloudfront ranges for reasons that are unclear.

Latest Action trigger output:

[https://github.com/projectdiscovery/cdncheck/actions/runs/13477764514/job/37659026556]

2025/02/23 00:10:00 [err] could not fetch cdn item: could not get url https://k3t9x2h3.map2.ssl.hwcdn.net/ipblocks.txt: Get "https://k3t9x2h3.map2.ssl.hwcdn.net/ipblocks.txt": dial tcp: lookup k3t9x2h3.map2.ssl.hwcdn.net on 127.0.0.53:53: no such host
2025/02/23 00:10:00 [err] could not fetch waf item: could not get asn AS12222: GET https://ipinfo.io/AS12222: 401 { }
2025/02/23 00:10:32 [err] could not fetch cloud item: could not get url https://www.microsoft.com/en-us/download/confirmation.aspx?id=56519: Get "": unsupported protocol scheme ""
[common/fqdn] Defined 2 items for amazon
[common/fqdn] Defined 5 items for akamai
[common/fqdn] Defined 1 items for cloudflare
[common/fqdn] Defined 1 items for fastly
[common/fqdn] Defined 2 items for edgecast
[common/fqdn] Defined 1 items for incapsula
[cdn] Got 26 items for gocache
[cdn] Got 85 items for google
[waf] Got 15 items for cloudflare
[waf] Got 11 items for incapsula
[cloud] Got 8451 items for aws
[cloud] Got 708 items for google
[cloud] Got 777 items for oracle
[cloud] Got 53 items for office365

Running the same command locally:

go run . -output ../../sources_data.json
[common/fqdn] Defined 1 items for cloudflare
[common/fqdn] Defined 1 items for fastly
[common/fqdn] Defined 2 items for edgecast
[common/fqdn] Defined 1 items for incapsula
[common/fqdn] Defined 2 items for amazon
[common/fqdn] Defined 5 items for akamai
[cdn] Got 183 items for cloudfront
[cdn] Got 25 items for gocache
[cdn] Got 19 items for fastly
[cdn] Got 85 items for google
[cdn] Got 759 items for gcore
[waf] Got 15 items for cloudflare
[waf] Got 11 items for incapsula
[cloud] Got 708 items for google
[cloud] Got 777 items for oracle
[cloud] Got 65866 items for azure
[cloud] Got 555 items for zscaler
[cloud] Got 53 items for office365
[cloud] Got 8451 items for aws
~/git/cdncheck/cmd/generate-index git:[main]
go run . -output ../../sources_data.json
[common/fqdn] Defined 2 items for edgecast
[common/fqdn] Defined 1 items for incapsula
[common/fqdn] Defined 2 items for amazon
[common/fqdn] Defined 5 items for akamai
[common/fqdn] Defined 1 items for cloudflare
[common/fqdn] Defined 1 items for fastly
[cdn] Got 85 items for google
[cdn] Got 759 items for gcore
[cdn] Got 183 items for cloudfront
[cdn] Got 25 items for gocache
[cdn] Got 19 items for fastly
[waf] Got 15 items for cloudflare
[waf] Got 11 items for incapsula
[cloud] Got 555 items for zscaler
[cloud] Got 53 items for office365
[cloud] Got 8451 items for aws
[cloud] Got 708 items for google
[cloud] Got 777 items for oracle
[cloud] Got 65866 items for azure

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant