diff --git a/.gitignore b/.gitignore
index 6a92a2b..9c389b6 100644
--- a/.gitignore
+++ b/.gitignore
@@ -6,3 +6,4 @@ erl_crash.dump
/doc
.rvmrc
.tool-versions
+.claude
\ No newline at end of file
diff --git a/AGENTS.md b/AGENTS.md
new file mode 100644
index 0000000..c5b00d2
--- /dev/null
+++ b/AGENTS.md
@@ -0,0 +1,90 @@
+# AGENTS.md
+
+This file provides guidance to AI agents when working with code in this repository.
+
+## Project Overview
+
+Funkspector is an Elixir library (Hex package) for web scraping. It extracts data from HTML pages, XML sitemaps, and text sitemaps. Version 1.6.0, requires Elixir ~> 1.17.
+
+## Common Commands
+
+```bash
+# Install dependencies
+mix deps.get
+
+# Run unit tests (default, no network required)
+mix test
+
+# Run a single test file
+mix test test/page_scraper_test.exs
+
+# Run a specific test by line number
+mix test test/page_scraper_test.exs:15
+
+# Run integration tests (hit live URLs, require network)
+mix test --include integration
+
+# Run only integration tests
+mix test --include integration --only integration
+
+# Run everything (unit + integration)
+mix test --include integration
+
+# Lint (CI runs these)
+mix compile --warnings-as-errors
+mix format --check-formatted
+mix deps.unlock --check-unused
+
+# Auto-format code
+mix format
+```
+
+## Architecture
+
+All public functions live in `Funkspector` (`lib/funkspector.ex`) and follow the pattern:
+- Success: `{:ok, %Funkspector.Document{url, contents, data}}` or `{:ok, final_url, response}` for resolve
+- Error: `{:error, original_url, reason}`
+
+### Module Pipeline
+
+```
+Funkspector (public API) → Resolver → Document → Scraper
+```
+
+- **Funkspector** — four public functions: `resolve/2`, `page_scrape/2`, `sitemap_scrape/2`, `text_sitemap_scrape/2`. All accept an optional `contents:` keyword to skip the HTTP request.
+- **Resolver** — follows redirects (max 5 hops), handles SSL/TLS retries (fallback to tlsv1.2), decompresses gzip, supports basic auth.
+- **Document** — struct holding `url`, `contents`, `data`. Handles HTTP requests via Resolver (`request/2`) or loads pre-fetched contents (`load/2`).
+- **PageScraper** — uses Floki to extract `` links. Separates into `http.internal`, `http.external`, and `non_http`. Resolves relative URLs. Finds base href and canonical URL.
+- **SitemapScraper** — uses SweetXml with XPath `//url/loc/text()` to extract URLs from XML sitemaps.
+- **TextSitemapScraper** — splits text by newlines, filters empty lines.
+- **Utils** — `absolutify/2` (relative→absolute URL via URI.merge), `valid_url?/1` (regex, supports internationalized domains).
+
+### Key Dependencies
+
+- `httpoison` / `hackney` — HTTP client. Hackney pinned to ~> 1.21.0 due to [httpoison#501](https://github.com/edgurgel/httpoison/issues/501).
+- `floki` — HTML parsing
+- `sweet_xml` — XML/XPath parsing
+- `mock` (test only) — mocks HTTPoison in tests
+
+## Testing Conventions
+
+Tests are split into two categories:
+
+### Unit tests (default)
+Run with `mix test`. No network access required. All HTTP calls are mocked using the `Mock` library. Mock responses are defined in `test/support/mocked_connections.exs`.
+
+Pattern used across test files:
+```elixir
+with_mock HTTPoison, [get: fn(_url, _headers, _options) -> MockedConnections.successful_response() end] do
+ # test assertions
+end
+```
+
+### Integration tests
+Tagged with `@tag :integration` or `@moduletag :integration`. These hit live URLs and require network access. Excluded by default via `test_helper.exs`.
+
+Run with `mix test --include integration`.
+
+Integration tests live in:
+- `test/docs_test.exs` — doctests for `Funkspector` and `Funkspector.Resolver` (live URL examples from `@doc`)
+- `test/resolver_test.exs` — hackney regression test (tagged individually with `@tag :integration`)
diff --git a/CHANGELOG.md b/CHANGELOG.md
new file mode 100644
index 0000000..45077d5
--- /dev/null
+++ b/CHANGELOG.md
@@ -0,0 +1,172 @@
+# Changelog
+
+All notable changes to Funkspector are documented in this file.
+
+## [1.6.0](https://github.com/jaimeiniesta/funkspector/compare/940247c...842fe34) - 2025-11-20
+
+- Add another TLS retry reason for SSL handshake failures.
+- Upgrade HTTPoison to 2.3.0.
+- Pin hackney to ~> 1.21.0 to work around [httpoison#501](https://github.com/edgurgel/httpoison/issues/501).
+- Remove duplicated code.
+
+## [1.5.0](https://github.com/jaimeiniesta/funkspector/compare/2f0b9ec...940247c) - 2025-11-07
+
+- Extract and return canonical URL from `` in page scrapes.
+
+## [1.4.1](https://github.com/jaimeiniesta/funkspector/compare/c865fe5...2f0b9ec) - 2025-09-11
+
+- Upgrade hackney.
+- Remove support for Elixir 1.16.
+- Fix tests and warnings.
+
+## [1.4.0](https://github.com/jaimeiniesta/funkspector/compare/576f390...c865fe5) - 2024-12-24
+
+- Add support for HTTP Basic Authentication via `:basic_auth` option ([#12](https://github.com/jaimeiniesta/funkspector/pull/12)).
+
+## [1.3.0](https://github.com/jaimeiniesta/funkspector/compare/14483e0...576f390) - 2024-11-26
+
+- Support Elixir 1.17 ([#11](https://github.com/jaimeiniesta/funkspector/pull/11)).
+
+## [1.2.2](https://github.com/jaimeiniesta/funkspector/compare/daf6a55...14483e0) - 2024-09-04
+
+- Return proper error for HTTP Status 300 (Multiple Choices) ([#10](https://github.com/jaimeiniesta/funkspector/pull/10)).
+
+## [1.2.1](https://github.com/jaimeiniesta/funkspector/compare/ccf8442...daf6a55) - 2024-04-25
+
+- Upgrade Floki dependency.
+
+## [1.2.0](https://github.com/jaimeiniesta/funkspector/compare/5b0abba...ccf8442) - 2024-03-01
+
+- Validate URLs before making requests, returning `{:error, url, :invalid_url}` for invalid ones ([#9](https://github.com/jaimeiniesta/funkspector/pull/9)).
+- Add `Funkspector.resolve/2` as a top-level wrapper for `Resolver.resolve/2`.
+- Upgrade HTTPoison to 2.2.1 and Floki to 0.35.4.
+
+## [1.1.0](https://github.com/jaimeiniesta/funkspector/compare/c02012e...5b0abba) - 2023-10-23
+
+- Upgrade Floki, ex_doc, and Credo.
+
+## [1.0.0](https://github.com/jaimeiniesta/funkspector/compare/8478fd0...c02012e) - 2023-10-22
+
+- Introduce `Funkspector.Document` struct to hold `url`, `contents`, and `data`.
+- Refactor `PageScraper`, `SitemapScraper`, and `TextSitemapScraper` to use `Document` ([#8](https://github.com/jaimeiniesta/funkspector/pull/8)).
+- Support loading pre-fetched contents via `:contents` option, skipping the HTTP request.
+- Require Elixir 1.14 ([#7](https://github.com/jaimeiniesta/funkspector/pull/7)).
+- Add new SSL retry reason.
+
+## [0.9.1](https://github.com/jaimeiniesta/funkspector/compare/2c242be...8478fd0) - 2021-04-27
+
+- Rename `:urls` to `:lines` in text sitemap data for consistency with other scrapers (`:links`, `:locs`).
+
+## [0.9.0](https://github.com/jaimeiniesta/funkspector/compare/4891100...2c242be) - 2021-04-27
+
+- Add `TextSitemapScraper` and `Funkspector.text_sitemap_scrape/2` for plain text sitemaps.
+
+## [0.8.1](https://github.com/jaimeiniesta/funkspector/compare/b38cddf...4891100) - 2020-07-03
+
+- Add another SSL retry case.
+- Fix internal/external link classification to compare by host correctly.
+
+## [0.8.0](https://github.com/jaimeiniesta/funkspector/compare/c2081b0...b38cddf) - 2020-04-28
+
+- Include response headers in scraped data ([#5](https://github.com/jaimeiniesta/funkspector/pull/5)).
+- Add default connect timeout.
+- Trim raw links.
+- Upgrade to Elixir 1.8, hackney, and HTTPoison.
+
+## [0.7.1](https://github.com/jaimeiniesta/funkspector/compare/e5be0ca...c2081b0) - 2018-10-23
+
+- Fix absolutification of relative links when `base_href` is itself a relative link.
+
+## [0.7.0](https://github.com/jaimeiniesta/funkspector/compare/3fc533e...e5be0ca) - 2018-08-24
+
+- Move default options (hackney, recv_timeout, user_agent) into `Funkspector` module.
+- Pass options through to `Resolver`, `PageScraper`, and `SitemapScraper`.
+- Accept configurable `user_agent` via options.
+- Refactor Resolver to use a private `resolve_url/4` with a public `resolve/2` API.
+- Switch from `Enum.partition` to `Enum.split_with`.
+- Replace Friendly XML library with SweetXml.
+
+## [0.5.0](https://github.com/jaimeiniesta/funkspector/compare/00a3d94...3fc533e) - 2018-05-13
+
+- Upgrade dependencies.
+
+## [0.4.2](https://github.com/jaimeiniesta/funkspector/compare/2ca1504...00a3d94) - 2017-11-30
+
+- Use `` if present to absolutify relative links.
+
+## [0.4.1](https://github.com/jaimeiniesta/funkspector/compare/c4f5bcd...2ca1504) - 2017-09-18
+
+- Add another scenario for SSL version retry.
+
+## [0.4.0](https://github.com/jaimeiniesta/funkspector/compare/77511e3...c4f5bcd) - 2017-06-05
+
+- Upgrade HTTPoison to 0.11.
+- Provide a browser-like User-Agent header.
+- Retry with TLS version on SSL connection closed errors.
+
+## [0.3.4](https://github.com/jaimeiniesta/funkspector/compare/62df20e...77511e3) - 2017-06-03
+
+- Stop setting SSL option explicitly when using hackney insecure mode.
+- Increase receive timeout to 25 seconds.
+
+## [0.3.3](https://github.com/jaimeiniesta/funkspector/compare/9042413...62df20e) - 2017-05-11
+
+- Handle `x-gzip` Content-Encoding in addition to `gzip`.
+
+## [0.3.2](https://github.com/jaimeiniesta/funkspector/compare/d42f4af...9042413) - 2017-04-11
+
+- Version bump (no functional changes).
+
+## [0.3.1](https://github.com/jaimeiniesta/funkspector/compare/11eb8d7...d42f4af) - 2017-04-11
+
+- Fix HTTPoison SSL connection closed error.
+
+## [0.3.0](https://github.com/jaimeiniesta/funkspector/compare/8acf145...11eb8d7) - 2017-03-07
+
+- Avoid crashes when parsing XML sitemaps that include comments.
+
+## [0.2.0](https://github.com/jaimeiniesta/funkspector/compare/a574f0e...8acf145) - 2016-12-23
+
+- Decompress gzip-encoded response bodies.
+- Disable SSL certificate verification ([#4](https://github.com/jaimeiniesta/funkspector/pull/4)).
+- Update dependencies.
+
+## [0.1.5](https://github.com/jaimeiniesta/funkspector/compare/7a60b7d...a574f0e) - 2016-08-19
+
+- Fix relative link absolutification to use the base URL instead of the root URL.
+
+## [0.1.4](https://github.com/jaimeiniesta/funkspector/compare/41141d8...7a60b7d) - 2016-08-16
+
+- Follow `Location` headers case-insensitively.
+
+## [0.1.3](https://github.com/jaimeiniesta/funkspector/compare/c3a7053...41141d8) - 2016-08-04
+
+- Revert Quinn dependency upgrade; pin to 0.0.4.
+
+## [0.1.1](https://github.com/jaimeiniesta/funkspector/compare/940b482...c3a7053) - 2016-07-19
+
+- Recover gracefully from malformed XML when scraping sitemaps.
+
+## [0.1.0](https://github.com/jaimeiniesta/funkspector/compare/b995cc8...940b482) - 2016-07-19
+
+- Rename `Scraper` to `PageScraper`.
+- Add `SitemapScraper` for XML sitemaps using XPath.
+
+## [0.0.3](https://github.com/jaimeiniesta/funkspector/compare/9b5dc02...b995cc8) - 2016-06-29
+
+- Update dependencies.
+- Add Travis CI configuration.
+
+## [0.0.2](https://github.com/jaimeiniesta/funkspector/compare/7cf2fe4...9b5dc02) - 2016-06-22
+
+- Absolutify relative links.
+- Follow relative redirections.
+- Set Elixir 1.3 as minimum version.
+- Add documentation.
+
+## [0.0.1](https://github.com/jaimeiniesta/funkspector/commit/7cf2fe4) - 2016-06-02
+
+- Initial release.
+- Page scraping: extract internal, external, and non-HTTP links.
+- Handle redirections.
+- Deduplicate links.
diff --git a/CLAUDE.md b/CLAUDE.md
new file mode 100644
index 0000000..eef4bd2
--- /dev/null
+++ b/CLAUDE.md
@@ -0,0 +1 @@
+@AGENTS.md
\ No newline at end of file
diff --git a/README.md b/README.md
index fa0e602..230cd20 100644
--- a/README.md
+++ b/README.md
@@ -6,102 +6,116 @@ Web page inspector for Elixir.
Funkspector is a web scraper that lets you extract data from web pages and XML or TXT sitemaps.
+## Installation
+
+Add funkspector to your list of dependencies in `mix.exs`:
+
+```elixir
+def deps do
+ [{:funkspector, "~> 1.6"}]
+end
+```
+
## Usage
### Resolving URLs
-Simply pass Funkspector the URL to resolve and it will return its final URL after following redirections:
+Pass Funkspector a URL to resolve and it will return the final URL after following redirections:
```elixir
-iex> { :ok, final_url, _ } = Funkspector.resolve("http://example.com")
+{:ok, final_url, response} = Funkspector.resolve("http://example.com")
```
### Page Scraping
-Simply pass Funkspector the URL of a web page to inspect and it will return its scraped data:
+Pass Funkspector the URL of a web page and it will return a `Funkspector.Document` with the scraped data:
```elixir
-iex> { :ok, document } = Funkspector.page_scrape("https://rocketvalidator.com")
+{:ok, document} = Funkspector.page_scrape("https://example.com")
```
-### Sitemap Scraping
-
-Funkspector can extract the locations from XML sitemaps, like this:
+The returned `document.data` contains:
```elixir
-iex> { :ok, document } = Funkspector.sitemap_scrape("https://rocketvalidator.com/sitemap.xml")
+%{
+ urls: %{
+ original: "https://example.com",
+ base: "https://example.com",
+ canonical: nil,
+ parsed: %{scheme: "https", host: "example.com", ...},
+ root: "https://example.com/"
+ },
+ links: %{
+ raw: ["/about", "https://other.com", ...],
+ http: %{
+ internal: ["https://example.com/about", ...],
+ external: ["https://other.com", ...]
+ },
+ non_http: ["mailto:hi@example.com", ...]
+ },
+ headers: %{"content-type" => "text/html;charset=utf-8", ...}
+}
```
-It also supports TXT sitemaps:
+### Sitemap Scraping
+
+Funkspector can extract the locations from XML sitemaps:
```elixir
-iex> { :ok, document } = Funkspector.text_sitemap_scrape("https://rocketvalidator.com/sitemap.txt")
+{:ok, document} = Funkspector.sitemap_scrape("https://example.com/sitemap.xml")
+document.data.locs
+# => ["https://example.com/", "https://example.com/about", ...]
```
-### Custom User Agent
-You can specify a custom User Agent string using the `user_agent` option.
+It also supports plain text sitemaps:
-Example:
```elixir
- Funkspector.page_scrape("http://example.com", %{user_agent: "My Bot"})
+{:ok, document} = Funkspector.text_sitemap_scrape("https://example.com/sitemap.txt")
+document.data.lines
+# => ["https://example.com/", "https://example.com/about", ...]
```
-### Basic Auth
+### Options
-You can specify a basic auth username and password using the `basic_auth` option, which will be passed as an `Authorization` request header.
+#### Custom User Agent
-Example:
```elixir
- Funkspector.page_scrape("http://example.com", %{basic_auth: {"user", "secret"}})
+Funkspector.page_scrape("https://example.com", %{user_agent: "My Bot"})
```
-### Setting a custom timeout
-
-Use `recv_timeout` to set a custom timeout for the request, in milliseconds.
+#### Basic Auth
-Example:
```elixir
- Funkspector.page_scrape("http://example.com", %{recv_timeout: 5_000})
+Funkspector.page_scrape("https://example.com", %{basic_auth: {"user", "secret"}})
```
-### Loading a document contents instead of requesting
+#### Custom timeout
-You can skip the HTTP request of the document if you already have the contents of the document. This is useful in cases where you already have the contents from a previous request or cache. For example:
+Use `recv_timeout` to set a custom timeout in milliseconds:
```elixir
-Funkspector.page_scrape("https://example.com", contents: "...")
-
+Funkspector.page_scrape("https://example.com", %{recv_timeout: 5_000})
```
-### Scraped data
-
-For a successful response you'll get a `Funkspector.Document` with the scraped data, which will depend on the kind of scraper used. All data will be found inside the `:data` attribute.
-
-## Error response
+#### Loading pre-fetched contents
-In case of error, Funkspector will return the `original_url` and the reason from the server:
+You can skip the HTTP request if you already have the document contents:
```elixir
-case Funkspector.page_scrape("http://example.com") do
- { :ok, document } ->
- IO.inspect(data)
- { :error, url, reason } ->
- IO.puts "Could not scrape #{url} because of #{reason}"
-end
+Funkspector.page_scrape("https://example.com", %{contents: "..."})
+Funkspector.sitemap_scrape("https://example.com/sitemap.xml", %{contents: "..."})
+Funkspector.text_sitemap_scrape("https://example.com/sitemap.txt", %{contents: "..."})
```
-## Installation
-
-If [available in Hex](https://hex.pm/docs/publish), the package can be installed as:
+## Error handling
- 1. Add funkspector to your list of dependencies in `mix.exs`:
+In case of error, Funkspector returns the original URL and the reason:
- def deps do
- [{:funkspector, "~> 0.10"}]
- end
-
- 2. Ensure funkspector is started before your application:
-
- def application do
- [applications: [:funkspector]]
- end
+```elixir
+case Funkspector.page_scrape("https://example.com") do
+ {:ok, document} ->
+ IO.inspect(document.data)
+ {:error, url, reason} ->
+ IO.puts("Could not scrape #{url} because of #{inspect(reason)}")
+end
+```
diff --git a/ROADMAP.md b/ROADMAP.md
new file mode 100644
index 0000000..17c8835
--- /dev/null
+++ b/ROADMAP.md
@@ -0,0 +1,160 @@
+# Roadmap: Porting MetaInspector features to Funkspector
+
+Feature parity target: [MetaInspector](https://github.com/jaimeiniesta/metainspector) v5.17.1 (Ruby gem).
+
+## What Funkspector already has
+
+- [x] URL resolution with redirect following (up to 5 hops)
+- [x] SSL/TLS version fallback (retry with TLSv1.2)
+- [x] Gzip decompression
+- [x] Basic authentication
+- [x] Custom User-Agent
+- [x] Link extraction (raw, internal, external, non-HTTP)
+- [x] Base href detection
+- [x] Canonical URL detection
+- [x] URL absolutification (relative to absolute)
+- [x] URL validation (with internationalized domain support)
+- [x] Pre-loaded content support (skip HTTP request via `contents:` option)
+- [x] XML sitemap scraping (not in MetaInspector)
+- [x] Text sitemap scraping (not in MetaInspector)
+
+## Phase 1: Meta tags and text extraction
+
+Core content extraction that MetaInspector is best known for.
+
+### Meta tags
+
+Extract all `` tags from ``, grouped by attribute type:
+
+- `meta_tags` - nested map with keys `"name"`, `"http-equiv"`, `"property"`, `"charset"`. Values are lists (to support repeated tags like multiple `og:image`).
+- `meta_tag` - same structure but singular values (first occurrence only).
+- `meta` - flat merged map for simple access (`meta["og:title"]`, `meta["description"]`).
+
+All keys should be downcased.
+
+### Page title
+
+- `title` - text from `` tag.
+- `best_title` - smart selection in priority order:
+ 1. `<meta name="title">` value
+ 2. `og:title` meta property
+ 3. `<head><title>` text
+ 4. `<body><title>` text (rare but exists)
+ 5. First `<h1>` text
+
+### Description
+
+- `description` - `<meta name="description">` content.
+- `best_description` - smart selection:
+ 1. Meta description
+ 2. `og:description` meta property
+ 3. `twitter:description` meta property
+ 4. First `<p>` tag with 120+ characters
+
+### Author
+
+- `author` - `<meta name="author">` content.
+- `best_author` - smart selection:
+ 1. Meta author
+ 2. `<a rel="author">` link text
+ 3. `<address>` tag text
+ 4. `twitter:creator` meta property
+
+### Headings
+
+- `h1` through `h6` - lists of text content from each heading level.
+
+### Charset
+
+- `charset` - from `<meta charset="...">` or `<meta http-equiv="Content-Type">`.
+
+## Phase 2: Images and head links
+
+### Image extraction
+
+- `images` - list of all `<img src>` URLs, absolutified.
+- `images.best` - best image: `og:image` > `twitter:image` > largest image.
+- `images.largest` - largest image by area, filtering extreme aspect ratios (ratio between 0.1 and 10).
+- `images.owner_suggested` - `og:image` or `twitter:image`, or nil.
+- `images.favicon` - from `<link rel="icon">` or `<link rel="shortcut icon">`.
+- `images.with_size` - list of `{url, width, height}` tuples sorted by descending area. Uses HTML `width`/`height` attributes.
+
+### Head links
+
+- `head_links` - list of all `<link>` elements from `<head>`, each as a map of attributes with absolutified `href`.
+- `stylesheets` - filtered head links where `rel="stylesheet"`.
+- `canonicals` - filtered head links where `rel="canonical"` (as list, since there could be multiple).
+- `feeds` - `<link rel="alternate">` with type `application/rss+xml`, `application/atom+xml`, or `application/json`. Returns list of maps with `title`, `href`, `type`.
+
+## Phase 3: URL features and response access
+
+### URL normalization
+
+- Normalize URLs by default (add scheme, trailing slash, percent-encode international characters).
+- `normalize_url: false` option to disable.
+
+### UTM tracking detection
+
+- `tracked?` - boolean, true if URL contains `utm_source`, `utm_medium`, `utm_term`, `utm_content`, or `utm_campaign` parameters.
+- `untracked_url` - URL with tracking parameters stripped.
+
+### Response access
+
+- `response.status` - HTTP status code.
+- `response.headers` - response headers as a map.
+- `content_type` - MIME type from Content-Type header (without charset).
+
+### URL properties
+
+Funkspector already has `urls.parsed`, `urls.root`, etc. Expose these more explicitly:
+
+- `scheme` - URL scheme (`http`, `https`).
+- `host` - hostname.
+- `root_url` - `scheme://host/`.
+
+## Phase 4: Configuration and error handling
+
+### Options
+
+- `connection_timeout` - connection timeout (already have `timeout`).
+- `read_timeout` - receive timeout (already have `recv_timeout`).
+- `retries` - number of retry attempts on failure (MetaInspector defaults to 3).
+- `headers` - custom HTTP request headers (generalize beyond User-Agent).
+- `allow_redirections` - boolean to enable/disable redirect following (currently always enabled).
+- `max_redirects` - maximum number of redirects to follow (currently hardcoded to 5, MetaInspector uses 10).
+- `allow_non_html_content` - boolean, return error for non-HTML content types (default false).
+- `encoding` - force document encoding for pages with invalid UTF-8.
+
+### Error types
+
+Define specific error atoms or structs:
+
+- `:timeout` - request timed out.
+- `:request_error` - connection failed, invalid URI, SSL error.
+- `:non_html_content` - response is not HTML (when `allow_non_html_content: false`).
+
+### Cookie handling
+
+- Persist cookies across redirect hops (MetaInspector uses a cookie jar during redirects).
+
+## Phase 5: Serialization and advanced features
+
+### Serialization
+
+- `to_map` - returns a flat map with all extracted data (equivalent to MetaInspector's `to_hash`). Useful for JSON serialization or storage.
+
+### HTTP caching
+
+- Optional response caching to avoid re-fetching the same URL. Consider integration with ETS or a configurable cache backend.
+
+### Image size detection
+
+- Option to fetch image headers to determine dimensions (equivalent to FastImage). Lower priority since it requires additional HTTP requests per image.
+
+## Out of scope
+
+These MetaInspector features don't apply or have Elixir equivalents:
+
+- **Nokogiri document access** (`parsed`) - Funkspector uses Floki; users can call `Floki.parse_document!/1` on `document.contents` directly.
+- **Faraday middleware** - Funkspector uses HTTPoison/Hackney; middleware patterns differ in Elixir.
+- **Lazy evaluation** - MetaInspector defers HTTP requests until data is accessed. Funkspector's functional API (`page_scrape/2` returns everything at once) is more idiomatic in Elixir.
diff --git a/lib/funkspector.ex b/lib/funkspector.ex
index d88b282..1651441 100644
--- a/lib/funkspector.ex
+++ b/lib/funkspector.ex
@@ -1,12 +1,22 @@
defmodule Funkspector do
@moduledoc """
- Funkspector is a web scraper that lets you extract data from web pages.
+ A web scraper that extracts data from HTML pages, XML sitemaps, and text sitemaps.
+
+ Provides four public functions:
+
+ * `resolve/2` - follows URL redirections and returns the final URL
+ * `page_scrape/2` - parses an HTML page and extracts links
+ * `sitemap_scrape/2` - parses an XML sitemap and extracts URLs
+ * `text_sitemap_scrape/2` - parses a text sitemap and extracts URLs
+
+ All scrape functions accept an optional `contents:` keyword in the options
+ map to skip the HTTP request and scrape pre-fetched content instead.
"""
alias Funkspector.{Resolver, Document, PageScraper, SitemapScraper, TextSitemapScraper}
@doc """
- Given a URL, it will follow the redirections and return the final URL and the final response.
+ Follows redirections for the given URL, returning the final URL and response.
## Examples
@@ -14,6 +24,8 @@ defmodule Funkspector do
iex> final_url
"https://github.com/"
"""
+ @spec resolve(String.t(), map()) ::
+ {:ok, String.t(), map()} | {:error, String.t() | any(), any()}
def resolve(url, options \\ %{}) do
options = Map.merge(default_options(), options)
@@ -21,15 +33,15 @@ defmodule Funkspector do
end
@doc """
- Parses an HTML document.
+ Parses an HTML document, extracting links and metadata.
- This can be used to request a document by passing its URL, like:
+ Makes an HTTP request to the URL (following redirects), then parses the HTML
+ to extract internal/external links, raw links, non-HTTP links, canonical URL,
+ and base href.
- Funkspector.page_scrape("https://example.com")
+ Pass `contents:` in the options map to scrape pre-fetched HTML instead:
- Or to scrape an already loaded document, by passing its HTML contents:
-
- Funkspector.page_scrape("https://example.com", contents: "<html>...</html>")
+ Funkspector.page_scrape("https://example.com", %{contents: "<html>...</html>"})
## Example: request a document
@@ -42,52 +54,56 @@ defmodule Funkspector do
iex> Funkspector.page_scrape("https://notfoundwebsite.com")
{:error, "https://notfoundwebsite.com", %HTTPoison.Error{reason: :nxdomain, id: nil}}
"""
+ @spec page_scrape(String.t(), map()) ::
+ {:ok, Document.t()} | {:error, String.t() | any(), any()}
def page_scrape(url, options \\ %{}) do
scrape(url, options, &PageScraper.scrape/1)
end
@doc """
- Parses an XML sitemap.
-
- This can be used to request a document by passing its URL, like:
+ Parses an XML sitemap, extracting the list of URLs.
- Funkspector.sitemap_scrape("https://example.com")
+ Makes an HTTP request to the URL, then parses the XML to extract all
+ `<loc>` elements from `<url>` entries.
- Or to scrape an already loaded document, by passing its XML contents:
+ Pass `contents:` in the options map to scrape pre-fetched XML instead:
- Funkspector.sitemap_scrape("https://example.com/sitemap.xml", contents: "<xml>...</xml>")
+ Funkspector.sitemap_scrape("https://example.com/sitemap.xml", %{contents: "<xml>...</xml>"})
## Example
iex> { :ok, document } = Funkspector.sitemap_scrape("https://rocketvalidator.com/sitemap.xml")
- iex> length document.data.locs
- 2243
- iex> Enum.take(document.data.locs, 3)
- ["https://rocketvalidator.com/", "https://rocketvalidator.com/html-validation", "https://rocketvalidator.com/accessibility-validation"]
+ iex> length(document.data.locs) > 0
+ true
+ iex> hd(document.data.locs)
+ "https://rocketvalidator.com/"
"""
+ @spec sitemap_scrape(String.t(), map()) ::
+ {:ok, Document.t()} | {:error, String.t() | any(), any()}
def sitemap_scrape(url, options \\ %{}) do
scrape(url, options, &SitemapScraper.scrape/1)
end
@doc """
- Parses a text sitemap.
-
- This can be used to request a document by passing its URL, like:
+ Parses a plain text sitemap, extracting the list of URLs.
- Funkspector.text_sitemap_scrape("https://example.com")
+ Makes an HTTP request to the URL, then splits the text by newlines to
+ extract one URL per line.
- Or to scrape an already loaded document, by passing its text contents:
+ Pass `contents:` in the options map to scrape pre-fetched text instead:
- Funkspector.text_sitemap_scrape("https://example.com/sitemap.txt", contents: "...")
+ Funkspector.text_sitemap_scrape("https://example.com/sitemap.txt", %{contents: "..."})
## Example
iex> { :ok, document } = Funkspector.text_sitemap_scrape("https://rocketvalidator.com/sitemap.txt")
- iex> length document.data.lines
- 2243
- iex> Enum.take(document.data.lines, 3)
- ["https://rocketvalidator.com/", "https://rocketvalidator.com/html-validation", "https://rocketvalidator.com/accessibility-validation"]
+ iex> length(document.data.lines) > 0
+ true
+ iex> hd(document.data.lines)
+ "https://rocketvalidator.com/"
"""
+ @spec text_sitemap_scrape(String.t(), map()) ::
+ {:ok, Document.t()} | {:error, String.t() | any(), any()}
def text_sitemap_scrape(url, options \\ %{}) do
scrape(url, options, &TextSitemapScraper.scrape/1)
end
@@ -105,7 +121,7 @@ defmodule Funkspector do
}
end
- def scrape(url, options, scraping_function) do
+ defp scrape(url, options, scraping_function) do
options = Map.merge(default_options(), options)
case request_or_load_contents(url, options) do
diff --git a/lib/funkspector/document.ex b/lib/funkspector/document.ex
index adba561..c708813 100644
--- a/lib/funkspector/document.ex
+++ b/lib/funkspector/document.ex
@@ -1,8 +1,17 @@
defmodule Funkspector.Document do
@moduledoc """
Defines the Document struct and functions to request or load its contents.
+
+ A Document holds the final URL, the raw contents (HTML, XML, or text),
+ and a `data` map populated by the scrapers with extracted information.
"""
+ @type t :: %__MODULE__{
+ url: String.t() | nil,
+ contents: String.t() | nil,
+ data: map() | nil
+ }
+
defstruct [:url, :contents, :data]
alias __MODULE__
@@ -11,8 +20,14 @@ defmodule Funkspector.Document do
import Funkspector.Utils, only: [valid_url?: 1]
@doc """
- Retrieves the url and returns a document with its contents.
+ Retrieves the URL contents via HTTP and returns a Document.
+
+ Follows redirects (via `Funkspector.Resolver`) and returns the final URL
+ along with the response body and headers. Returns an error tuple if the
+ URL is invalid, the host cannot be resolved, or the response status is not 2xx.
"""
+ @spec request(String.t(), map()) ::
+ {:ok, t()} | {:error, String.t() | any(), any()}
def request(url, options \\ %{}) do
case Resolver.resolve(url, options) do
{:ok, final_url, response} ->
@@ -24,8 +39,13 @@ defmodule Funkspector.Document do
end
@doc """
- Returns a document with the given url and contents.
+ Creates a Document from pre-fetched contents without making an HTTP request.
+
+ Useful when the content has already been retrieved or for testing.
+ Returns an error if the URL is not valid.
"""
+ @spec load(String.t(), String.t()) ::
+ {:ok, t()} | {:error, String.t() | any(), :invalid_url}
def load(url, contents) do
if valid_url?(url) do
data = %{urls: parsed_url(url)}
diff --git a/lib/funkspector/page_scraper.ex b/lib/funkspector/page_scraper.ex
index ca113e3..36514fe 100644
--- a/lib/funkspector/page_scraper.ex
+++ b/lib/funkspector/page_scraper.ex
@@ -1,6 +1,10 @@
defmodule Funkspector.PageScraper do
@moduledoc """
- Scrapes an HTML page.
+ Extracts links and metadata from an HTML page.
+
+ Parses the HTML to find all `<a href>` links, then classifies them as
+ internal/external HTTP links or non-HTTP links (mailto, javascript, ftp, etc.).
+ Also extracts the `<base href>` and `<link rel="canonical">` values.
"""
import Funkspector.Utils
@@ -8,8 +12,18 @@ defmodule Funkspector.PageScraper do
alias Funkspector.Document
@doc """
- Scrapes the Document contents and returns the data scraped from its HTML.
+ Scrapes the Document contents and returns the data extracted from its HTML.
+
+ Populates the document's `data` map with:
+
+ * `urls.base` - the base URL (from `<base href>` or the document URL)
+ * `urls.canonical` - the canonical URL if present, otherwise `nil`
+ * `links.raw` - all raw `href` values found in `<a>` tags (deduplicated)
+ * `links.http.internal` - absolute HTTP links matching the document's host
+ * `links.http.external` - absolute HTTP links to other hosts
+ * `links.non_http` - non-HTTP links (mailto, javascript, ftp, etc.)
"""
+ @spec scrape(Document.t()) :: {:ok, Document.t()}
def scrape(%Document{} = document) do
{:ok, %{document | data: scraped_data(document)}}
end
diff --git a/lib/funkspector/resolver.ex b/lib/funkspector/resolver.ex
index 5d62542..1bac7d2 100644
--- a/lib/funkspector/resolver.ex
+++ b/lib/funkspector/resolver.ex
@@ -1,6 +1,10 @@
defmodule Funkspector.Resolver do
@moduledoc """
- Provides a method to follow URL redirections, returning the final URL.
+ Follows HTTP redirections and returns the final URL and response.
+
+ Handles up to 5 redirect hops, supports SSL/TLS version fallback
+ (retrying with TLSv1.2 on handshake failures), automatic gzip
+ decompression, basic authentication, and custom User-Agent headers.
"""
import Funkspector.Utils, only: [valid_url?: 1]
@@ -24,7 +28,20 @@ defmodule Funkspector.Resolver do
]
@doc """
- Given a URL, it will follow the redirections and return the final URL and the final response.
+ Follows redirections for the given URL and returns the final URL and response.
+
+ Validates the URL, then follows up to 5 HTTP redirects (status 301-399).
+ Returns an error for invalid URLs, non-2xx final responses, and HTTP 300
+ (Multiple Choices). On certain SSL errors, retries with TLSv1.2.
+
+ ## Options
+
+ * `:basic_auth` - `{username, password}` tuple for HTTP Basic Authentication
+ * `:user_agent` - custom User-Agent header string
+ * `:ssl` - SSL options passed to hackney
+ * `:hackney` - options passed directly to hackney
+ * `:timeout` - connection timeout in milliseconds
+ * `:recv_timeout` - receive timeout in milliseconds
## Examples
@@ -32,6 +49,8 @@ defmodule Funkspector.Resolver do
iex> final_url
"https://github.com/"
"""
+ @spec resolve(String.t() | any(), map()) ::
+ {:ok, String.t(), map()} | {:error, String.t() | any(), any()}
def resolve(url, options \\ %{}) do
if valid_url?(url) do
resolve_url(url, 5, %{}, options)
diff --git a/lib/funkspector/sitemap_scraper.ex b/lib/funkspector/sitemap_scraper.ex
index d50ca0b..3d15ad0 100644
--- a/lib/funkspector/sitemap_scraper.ex
+++ b/lib/funkspector/sitemap_scraper.ex
@@ -1,6 +1,11 @@
defmodule Funkspector.SitemapScraper do
@moduledoc """
- Scrapes an XML sitemap.
+ Extracts URLs from an XML sitemap.
+
+ Parses XML sitemaps conforming to the sitemaps.org protocol, extracting
+ `<loc>` elements from `<url>` entries using XPath. Relative URLs are
+ converted to absolute. Gracefully handles malformed XML by returning
+ an empty list.
"""
import Funkspector.Utils
@@ -9,8 +14,13 @@ defmodule Funkspector.SitemapScraper do
alias Funkspector.Document
@doc """
- Scrapes the Document contents and returns the data scraped from its XML.
+ Scrapes the Document contents and returns URLs extracted from the XML sitemap.
+
+ Populates the document's `data` map with a `:locs` key containing a
+ deduplicated list of absolute URLs found in `//url/loc` elements.
+ Returns an empty list if the XML cannot be parsed.
"""
+ @spec scrape(Document.t()) :: {:ok, Document.t()}
def scrape(%Document{} = document) do
{:ok, %{document | data: scraped_data(document)}}
end
diff --git a/lib/funkspector/text_sitemap_scraper.ex b/lib/funkspector/text_sitemap_scraper.ex
index daa72c1..b1c858d 100644
--- a/lib/funkspector/text_sitemap_scraper.ex
+++ b/lib/funkspector/text_sitemap_scraper.ex
@@ -1,6 +1,10 @@
defmodule Funkspector.TextSitemapScraper do
@moduledoc """
- Scrapes a text sitemap.
+ Extracts URLs from a plain text sitemap.
+
+ Parses text sitemaps where each line contains a single URL.
+ Empty lines and whitespace are stripped, duplicates are removed,
+ and relative URLs are converted to absolute.
"""
import Funkspector.Utils
@@ -8,8 +12,12 @@ defmodule Funkspector.TextSitemapScraper do
alias Funkspector.Document
@doc """
- Scrapes the Document contents and returns the data scraped from the text lines.
+ Scrapes the Document contents and returns URLs extracted from the text lines.
+
+ Populates the document's `data` map with a `:lines` key containing a
+ deduplicated list of absolute URLs, one per non-empty line in the text.
"""
+ @spec scrape(Document.t()) :: {:ok, Document.t()}
def scrape(%Document{} = document) do
{:ok, %{document | data: scraped_data(document)}}
end
diff --git a/lib/funkspector/utils.ex b/lib/funkspector/utils.ex
index 6d546b3..b303f2f 100644
--- a/lib/funkspector/utils.ex
+++ b/lib/funkspector/utils.ex
@@ -1,44 +1,57 @@
defmodule Funkspector.Utils do
@moduledoc """
- Common functionality for the scrapers.
+ Common utility functions shared across the scrapers.
+
+ Provides URL absolutification (converting relative URLs to absolute)
+ and URL validation using a regular expression that supports internationalized domain names.
"""
@url_regexp ~r/\Ahttp(s?)\:\/\/[a-zñäëïöüáéíóúàèìòùâêîôû0-9\-_]+([\.]{1}[a-zñäëïöüáéíóúàèìòùâêîôû0-9\-]+)*\.[a-z0-9]{2,5}(([0-9]{1,5})?(:(\d{1,5}))?\/?.*)?\z/i
@doc """
- Given a collection of URLs and a base URL, absolutifies the relative links.
+ Converts relative URLs to absolute URLs using the given base URL.
+
+ Accepts either a single URL string or a list of URL strings. Non-HTTP URLs
+ (like `javascript:` or `mailto:`) are passed through `URI.merge/2` unchanged
+ in practice, though the result depends on the URI scheme.
## Examples
iex> Funkspector.Utils.absolutify(["javascript:alert(hi)", "/faqs?section=legal", "/about", "http://example.com/faqs"], "http://example.com")
["javascript:alert(hi)", "http://example.com/faqs?section=legal", "http://example.com/about", "http://example.com/faqs"]
"""
+ @spec absolutify([String.t()], String.t()) :: [String.t()]
def absolutify(links, base_url) when is_list(links) do
links
|> Enum.map(&absolutify(&1, base_url))
end
+ @spec absolutify(String.t(), String.t()) :: String.t()
def absolutify(link, base_url) when is_binary(link) do
URI.merge(base_url, link) |> to_string()
end
@doc """
- Returns a boolean telling if the URL seems valid according to the regexp.
+ Returns whether the given URL matches a valid HTTP/HTTPS URL pattern.
+
+ Uses a regular expression that supports internationalized domain names
+ (with characters like ñ, ä, ë, etc.). Returns `false` for non-binary values.
## Examples
- iex> Funkspector.Utils.valid_url?("https://example.com")
- true
+ iex> Funkspector.Utils.valid_url?("https://example.com")
+ true
- iex> Funkspector.Utils.valid_url?("joe@example.com")
- false
+ iex> Funkspector.Utils.valid_url?("joe@example.com")
+ false
- iex> Funkspector.Utils.valid_url?(nil)
- false
+ iex> Funkspector.Utils.valid_url?(nil)
+ false
- iex> Funkspector.Utils.valid_url?(" ")
- false
+ iex> Funkspector.Utils.valid_url?(" ")
+ false
"""
+ @spec valid_url?(any()) :: boolean()
def valid_url?(url) when is_binary(url) do
Regex.match?(@url_regexp, url)
end
diff --git a/test/docs_test.exs b/test/docs_test.exs
new file mode 100644
index 0000000..63867e8
--- /dev/null
+++ b/test/docs_test.exs
@@ -0,0 +1,24 @@
+defmodule Funkspector.DocsTest do
+ @moduledoc """
+ Runs doctests from modules whose `@doc` examples use live URLs.
+
+ The `@doc` blocks in `Funkspector` and `Funkspector.Resolver` contain
+ examples that hit real URLs (e.g. `http://github.com`), so they require
+ network access. This module runs those doctests behind the `:integration`
+ tag so they are excluded from the default `mix test` run.
+
+ This lets us keep realistic examples in the module documentation without
+ breaking offline or CI test runs.
+
+ Excluded by default. Run with:
+
+ mix test --include integration
+ """
+
+ use ExUnit.Case
+
+ @moduletag :integration
+
+ doctest Funkspector
+ doctest Funkspector.Resolver
+end
diff --git a/test/document_test.exs b/test/document_test.exs
index 093cdfb..0af0097 100644
--- a/test/document_test.exs
+++ b/test/document_test.exs
@@ -54,6 +54,35 @@ defmodule Funkspector.DocumentTest do
}}
end
end
+
+ test "returns error for non-2xx response status" do
+ with_mock HTTPoison, get: fn _url, _headers, _options -> server_error_response(500) end do
+ {:error, @url, response} = Document.request(@url)
+ assert response.status_code == 500
+ end
+ end
+
+ test "returns error for 403 forbidden" do
+ with_mock HTTPoison, get: fn _url, _headers, _options -> forbidden_response() end do
+ {:error, @url, response} = Document.request(@url)
+ assert response.status_code == 403
+ end
+ end
+
+ test "returns error when host cannot be resolved" do
+ with_mock HTTPoison, get: fn _url, _headers, _options -> http_error_response() end do
+ assert Document.request(@url) ==
+ {:error, @url, %HTTPoison.Error{id: nil, reason: :nxdomain}}
+ end
+ end
+
+ test "preserves original URL when following redirects" do
+ with_mock HTTPoison, get: fn url, _headers, _options -> redirect_from(url) end do
+ {:ok, document} = Document.request("http://example.com/redirect/1")
+ assert document.url == "http://example.com/redirect/3"
+ assert document.data.urls.original == "http://example.com/redirect/1"
+ end
+ end
end
describe "load" do
@@ -86,5 +115,23 @@ defmodule Funkspector.DocumentTest do
}
}}
end
+
+ test "does not include original URL in loaded document" do
+ {:ok, document} = Document.load(@url, @html)
+ refute Map.has_key?(document.data.urls, :original)
+ end
+
+ test "does not include headers in loaded document" do
+ {:ok, document} = Document.load(@url, @html)
+ refute Map.has_key?(document.data, :headers)
+ end
+
+ test "parses URL with path and query" do
+ url = "https://example.com/page?q=test"
+ {:ok, document} = Document.load(url, @html)
+ assert document.data.urls.parsed.path == "/page"
+ assert document.data.urls.parsed.query == "q=test"
+ assert document.data.urls.root == "https://example.com/"
+ end
end
end
diff --git a/test/funkspector_test.exs b/test/funkspector_test.exs
index 36d078e..a965a39 100644
--- a/test/funkspector_test.exs
+++ b/test/funkspector_test.exs
@@ -1,8 +1,6 @@
defmodule FunkspectorTest do
use ExUnit.Case
- doctest Funkspector
-
import Mock
import FunkspectorTest.MockedConnections
@@ -298,10 +296,10 @@ defmodule FunkspectorTest do
end
end
- test "retuns error if page does not exist" do
+ test "returns error if page does not exist" do
with_mock HTTPoison, get: fn _url, _headers, _options -> http_error_response() end do
- assert Funkspector.sitemap_scrape("https://example.com/sitemap.xml") ==
- {:error, "https://example.com/sitemap.xml",
+ assert Funkspector.text_sitemap_scrape("https://example.com/sitemap.txt") ==
+ {:error, "https://example.com/sitemap.txt",
%HTTPoison.Error{reason: :nxdomain, id: nil}}
end
end
@@ -316,6 +314,28 @@ defmodule FunkspectorTest do
end
end
+ describe "resolve" do
+ test "follows redirections and returns final URL" do
+ with_mock HTTPoison, get: fn url, _headers, _options -> redirect_from(url) end do
+ {:ok, final_url, _response} = Funkspector.resolve("http://example.com/redirect/1")
+ assert final_url == "http://example.com/redirect/3"
+ end
+ end
+
+ test "returns error if URL is invalid" do
+ for url <- @invalid_urls do
+ assert Funkspector.resolve(url) == {:error, url, :invalid_url}
+ end
+ end
+
+ test "returns error if host does not exist" do
+ with_mock HTTPoison, get: fn _url, _headers, _options -> http_error_response() end do
+ assert Funkspector.resolve("https://example.com") ==
+ {:error, "https://example.com", %HTTPoison.Error{reason: :nxdomain, id: nil}}
+ end
+ end
+ end
+
describe "loading a document contents" do
test "page_scrape" do
{:ok, document} = Funkspector.page_scrape("https://example.com", %{contents: @html_doc})
diff --git a/test/page_scraper_test.exs b/test/page_scraper_test.exs
index 935381c..18b5688 100644
--- a/test/page_scraper_test.exs
+++ b/test/page_scraper_test.exs
@@ -270,4 +270,94 @@ defmodule PageScraperTest do
"ftp://ftp.example.com"
]
end
+
+ test "handles HTML with no links" do
+ html = "<html><head><title>No linksHello
"
+ {:ok, document} = Document.load("https://example.com", html)
+ {:ok, %Document{data: data}} = PageScraper.scrape(document)
+
+ assert data.links.raw == []
+ assert data.links.http.internal == []
+ assert data.links.http.external == []
+ assert data.links.non_http == []
+ end
+
+ test "handles empty HTML body" do
+ html = ""
+ {:ok, document} = Document.load("https://example.com", html)
+ {:ok, %Document{data: data}} = PageScraper.scrape(document)
+
+ assert data.links.raw == []
+ end
+
+ test "deduplicates identical links" do
+ html = """
+
+ Link 1
+ Link 2
+ Link 3
+
+ """
+
+ {:ok, document} = Document.load("https://example.com", html)
+ {:ok, %Document{data: data}} = PageScraper.scrape(document)
+
+ assert data.links.raw == ["/page"]
+ assert data.links.http.internal == ["https://example.com/page"]
+ end
+
+ test "trims whitespace from link hrefs" do
+ html = """
+
+ Padded link
+
+ """
+
+ {:ok, document} = Document.load("https://example.com", html)
+ {:ok, %Document{data: data}} = PageScraper.scrape(document)
+
+ assert data.links.raw == ["/page"]
+ end
+
+ test "handles anchor-only links as internal" do
+ html = """
+
+ Anchor
+
+ """
+
+ {:ok, document} = Document.load("https://example.com/page", html)
+ {:ok, %Document{data: data}} = PageScraper.scrape(document)
+
+ assert data.links.raw == ["#section"]
+ assert data.links.http.internal == ["https://example.com/page#section"]
+ end
+
+ test "handles links with empty href" do
+ html = """
+
+ Empty href
+
+ """
+
+ {:ok, document} = Document.load("https://example.com/page", html)
+ {:ok, %Document{data: data}} = PageScraper.scrape(document)
+
+ assert data.links.raw == [""]
+ assert data.links.http.internal == ["https://example.com/page"]
+ end
+
+ test "handles absolutifying relative base href" do
+ html = """
+
+ A page
+
+ """
+
+ {:ok, document} = Document.load("https://example.com/dir/", html)
+ {:ok, %Document{data: data}} = PageScraper.scrape(document)
+
+ assert data.urls.base == "https://example.com/base/"
+ assert data.links.http.internal == ["https://example.com/base/page"]
+ end
end
diff --git a/test/resolver_test.exs b/test/resolver_test.exs
index 928bfd5..3c00f8c 100644
--- a/test/resolver_test.exs
+++ b/test/resolver_test.exs
@@ -1,6 +1,5 @@
defmodule Funkspector.ResolverTest do
use ExUnit.Case
- doctest Funkspector.Resolver
import Mock
import FunkspectorTest.MockedConnections
@@ -105,7 +104,97 @@ defmodule Funkspector.ResolverTest do
end
end
+ test "returns error for server error status codes" do
+ with_mock HTTPoison,
+ get: fn _url, _headers, _options -> {:ok, %{status_code: 500, body: "error"}} end do
+ {:error, "https://example.com/", _} = resolve("https://example.com/")
+ end
+ end
+
+ test "returns error for 403 forbidden" do
+ with_mock HTTPoison,
+ get: fn _url, _headers, _options -> {:ok, %{status_code: 403, body: "forbidden"}} end do
+ {:error, "https://example.com/secret", _} = resolve("https://example.com/secret")
+ end
+ end
+
+ test "decompresses gzip-encoded responses" do
+ with_mock HTTPoison, get: fn _url, _headers, _options -> gzip_response() end do
+ {:ok, "https://example.com/", response} = resolve("https://example.com/")
+ assert response.body == mocked_html()
+ end
+ end
+
+ test "retries with TLSv1.2 on SSL closed error" do
+ call_count = :counters.new(1, [:atomics])
+
+ with_mock HTTPoison,
+ get: fn _url, _headers, options ->
+ :counters.add(call_count, 1, 1)
+ count = :counters.get(call_count, 1)
+
+ if count == 1 do
+ # First call fails with SSL error
+ ssl_closed_error()
+ else
+ # Retry with SSL option should include ssl version
+ assert Keyword.get(options, :ssl) == [versions: [:"tlsv1.2"]]
+ successful_response()
+ end
+ end do
+ {:ok, "https://example.com/", _} = resolve("https://example.com/")
+ assert :counters.get(call_count, 1) == 2
+ end
+ end
+
+ test "returns error on SSL failure when already retried with ssl option" do
+ with_mock HTTPoison,
+ get: fn _url, _headers, _options ->
+ ssl_closed_error()
+ end do
+ {:error, "https://example.com/", _} =
+ resolve("https://example.com/", %{ssl: [versions: [:"tlsv1.2"]]})
+ end
+ end
+
+ test "retries with TLSv1.2 on SSL handshake failure" do
+ call_count = :counters.new(1, [:atomics])
+
+ with_mock HTTPoison,
+ get: fn _url, _headers, options ->
+ :counters.add(call_count, 1, 1)
+ count = :counters.get(call_count, 1)
+
+ if count == 1 do
+ ssl_handshake_error()
+ else
+ assert Keyword.get(options, :ssl) == [versions: [:"tlsv1.2"]]
+ successful_response()
+ end
+ end do
+ {:ok, "https://example.com/", _} = resolve("https://example.com/")
+ assert :counters.get(call_count, 1) == 2
+ end
+ end
+
+ test "stops following redirects after 5 hops" do
+ with_mock HTTPoison, get: fn url, _headers, _options -> long_redirect_chain(url) end do
+ # Starts at /chain/1, follows 5 hops (max_redirects decrements each time),
+ # arriving at /chain/6 which returns a 200 response on the 5th redirect hop.
+ {:ok, final_url, _} = resolve("http://example.com/chain/1")
+ assert final_url == "http://example.com/chain/6"
+ end
+ end
+
+ test "returns error for 1xx status codes" do
+ with_mock HTTPoison,
+ get: fn _url, _headers, _options -> {:ok, %{status_code: 100, body: ""}} end do
+ {:error, "https://example.com/", _} = resolve("https://example.com/")
+ end
+ end
+
# This fails with hackney greater than 1.21.0
+ @tag :integration
test "https://github.com/edgurgel/httpoison/issues/501 regression test" do
assert {:ok, "https://www.freedomfromtorture.org/", %HTTPoison.Response{status_code: 200}} =
resolve("https://www.freedomfromtorture.org/")
diff --git a/test/sitemap_scraper_test.exs b/test/sitemap_scraper_test.exs
index f881d75..84bf472 100644
--- a/test/sitemap_scraper_test.exs
+++ b/test/sitemap_scraper_test.exs
@@ -52,4 +52,85 @@ defmodule SitemapScraperTest do
assert data.locs == []
end
+
+ test "returns empty locs for empty XML sitemap" do
+ xml = """
+
+
+
+ """
+
+ {:ok, document} = Document.load("https://example.com/sitemap.xml", xml)
+ {:ok, %Document{data: data}} = SitemapScraper.scrape(document)
+
+ assert data.locs == []
+ end
+
+ test "absolutifies relative locs", %{url: url} do
+ xml = """
+
+
+ /page1
+ /page2
+
+ """
+
+ {:ok, document} = Document.load(url, xml)
+ {:ok, %Document{data: data}} = SitemapScraper.scrape(document)
+
+ assert data.locs == [
+ "https://example.com/page1",
+ "https://example.com/page2"
+ ]
+ end
+
+ test "deduplicates locs" do
+ xml = """
+
+
+ /page
+ /page
+ /other
+
+ """
+
+ {:ok, document} = Document.load("https://example.com/sitemap.xml", xml)
+ {:ok, %Document{data: data}} = SitemapScraper.scrape(document)
+
+ assert data.locs == [
+ "https://example.com/page",
+ "https://example.com/other"
+ ]
+ end
+
+ test "handles XML with absolute URLs" do
+ xml = """
+
+
+ https://example.com/page1
+ https://example.com/page2
+
+ """
+
+ {:ok, document} = Document.load("https://example.com/sitemap.xml", xml)
+ {:ok, %Document{data: data}} = SitemapScraper.scrape(document)
+
+ assert data.locs == [
+ "https://example.com/page1",
+ "https://example.com/page2"
+ ]
+ end
+
+ test "returns empty locs for completely empty content" do
+ {:ok, document} = Document.load("https://example.com/sitemap.xml", "")
+ {:ok, %Document{data: data}} = SitemapScraper.scrape(document)
+
+ assert data.locs == []
+ end
+
+ test "ignores commented-out URLs", %{document: document} do
+ {:ok, %Document{data: data}} = SitemapScraper.scrape(document)
+
+ refute "https://example.com/commented-out-should-not-be-included" in data.locs
+ end
end
diff --git a/test/support/mocked_connections.exs b/test/support/mocked_connections.exs
index ea22fe7..5c1b118 100644
--- a/test/support/mocked_connections.exs
+++ b/test/support/mocked_connections.exs
@@ -44,6 +44,37 @@ defmodule FunkspectorTest.MockedConnections do
{:error, %HTTPoison.Error{id: nil, reason: :nxdomain}}
end
+ def gzip_response() do
+ body = :zlib.gzip(mocked_html())
+
+ {:ok,
+ %{
+ status_code: 200,
+ headers: [
+ {"content-length", "12345"},
+ {"content-type", "text/html;charset=utf-8"},
+ {"Content-Encoding", "gzip"}
+ ],
+ body: body
+ }}
+ end
+
+ def ssl_closed_error() do
+ {:error, %HTTPoison.Error{id: nil, reason: :closed}}
+ end
+
+ def ssl_handshake_error() do
+ {:error, %HTTPoison.Error{id: nil, reason: {:tls_alert, ~c"handshake failure"}}}
+ end
+
+ def server_error_response(status \\ 500) do
+ {:ok, %{status_code: status, body: "Internal Server Error"}}
+ end
+
+ def forbidden_response() do
+ {:ok, %{status_code: 403, body: "Forbidden"}}
+ end
+
def redirect_from(url) do
case url do
"http://example.com/redirect/1" ->
@@ -63,6 +94,28 @@ defmodule FunkspectorTest.MockedConnections do
end
end
+ def long_redirect_chain(url) do
+ case url do
+ "http://example.com/chain/1" ->
+ redirection_response("Location", "http://example.com/chain/2")
+
+ "http://example.com/chain/2" ->
+ redirection_response("Location", "http://example.com/chain/3")
+
+ "http://example.com/chain/3" ->
+ redirection_response("Location", "http://example.com/chain/4")
+
+ "http://example.com/chain/4" ->
+ redirection_response("Location", "http://example.com/chain/5")
+
+ "http://example.com/chain/5" ->
+ redirection_response("Location", "http://example.com/chain/6")
+
+ "http://example.com/chain/6" ->
+ successful_response()
+ end
+ end
+
def mocked_html() do
"""
diff --git a/test/test_helper.exs b/test/test_helper.exs
index 6ef6019..8d5195e 100644
--- a/test/test_helper.exs
+++ b/test/test_helper.exs
@@ -1,4 +1,4 @@
-ExUnit.start()
+ExUnit.start(exclude: [:integration])
{:ok, files} = File.ls("./test/support")
diff --git a/test/text_sitemap_scraper_test.exs b/test/text_sitemap_scraper_test.exs
index 5b9bf88..1f956be 100644
--- a/test/text_sitemap_scraper_test.exs
+++ b/test/text_sitemap_scraper_test.exs
@@ -44,4 +44,69 @@ defmodule TextSitemapScraperTest do
}
}
end
+
+ test "handles empty text content" do
+ {:ok, document} = Document.load("https://example.com/sitemap.txt", "")
+ {:ok, %Document{data: data}} = TextSitemapScraper.scrape(document)
+
+ assert data.lines == []
+ end
+
+ test "handles whitespace-only content" do
+ {:ok, document} = Document.load("https://example.com/sitemap.txt", " \n \n ")
+ {:ok, %Document{data: data}} = TextSitemapScraper.scrape(document)
+
+ assert data.lines == []
+ end
+
+ test "handles single URL" do
+ {:ok, document} = Document.load("https://example.com/sitemap.txt", "https://example.com/page")
+ {:ok, %Document{data: data}} = TextSitemapScraper.scrape(document)
+
+ assert data.lines == ["https://example.com/page"]
+ end
+
+ test "deduplicates URLs" do
+ text = "/page\n/page\n/other\n"
+ {:ok, document} = Document.load("https://example.com/sitemap.txt", text)
+ {:ok, %Document{data: data}} = TextSitemapScraper.scrape(document)
+
+ assert data.lines == [
+ "https://example.com/page",
+ "https://example.com/other"
+ ]
+ end
+
+ test "absolutifies relative URLs" do
+ text = "/about\n/contact\n"
+ {:ok, document} = Document.load("https://example.com/sitemap.txt", text)
+ {:ok, %Document{data: data}} = TextSitemapScraper.scrape(document)
+
+ assert data.lines == [
+ "https://example.com/about",
+ "https://example.com/contact"
+ ]
+ end
+
+ test "trims whitespace from URLs" do
+ text = " /about \n /contact \n"
+ {:ok, document} = Document.load("https://example.com/sitemap.txt", text)
+ {:ok, %Document{data: data}} = TextSitemapScraper.scrape(document)
+
+ assert data.lines == [
+ "https://example.com/about",
+ "https://example.com/contact"
+ ]
+ end
+
+ test "skips blank lines between URLs" do
+ text = "/page1\n\n\n/page2\n\n"
+ {:ok, document} = Document.load("https://example.com/sitemap.txt", text)
+ {:ok, %Document{data: data}} = TextSitemapScraper.scrape(document)
+
+ assert data.lines == [
+ "https://example.com/page1",
+ "https://example.com/page2"
+ ]
+ end
end
diff --git a/test/utils_test.exs b/test/utils_test.exs
index 0e8a251..12cbe10 100644
--- a/test/utils_test.exs
+++ b/test/utils_test.exs
@@ -2,4 +2,124 @@ defmodule UtilsTest do
use ExUnit.Case
doctest Funkspector.Utils
+
+ alias Funkspector.Utils
+
+ describe "absolutify/2 with a list" do
+ test "returns empty list for empty input" do
+ assert Utils.absolutify([], "https://example.com") == []
+ end
+
+ test "absolutifies relative paths" do
+ links = ["/about", "/contact"]
+
+ assert Utils.absolutify(links, "https://example.com") == [
+ "https://example.com/about",
+ "https://example.com/contact"
+ ]
+ end
+
+ test "preserves absolute URLs" do
+ links = ["https://other.com/page"]
+ assert Utils.absolutify(links, "https://example.com") == ["https://other.com/page"]
+ end
+
+ test "handles URLs with query parameters" do
+ links = ["/search?q=test&page=2"]
+
+ assert Utils.absolutify(links, "https://example.com") == [
+ "https://example.com/search?q=test&page=2"
+ ]
+ end
+
+ test "handles URLs with fragments" do
+ links = ["/page#section"]
+
+ assert Utils.absolutify(links, "https://example.com") == [
+ "https://example.com/page#section"
+ ]
+ end
+
+ test "resolves relative to base URL path" do
+ assert Utils.absolutify(["page"], "https://example.com/dir/") == [
+ "https://example.com/dir/page"
+ ]
+ end
+ end
+
+ describe "absolutify/2 with a single string" do
+ test "absolutifies a relative URL" do
+ assert Utils.absolutify("/about", "https://example.com") == "https://example.com/about"
+ end
+
+ test "preserves an absolute URL" do
+ assert Utils.absolutify("https://other.com", "https://example.com") == "https://other.com"
+ end
+ end
+
+ describe "valid_url?/1" do
+ test "returns true for valid HTTP URLs" do
+ assert Utils.valid_url?("http://example.com")
+ assert Utils.valid_url?("https://example.com")
+ assert Utils.valid_url?("https://example.com/path")
+ assert Utils.valid_url?("https://example.com/path?query=value")
+ assert Utils.valid_url?("https://example.com:8080/path")
+ end
+
+ test "returns true for internationalized domain names" do
+ assert Utils.valid_url?("https://ejemplo.ñandú.com")
+ assert Utils.valid_url?("https://ñandú.com")
+ assert Utils.valid_url?("https://münchen.de")
+ end
+
+ test "returns true for subdomains" do
+ assert Utils.valid_url?("https://sub.example.com")
+ assert Utils.valid_url?("https://a.b.c.example.com")
+ end
+
+ test "returns false for invalid TLDs" do
+ refute Utils.valid_url?("https://example.x")
+ refute Utils.valid_url?("https://example.c")
+ refute Utils.valid_url?("https://example.ñ")
+ refute Utils.valid_url?("https://example.ññ")
+ end
+
+ test "returns false for non-HTTP URLs" do
+ refute Utils.valid_url?("ftp://example.com")
+ refute Utils.valid_url?("mailto:user@example.com")
+ refute Utils.valid_url?("javascript:alert(1)")
+ end
+
+ test "returns false for invalid formats" do
+ refute Utils.valid_url?("not a url")
+ refute Utils.valid_url?("http://")
+ refute Utils.valid_url?("")
+ refute Utils.valid_url?(" ")
+ end
+
+ test "returns false for non-binary values" do
+ refute Utils.valid_url?(nil)
+ refute Utils.valid_url?(123)
+ refute Utils.valid_url?(:atom)
+ refute Utils.valid_url?([])
+ end
+
+ @tag :integration
+ test "returns true for all IANA TLDs" do
+ %{status_code: 200, body: body} =
+ HTTPoison.get!("https://data.iana.org/TLD/tlds-alpha-by-domain.txt")
+
+ tlds =
+ body
+ |> String.split("\n", trim: true)
+ |> Enum.reject(&String.starts_with?(&1, "#"))
+
+ assert length(tlds) > 0
+
+ for tld <- tlds do
+ url = "http://example.#{String.downcase(tld)}"
+ assert Utils.valid_url?(url), "Expected #{url} to be valid"
+ end
+ end
+ end
end