Update tests, ROADMAP and README#13
Merged
Merged
Conversation
- Add @SPEC to all public functions and @type for Document struct - Improve @moduledoc and @doc across all modules - Fix scrape/3 visibility from def to defp (not part of public API) - Fix text_sitemap_scrape error test that was calling sitemap_scrape - Add 43 new unit tests covering SSL retry, gzip, redirects, edge cases - Add new mock helpers (gzip, SSL errors, 500/403, long redirect chain) - Move live-URL doctests to test/integration_test.exs with @moduletag - Exclude integration tests by default via test_helper.exs - Make fragile doctests resilient to live site content changes - Document unit vs integration test commands in AGENTS.md
README fixes: - Update install version from ~> 0.10 to ~> 1.6 - Remove outdated OTP application config (not needed since Elixir 1.4) - Fix error example using undefined variable (data -> document.data) - Fix contents option using keyword syntax instead of map - Add document.data structure example for page scraping - Show data.locs and data.lines for sitemap scrapers - Reorganize options under a single section ROADMAP.md: - 5-phase plan to port MetaInspector v5.17.1 features to Funkspector - Maps existing features, identifies gaps, and prioritizes additions
There was a problem hiding this comment.
Pull request overview
This PR improves Funkspector’s developer experience by expanding unit test coverage, separating network-dependent doctests into integration tests, and adding clearer module documentation/typespecs across the public API and scrapers.
Changes:
- Expanded unit tests for URL utilities, document loading/requesting, resolver behavior, and sitemap/page scraping edge cases.
- Introduced integration-test tagging/exclusion-by-default and moved live-URL doctests into a dedicated integration test module.
- Added/updated moduledocs and typespecs across core modules; added ROADMAP/agent guidance docs.
Reviewed changes
Copilot reviewed 21 out of 22 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| test/utils_test.exs | Adds focused tests for absolutify/2 and valid_url?/1. |
| test/text_sitemap_scraper_test.exs | Adds edge-case coverage for text sitemap parsing (empty/whitespace, dedupe, trimming). |
| test/sitemap_scraper_test.exs | Adds coverage for empty/absolute/relative/dedup/commented locs. |
| test/page_scraper_test.exs | Adds coverage for no-links, empty-body, dedupe, trimming, anchors, empty href, base href behaviors. |
| test/resolver_test.exs | Adds tests for status-code errors, gzip deflation, TLS retry behavior, redirect hop limiting; tags live regression as integration. |
| test/document_test.exs | Adds coverage for non-2xx handling, redirects preserving original URL, and load-vs-request data shape. |
| test/test_helper.exs | Excludes :integration tests by default. |
| test/integration_test.exs | New integration test module containing live-URL doctests. |
| test/support/mocked_connections.exs | Adds additional mocked responses (gzip, SSL errors, server/403, redirect chain). |
| lib/funkspector/utils.ex | Improves docs and adds typespecs for absolutify/2 and valid_url?/1. |
| lib/funkspector/text_sitemap_scraper.ex | Updates docs and adds typespec for scrape/1. |
| lib/funkspector/sitemap_scraper.ex | Updates docs and adds typespec for scrape/1. |
| lib/funkspector/page_scraper.ex | Updates docs and adds typespec for scrape/1. |
| lib/funkspector/resolver.ex | Updates docs and adds a resolve/2 typespec. |
| lib/funkspector/document.ex | Adds Document.t() type and request/load typespecs; expands docs. |
| lib/funkspector.ex | Expands public API docs, adds typespecs, and makes scrape/3 private. |
| README.md | Updates installation/usage examples and error-handling docs. |
| ROADMAP.md | Adds a feature-parity roadmap document. |
| AGENTS.md / CLAUDE.md | Adds repository guidance for AI agents; CLAUDE points to AGENTS. |
| .gitignore | Ignores .claude. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Replace "https://ejemplo.com.ñ" with "https://ejemplo.ñandú.com" in the IDN test — the original passed due to regex backtracking treating .ñ as a path, not because it validated the domain correctly. Add tests asserting that single-char and non-ASCII TLDs are rejected.
Fetches the official TLD list from IANA and asserts that valid_url?/1 accepts every registered TLD.
The module only runs doctests from modules with live-URL examples, so DocsTest better reflects its purpose.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Improves the test suite, typespecs and documentation.