Add support for Text fragment feature (#1545) #1600

thiru-appitap · 2024-12-25T08:15:03Z

Text Fragment feature implementation pull request. The feature follows the published URL Fragment Text Directives specification (https://wicg.github.io/scroll-to-text-fragment/).

lychee -vv --include-text-fragments  https://developer.mozilla.org/en-US/docs/Web/URI/Fragment/Text_fragments

[DEBUG] tdirective: "text=From%20the%20foregoing%20remarks%20we%20may%20gather%20an%20idea%20of%20the%20importance
[DEBUG] status: Completed
[DEBUG] result: "From the foregoing remarks we may gather an idea of the importance"
[200] https://mdn.github.io/css-examples/target-text/index.html#:~:text=From%20the%20foregoing%20remarks%20we%20may%20gather%20an%20idea%20of%20the%20importance
[DEBUG] tdirective: "text=linked%20URL,-'s%20format"
[DEBUG] status: Completed
[DEBUG] result: "linked URL"
[DEBUG] tdirective: "text=Deprecated-,attributes,attribute"
[DEBUG] status: Completed
[DEBUG] result: "attributes     charset Deprecated   Hinted at the character encoding of the linked URL.   Note:This attribute"
[200] https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a#:~:text=linked%20URL,-'s%20format&text=Deprecated-,attributes,attribute
[DEBUG] tdirective: "text=downgrade:-,The%20Referer,be%20sent,-to%20origins"
[DEBUG] status: Completed
[DEBUG] result: "The Referer header will not be sent"
[200] https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a#:~:text=downgrade:-,The%20Referer,be%20sent,-to%20origins
[DEBUG] tdirective: "text=linked%20URL,defining%20a%20value"
[DEBUG] status: Completed
[DEBUG] result: "linked URL as a download. Can be used with or without a filename value:    Without a value, the browser will suggest a filename/extension, generated from various sources:   The Content-Disposition HTTP header  The final segment in the URL path  The media type (from the Content-Type header, the start of a data: URL, or Blob.type for a blob: URL)     filename: defining a value"
[200] https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a#:~:text=linked%20URL,defining%20a%20value

If the fragment directive is not found, a TextDirectiveNotFound error will be returned.

Below changes are completed:

Fragment Directive parser uses fancy-regex

this package was added as a dependency

a new flag, include-text-fragments is added to support the feature

this is a deviation from the original feature request (which asked for using the text-fragments flag itself)

Fragment (Text) Directive feature is tested on LTR sites only
new UrlExt trait is implemented to enhance Url's to support Fragment Directive
Support for multiple text fragment directives (for example, #:~:text=linked%20URL,-'s%20format&text=Deprecated-,attributes,attribute)
tests are added for validating the feature
cargo clippy & cargo tests were executed

thiru-appitap · 2024-12-25T09:09:00Z

I missed to run the clippy across the test modules - the related lint failure issues are now fixed and ready for review!

mre · 2024-12-25T11:24:31Z

lychee-lib/src/utils/url.rs

+        );
+        match url {
+            Ok(url) => {
+                eprintln!(


This could be an assertion

Modified the tests to have asserts (instead of the manual checks that was earlier)

mre · 2025-01-10T13:47:09Z

lychee-lib/src/checker/website.rs

+                let mut status = Status::new(&response, self.accepted.clone());
+                if self.validate_text_fragments && has_fragment_directive {
+                    if let Ok(res) = response.text().await {
+                        info!("checking fragment directive...");
+                        if let Some(fd) = req_url.fragment_directive() {
+                            info!("directive: {:?}", fd.text_directives);
+                            match fd.check(&res) {
+                                Ok(stat) => {
+                                    status = stat;
+                                }
+                                Err(e) => {
+                                    return e.into();
+                                }
+                            }
+                        }
+                    }
+                }
+                status


Maybe we can move that into a function/method?

some tests for that part would also be nice

the code is modified to move this into a separate function - as well, bulk of the logic is abstracted at the textfrag crate itself so that the websitechecker will deal with only error responses from the text fragment checker function.

mre · 2025-01-10T13:47:53Z

lychee-lib/src/client.rs

+        assert!(res.status().is_success());
+
+        // start with suffix
+        println!("\ntesting start with suffix...");


you'll probably remove the println!s right?

yes - the println's are removed now

mre · 2025-01-10T13:48:47Z

lychee-lib/src/extract/html/fragdirtok.rs

+
+use crate::types::TextDirective;
+
+const BLOCK_ELEMENTS: &[&str] = &[


This is a long list. Does that mean we'd have to maintain the HTML keywords here? Maybe we can avoid that as it would be an uphill battle.

I am yet to explore alternative approach to this and am open for suggestion - for now, this list is retained (reserving this change for a near future commit)

mre · 2025-01-10T13:50:24Z

lychee-lib/src/extract/html/fragdirtok.rs

I like that this functionality is isolated in its own module. But it's a looot of code. 😅 Not sure what to do here, but at least the ratio of code/tests could be improved.

Perhaps we could move it into a separate crate or use an upstream crate for that? I think it would be a nice library to maintain individually as more applications could profit from it

text fragment is now moved into its own crate, textfrag, in the lychee main tree itself - as I am new to the ecosystem, I'll need help in moving this into a separate crate - please suggest!

mre · 2025-01-10T13:50:54Z

lychee-lib/src/extract/html/fragdirtok.rs

+        let mut all_directives_found = false;
+        let directive = td.directive.borrow();
+
+        'directive_loop: while !all_directives_found {


The labels make it quite hard to read. Have you considered any alternatives?

yes - the code logic was cumbersome and I've rewritten this now - please share your comments!

mre · 2025-01-10T13:53:19Z

lychee-lib/src/types/text_fragment.rs

+pub(crate) const FRAGMENT_DIRECTIVE_DELIMITER: &str = ":~:";
+pub(crate) const TEXT_DIRECTIVE_DELIMITER: &str = "text=";
+
+pub(crate) const TEXT_DIRECTIVE_REGEX: &str = r"(?s)^text=(?:\s*(?P<prefix>[^,&-]*)-\s*[,$]?\s*)?(?:\s*(?P<start>[^-&,]*)\s*)(?:\s*,\s*(?P<end>[^,&-]*)\s*)?(?:\s*,\s*-(?P<suffix>[^,&-]*)\s*)?$";


Where does that regex come from? Did you write it yourself? If there's an "official" regex for those text fragments, we could perhaps add a link to the reference.

this regex was built by myself - I started searching for any equivalent but couldn't land on one when I started this implementation - I am open to revisit this.

In fact, i want to replace regex with a simple parser for this requirement - the specification is not particular about the order of the directives, whereas the regex assumes the order imperatively - with a parser, we might be able to get away from the ordering constraints.

mre · 2025-01-10T13:54:17Z

lychee-lib/src/utils/url.rs

@@ -23,10 +26,61 @@ pub(crate) fn find_links(input: &str) -> impl Iterator<Item = linkify::Link> {
    LINK_FINDER.links(input)
 }

+/// Fragment Directive feature trait
+/// we will use the extension trait pattern to extend the Url to support Text Fragment feature


Thank you - this has now moved into textfrag crate

mre · 2025-01-10T13:55:12Z

lychee-lib/src/utils/url.rs

@@ -23,10 +26,61 @@ pub(crate) fn find_links(input: &str) -> impl Iterator<Item = linkify::Link> {
    LINK_FINDER.links(input)
 }

+/// Fragment Directive feature trait
+/// we will use the extension trait pattern to extend the Url to support Text Fragment feature


Suggested change

/// we will use the extension trait pattern to extend the Url to support Text Fragment feature

/// We will use the extension trait pattern to extend [`url::Url`] to support the text fragment feature

Incorporated this comment in to the textfrag::utils::url file

mre · 2025-01-10T13:55:55Z

lychee-lib/src/utils/url.rs

+/// Fragment Directive feature trait
+/// we will use the extension trait pattern to extend the Url to support Text Fragment feature
+pub(crate) trait UrlExt {
+    /// Checks if the url has a fragment and if the fragment is has the fragment directive delimiter embedded


Suggested change

/// Checks if the url has a fragment and if the fragment is has the fragment directive delimiter embedded

/// Checks if the url has a fragment and if the fragment has the fragment directive delimiter embedded

Updated the comments section (with slight rewording) in the textfrag crate

mre · 2025-01-10T13:56:27Z

lychee-lib/src/utils/url.rs

+}
+
+impl UrlExt for Url {
+    /// Returns whether the URL has fragment directive or not


Suggested change

/// Returns whether the URL has fragment directive or not

/// Checks whether the URL has fragment directive or not

Have addressed this is in the textfrag::utils::url[:16]

mre

Left some comments. I like the overall structure. Good work so far!

thiru-appitap · 2025-01-12T21:45:25Z

Left some comments. I like the overall structure. Good work so far!

@mre
I'll take a look at each of the comments and work to address it - thank you!

mre · 2025-02-05T09:26:33Z

@thiru, any updates? Let me know in case you need any help. 😃

thiru-appitap · 2025-02-05T22:05:09Z

@thiru, any updates? Let me know in case you need any help. 😃

@mre, I will submit the fixes by this weekend for your review. Earlier I started moving the feature into a separate crate, while addressing your review comments, but got pulled into few other tasks and so couldn't get back earlier :-(.

mre · 2025-02-06T01:36:50Z

Thanks, sounds good!

thiru-appitap · 2025-02-12T23:34:54Z

Thanks, sounds good!

@mre my apologies for the delay - while refactoring, into a separate crate, I encountered few corner case issues and it took more time than planned - running the tests now and hoping to re-raise the pull-request by tomorrow!

thiru-appitap · 2025-02-14T04:15:27Z

I am committing the changes into my fork and will initiate a pull-request shortly!

lib's website checker continues to have the logic to validate text fragments clean-up of the tests were done review feedback incorporated - addressed structural, logic, tests and document comments

thiru-appitap · 2025-02-14T04:23:07Z

@mre request your help in addressing the CI / publish-check failure - please recommend if I've to make any changes on my end - thank you!

mre · 2025-02-14T20:42:12Z

lychee-lib/src/checker/website.rs

@@ -85,10 +94,51 @@ impl WebsiteChecker {
        status
    }

+    fn check_text_fragments(site_data: &str, url: &Url, mut status: Status) -> Status {
+        let res = check_text_fragments(site_data, url);
+        if res.is_err() {


You don't need the if condition here. You're matching on res.err() below, and in the case where res.err() is not set, it will be None. This case is already covered in your _res match arm since _res is just a placeholder, which also includes the value being None.

Yes - I've updated the code

mre · 2025-02-14T21:03:18Z