:bug: Your search has been filtered.

[guillaumesoucy94]

Description

Hello,

After having successfully installed Websurfx, searches request returns me this following error message:

Your search - test 555 - has been filtered.

Dear user,

All the search results contain results that has been configured to be filtered out via server configuration and henceforth has been completely filtered out.

In config.lua I do have safe_search = 0 enabled and also production_use = true

Please not that sometime the first search works but the subsequent ones fails.

Guillaume

Screenshots

No response

Do you want to work on this issue?

None

Additional information

No response

[github-actions]

To reduce notifications, issues are locked until they are 🏁 status: ready for dev and to be assigned. You can learn more in our contributing guide https://github.com/neon-mmd/websurfx/blob/rolling/CONTRIBUTING.md

[github-actions]

The issue has been unlocked and is now ready for dev. If you would like to work on this issue, you can comment to have it assigned to you. You can learn more in our contributing guide https://github.com/neon-mmd/websurfx/blob/rolling/CONTRIBUTING.md

[neon-mmd]

Sorry for the delay in the reply.

Thanks ❤️ for taking the time to report this issue as this really helps improve the project. 🙂

I think this issue may be caused by two areas in the code I suspect either this:

websurfx/src/templates/views/search.rs

Lines 49 to 118 in c6c967f

	@else if search_results.disallowed{
	.result_disallowed{
	.description{
	p{
	"Your search - "{span class="user_query"{(query)}}" -
	has been disallowed."
	}
	p class="description_paragraph"{"Dear user,"}
	p class="description_paragraph"{
	"The query - "{span class="user_query"{(query)}}" - has
	been blacklisted via server configuration and hence disallowed by the
	server. Henceforth no results could be displayed for your query."
	}
	}
	img src="./images/barricade.png" alt="Image of a Barricade";
	}
	}
	@else if search_results.filtered {
	.result_filtered{
	.description{
	p{
	"Your search - "{span class="user_query"{(query)}}" -
	has been filtered."
	}
	p class="description_paragraph"{"Dear user,"}
	p class="description_paragraph"{
	"All the search results contain results that has been configured to be
	filtered out via server configuration and henceforth has been
	completely filtered out."
	}
	}
	img src="./images/filter.png" alt="Image of a paper inside a funnel";
	}
	}
	@else if search_results.no_engines_selected {
	.result_engine_not_selected{
	.description{
	p{
	"No results could be fetched for your search '{span class="user_query"{(query)}}'."
	}
	p class="description_paragraph"{"Dear user,"}
	p class="description_paragraph"{
	"No results could be retrieved from the upstream search engines as no
	upstream search engines were selected from the settings page."
	}
	}
	img src="./images/no_selection.png" alt="Image of a white cross inside a red circle";
	}
	}
	@else {
	.result_not_found {
	p{"Your search - "{(query)}" - did not match any documents."}
	p class="suggestions"{"Suggestions:"}
	ul{
	li{"Make sure that all words are spelled correctly."}
	li{"Try different keywords."}
	li{"Try more general keywords."}
	}
	img src="./images/no_results.gif" alt="Man fishing gif";
	}
	}
	}
	.page_navigation {
	a href=(format!("/search?q={}&safesearch={}&page={}", query, search_results.safe_search_level, if page > 1 {page-1} else {1})) {
	(PreEscaped("←")) "previous"
	}
	a href=(format!("/search?q={}&safesearch={}&page={}", query, search_results.safe_search_level, page+2)) {
	"next" (PreEscaped("→"))}
	}
	}

Or maybe this:

websurfx/src/results/aggregator.rs

Lines 218 to 250 in c6c967f

	pub async fn filter_with_lists(
	map_to_be_filtered: &mut Vec<(String, SearchResult)>,
	resultant_map: &mut Vec<(String, SearchResult)>,
	file_path: &str,
	) -> Result<(), Box<dyn std::error::Error>> {
	let reader = BufReader::new(File::open(file_path).await?);
	let mut lines = reader.lines();
	while let Some(line) = lines.next_line().await? {
	let re = Regex::new(line.trim())?;
	let mut length = map_to_be_filtered.len();
	let mut idx: usize = Default::default();
	// Iterate over each search result in the map and check if it matches the regex pattern
	while idx < length {
	let ele = &map_to_be_filtered[idx];
	let ele_inner = &ele.1;
	match re.is_match(&ele.0.to_lowercase())
	|| re.is_match(&ele_inner.title.to_lowercase())
	|| re.is_match(&ele_inner.description.to_lowercase())
	{
	true => {
	// If the search result matches the regex pattern, move it from the original map to the resultant map
	resultant_map.push(map_to_be_filtered.swap_remove(idx));
	length -= 1;
	}
	false => idx += 1,
	};
	}
	}
	Ok(())
	}

Though I suspect more the second one because I have tested it and I find that when the file has something within it like word for blacklisting it works fine. So I think maybe regex maybe causing the issue. Also, many projects also have reported that improper regexing could be an issue like this one from pihole where they mention that adblock block list style (which btw is regex based) caused issues with blocking.

@spencerjibz What do you think? Could you investigate into this further?

[spencerjibz]

Sure, @neon-mind, I'll look into it at some point.

[neon-mmd]

I tried to reproduce this and I found that it only affects the release builds. Additionally when I replaced the FuturesUnordered in the PR #673 . I found that it got resolved. So seems like it might have been due to a bug in FuturesUnordered 🤷 . But still I would like to have a confirmation on this one if this was the case. 🙂