Releases: swirlai/swirl-search
Swirl Metasearch 2.0.2
Swirl Metasearch 2.0.2
Swirl Metasearch 2.0.2 adds support for the X-Api-Key authentication method to SearchProviders.
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
New Features
🔹 The X-Api-Key authentication method is now supported in RequestsGet and RequestsPost SearchProviders.
Known Issues
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.
Please report any issues with this to support.
Upgrading
Documentation Wiki
🔹 Quick Start
🔹 User Guide
🔹 Admin Guide
🔹 M365 Guide
🔹 Developer Guide
🔹 Developer Reference
Support
🔹 Join the Swirl Metasearch Community on Slack!
🔹 Create a GitHub Issue if something doesn't work, isn't clear, or should be documented
🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!
Swirl Metasearch 2.0.1
Swirl Metasearch 2.0.1
Swirl Metasearch 2.0.1 improves default relevancy ranking, especially for some M365 SearchProvider results.
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
Improvements
🔹 Default relevancy weights and similarity thresholds were adjusted to improve overall relevancy ranking, especially for M365 sources.
Known Issues
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.
Please report any issues with this to support.
Upgrading
Documentation Wiki
🔹 Quick Start
🔹 User Guide
🔹 Admin Guide
🔹 M365 Guide
🔹 Developer Guide
🔹 Developer Reference
Support
🔹 Join the Swirl Metasearch Community on Slack!
🔹 Create a GitHub Issue if something doesn't work, isn't clear, or should be documented
🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!
Swirl Metasearch 2.0
Swirl Metasearch 2.0
We are proud to annouce Swirl Metaseearch 2.0 - a major new version featuring enterprise-grade OAUTH2 support plus pre-configured connectors to Microsoft 365 Outlook, OneDrive, Teams and Sharepoint! Using the updated Spyglass UI, knowledge workers can systematically review the best results from their enterprise services, including search engines like Elastic, Solr, AWS OpenSearch, databases like PostgreSQL and Google BigQuery, plus online services like Northern Light Research and Miro.com.
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
New Features
🔹 Microsoft 365 Integration
This update of Spyglass includes a clean new layout, a live source facet, and the option to view results by relevancy, date or the top pick from each source. Swirl 2.0 can also be quickly and securely integrated with M365 as described in the new M365 Guide.
🔹 Query Transformations
Swirl now includes built in Query Transformations, including re-writing and replacement of matching terms. Full details are in the updated Developer's Guide.
🔹 Separate Blocks for Specific SearchProvider Results
ChatGPT's commentary on each query can now be presented separately on the results page.
🔹 New Options for SearchProvider result_mappings
Swirl now supports the FILE_SYSTEM directive, which causes the CosineRelevancyPostResultProcessor to boost matches in body when there are no matches in title. This is the default for Microsoft OneDrive.
Swirl also supports the BLOCK=some-block-name directive. This is configured by default for ChatGPT, as noted above.
Improvements
🔹 Simplified PAYLOAD field display - Swirl now displays just the right-most JSON key for maximum readability
🔹 Improved handling of Unix timestamp variations from sources
🔹 Refined handling of nested source response in JSON
Resolved Issues
🔹 Fixed issue with NOT searches being incorrectly handled in some cases
🔹 Addressed issues in Subscribe functionality
🔹 Hit highlighting: SWIRL adds a space before matching terms next to a comma #119
🔹 Removed single-processor properties
Known Issues
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.
Please report any issues with this to support.
Upgrading
Documentation Wiki
🔹 Quick Start
🔹 User Guide
🔹 Admin Guide
🔹 M365 Guide
🔹 Developer Guide
🔹 Developer Reference
Support
🔹 Join the Swirl Metasearch Community on Slack!
🔹 Create a GitHub Issue if something doesn't work, isn't clear, or should be documented
🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!
Swirl Metasearch 1.10.1
SWIRL SEARCH 1.10.1
This version adds SearchProviders for Atlassian's Confluence and Jira Cloud services.
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
New Features
🔹 New Atlassian SearchProviders allow searching through Confluence and Jira content.
Here's the new Confluence provider JSON:
{
"name": "Wiki (web/Atlassian.com)",
"active": false,
"default": false,
"connector": "RequestsGet",
"url": "https://<your-domain>.atlassian.net/wiki/rest/api/content/search?expand=history,space,metadata.labels.results,children.attachment,children.page,children.comment,ancestors,body.view",
"query_template": "{url}&cql=text~'{query_string}'",
"query_processor": "",
"query_processors": [
"AdaptiveQueryProcessor"
],
"query_mappings": "",
"result_processor": "",
"result_processors": [
"MappingResultProcessor"
],
"response_mappings": "FOUND=size,RESULTS=results",
"result_mappings": "title=title,body=body[*].view[*].value,author=history[*].createdBy[*].publicName,date_published=history[*].createdDate,url='https://<your-domain>.atlassian.net/wiki{_links[*].webui}',space.name,metadata[*].labels[*].results[*].name,children[*].attachment[*].results[*].title,children[*].page[*].results[*].title,ancestors[*].title,NO_PAYLOAD",
"results_per_query": 10,
"credentials": "HTTPBasicAuth('<your-username>','<your-atlassian-token>')",
"tags": [
"Confluence",
"Articles",
"Wiki",
"Atlassian"
]
}
Here is an example result in JSON:
{
"swirl_rank": 3,
"swirl_score": 86.49796345374605,
"searchprovider": "Wiki (web/Atlassian.com)",
"searchprovider_rank": 2,
"title": "Testing Space",
"url": "https://testingbw.atlassian.net/wiki/spaces/TS/overview",
"body": "Welcome to your team space! We've added some suggestions and placeholders. Everything is customizable. Get started with page templates: Template - Project plan Template - <em>Meeting</em> <em>notes</em> Template - Weekly status report Check out Get the most out of your team space for more tips. About Chat GPT Confluence: Confluence is a collaboration tool that enables teams to create, organize, and share their knowledge. It provides features such as document collaboration, team communication, and content management. Mission and vision What is your team's mission? What is your vision? <em>Meet</em> the team Add team members to your space. Team member Role Responsibility Team member Role Responsibility Team member Role Responsibility Contact us How can someone reach out to your team? [email protected] Tickets Jira board #channel Important Pages List them here Onboarding FAQs Add resources for new hires <em>Meeting</em> <em>notes</em> Add links to <em>meeting</em> <em>notes</em> Team goals List them here Team news Create a blog post to share team news. It will automatically appear here once it's published. Blog stream Create a blog post to share news and announcements with your team and company. Create blog post",
"date_published": "2023-03-28 16:54:28.874000+00:00",
"date_retrieved": "2023-04-14 11:13:38.740897",
"author": "Jason Temple",
"title_hit_highlights": [],
"body_hit_highlights": [],
"payload": {
"space.name": "Testing Space",
"children[*].attachment[*].results[*].title": [
"astronaut.svg",
"ledger.svg",
"raised_hand.svg",
"claudia.svg",
"gael.svg",
"angie.svg",
"header_team_space.svg"
],
"children[*].page[*].results[*].title": [
"Template - Project plan",
"Template - Meeting notes",
"Template - Weekly status report",
"Get the most out of your team space"
]
}
}
Here's another result in the Spyglass UI:
Resolved Issues
🔹 Result mappings now support underscore values within curly brace notation
🔹 Swirl results now aggregate JSON lists into strings correctly
🔹 Subscribe now correctly handles the relocated Results hits clause
Known Issues
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch
Please report any issues with this to support.
Upgrading
Documentation Wiki
🔹 Quick Start
🔹 User Guide
🔹 Developer Guide
🔹 Admin Guide
Support
🔹 Create an Issue if something doesn't work, isn't clear, or should be documented
🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!
Swirl Metasearch 1.10
SWIRL SEARCH 1.10
This version adds Querqy integration, improved and configurable hit highlighting, plus YouTrack SearchProviders and support for the open source Spyglass UI!
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
New Features
🔹 Swirl now incorporates feedback from Querqy including synonyms and other rewrites.
The terms are highlighted and used in relevancy.
🔹 Highlighting has been improved and supports tags as well as character markings. The default, specified in swirl_server/settings.py, is:
SWIRL_HIGHLIGHT_START_CHAR = '<em>'
SWIRL_HIGHLIGHT_END_CHAR = '</em>'
🔹 New YouTrack SearchProviders allows searching through Issues and Articles.
Here's the Issues provider JSON:
{
"name": "Issues (web/YouTrack.cloud)",
"connector": "RequestsGet",
"url": "https://<your-domain>.youtrack.cloud/api/issues?fields=project(name),idReadable,tags(name),summary,description,reporter(fullName),created,votes,commentsCount,comments(text),subtasks(issues(idReadable)),customFields(name,value(name))",
"query_template": "{url}&query={query_string}",
"query_processors": [
"AdaptiveQueryProcessor"
],
"result_processors": [
"MappingResultProcessor"
],
"result_mappings": "title=summary,body=description,date_published=created,author=reporter.fullName,url='https://<your-domain>.youtrack.cloud/issue/{idReadable}',project.name,customFields[3].value[*].name,customFields[0].value[*].name,customFields[2].value[*].name,customFields[6].value[*].name,votes,tags[*].name,commentsCount,comments[*].text,subtasks[*].issues[*].idReadable,NO_PAYLOAD",
"results_per_query": 10,
"credentials": "bearer=your-youtrack-permanent-token",
"tags": [
"Issues",
"YouTrack"
]
}
Here is an example result in JSON:
{
"swirl_rank": 5,
"searchprovider": "Issues (web/YouTrack.cloud)",
"searchprovider_rank": 3,
"title": "Bad <em>Relevancy</em> link in User Guide",
"url": "https://swirl.youtrack.cloud/issue/SP-1",
"body": "User Guide in '<em>Relevancy</em>' section shows link to: for 'Developers Guide, <em>Relevancy</em> Section' but it does not go anywhere.\n\n\n\n{width=70%}",
"date_published": "2023-03-22 11:35:30.956000",
"date_retrieved": "2023-03-28 17:56:41.043101",
"author": "Lee Eichelberger",
"title_hit_highlights": [],
"body_hit_highlights": [],
"payload": {
"project.name": "Swirl - Product",
"customFields[3].value[*].name": "Sid Probstein",
"customFields[0].value[*].name": "Normal",
"customFields[2].value[*].name": "Fixed",
"commentsCount": 1,
"comments[*].text": "Fixed by Sid. \n\nCommit: 1ffe987771e76643308ed2f97e1c0544be510ed8"
}
...etc...
Resolved Issues
🔹 BigQuery connector error with single result_mapping that is type date #146
🔹 Review parsing of credentials and tokens everywhere #159
🔹 Hit highlighting: no highlighting of title terms with adjacent punctuation #77
🔹 Swirl now handles Unix-style dates in SearchProvider responses
🔹 RequestsGet no longer requires a RESULTS entry in the response_mapping
Known Issues
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch
Please report any issues with this to support.
Upgrading
Documentation Wiki
🔹 Quick Start
🔹 User Guide
🔹 Developer Guide
🔹 Admin Guide
Support
🔹 Create an Issue if something doesn't work, isn't clear, or should be documented
🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!
Swirl Metasearch 1.10 RC1
Swirl Metasearch 1.10:
- Integrates Querqy feedback from Elastic, Solr, OpenSearch into relevancy and highlighting
- Improves hit highlighting, and now supports configurable highlighting character(s) or tags with default term
- Resolves issues with hit highlighting, BigQuery, providers that don't specify RESULTS and more
- Adds YouTrack SearchProviders for Issues and Articles
- Adds Optional Spyglass UI integration
SWIRL SEARCH 1.9
SWIRL SEARCH 1.9
This version adds a ChatGPT connector, SearchProvider for Miro.com and more!
PLEASE STAR OUR REPO: http://swirl.today/
New Features
🔹 OpenAI ChatGPT Connector & SearchProvider:
The Connector rewrites keyword queries into question form, and answers them. It doesn't alter queries that are already questions, as shown above. ChatGPT responses are usually relevant, regardless of accuracy.
🔹 A set of experimental ChatGPT Processors can be used to rewrite queries as questions, booleans and more.
- ChatGPTQueryImproverProcessor - writes a better query
- ChatGPTQueryExpanderProcessor - writes a broader query using top 5 concepts
- ChatGPTQueryBooleanProcessor - writes a boolean + phrases; 3+ terms works best
🔹 New Miro SearchProvider allows searching drawings.
{
"name": "Boards (web/Miro.com)",
"connector": "RequestsGet",
"url": "https://api.miro.com/v2/boards",
"query_template": "{url}?query={query_string}",
"query_processors": [
"AdaptiveQueryProcessor"
],
"query_mappings": "",
"result_processors": [
"MappingResultProcessor"
],
"response_mappings": "FOUND=total,RESULTS=data",
"result_mappings": "title=name,body=description,url=viewLink,date_published=modifiedAt,owner.name,picture.imageURL,NO_PAYLOAD",
"credentials": "bearer=your-miro-api-token",
"tags": [
"Drawings"
]
}
Changes
🔹 Starting queries with a SearchProvider tag now limits the search to just providers with that tag.
For example:
company:tesla
With the funding dataset installed, this query will return records like:
{
"swirl_rank": 1,
"swirl_score": 438.9910228828939,
"searchprovider": "Company Funding Records (cloud/BigQuery)",
"searchprovider_rank": 1,
"title": "*Tesla* Motors",
"url": "tesla-motors",
"body": "*Tesla* Motors raised $40000000 series c on 2006-05-01. The company is located in San Carlos CA and has 270 employees.",
"date_published": "2006-05-01 00:00:00",
"date_retrieved": "2023-02-13 23:34:52.306501",
"author": "",
"payload": {},
"explain": {
"stems": "tesla",
"title": {
"tesla_*": 0.8357298742623626,
"Tesla_0": 0.8357298742623626,
"result_length_adjust": 1.0,
"query_length_adjust": 1.0
},
"body": {
"Tesla_0": 0.7187157993182859,
"result_length_adjust": 1.0,
"query_length_adjust": 1.0
}
}
}
🔹 RequestsGet now supports Bearer Tokens
This is used by the Miro SearchProvider:
"credentials": "bearer=your-api-token-here",
Resolved Issues
🔹 BigQuery connector/SP returns "Error: list index out of range" on zero results #96
Known Issues
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch
Please report any issues with this to support.
Upgrading
Documentation Wiki
🔹 Quick Start
🔹 User Guide
🔹 Developer Guide
🔹 Admin Guide
Support
🔹 Create an Issue if something doesn't work, isn't clear, or should be documented
🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!
SWIRL SEARCH 1.8.2
SWIRL SEARCH 1.8.2
This version upgrades the version of python used in Docker setup.
PLEASE STAR OUR REPO: http://swirl.today/
New Features
🔹 Docker will now run python 3.11.1
FROM python:3.11.1-slim
Known Issues
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch
Please report any issues with this to support.
Upgrading
Documentation Wiki
🔹 Quick Start
🔹 User Guide
🔹 Developer Guide
🔹 Admin Guide
Support
🔹 Create an Issue if something doesn't work, isn't clear, or should be documented
🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!
SWIRL SEARCH 1.8.1
SWIRL SEARCH 1.8.1
This version resolves issues found in 1.8 and eliminates two installation steps.
PLEASE STAR OUR REPO: http://swirl.today/
New Features
🔹 SWIRL 1.8.1 ships with a SQLite3 database pre-loaded with a super user and the Google PSE examples!
This simplifies all Quick Start procedures dramatically.
Issues Resolved
🔹 Stack2Mixer produces 500 error
Known Issues
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch
Please report any issues with this to support.
Upgrading
Documentation Wiki
🔹 Quick Start
🔹 User Guide
🔹 Developer Guide
🔹 Admin Guide
Support
🔹 Create an Issue if something doesn't work, isn't clear, or should be documented
🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!
SWIRL SEARCH 1.8
SWIRL SEARCH 1.8
This version adds the ability to target specific providers with tag searches (e.g. company:tesla), and the new subscribe feature that causes SWIRL to continuously monitor for new, relevant results - plus a new BigQuery connector!
PLEASE STAR OUR REPO: http://swirl.today/
New Features
🔹 SWIRL now supports targeting of tagged SearchProviders using the tag: prefix.
For example:
electric vehicle company:tesla
The AdaptiveQueryProcessor rewrites this query to electric vehicle tesla for most providers. But providers with the tag company will have it rewritten to just tesla. So this search retrieves typical news results PLUS funding records from BigQuery:
The latter would not have matched the full query. Tag enables expressive querying where specific types of repositories are targeted with appropriate search terms, and SWIRL unifies the results.
🔹 Subscribe to any Search. SWIRL will check for new results every few hours, and automatically detects & discards duplicates by URL or document similarity.
More details: Subscribing to a Search
🔹 New Google BigQuery Connector plus SearchProvider for the Funding Dataset:
{
"name": "Company Funding Records (cloud/BigQuery)",
"connector": "BigQuery",
"query_template": "select {fields} from `{table}` where search({field1}, '{query_string}') or search({field2}, '{query_string}');",
"query_processor": "",
"query_processors": [
"AdaptiveQueryProcessor"
],
"query_mappings": "fields=*,sort_by_date=fundedDate,table=funding.funding,field1=company,field2=city",
"result_processor": "",
"result_processors": [
"MappingResultProcessor"
],
"result_mappings": "title='{company}',body='{company} raised ${raisedamt} series {round} on {fundeddate}. The company is located in {city} {state} and has {numemps} employees.',url=permalink,date_published=fundeddate,NO_PAYLOAD",
"credentials": "/path/to/bigquery/token.json",
"tags": [
"Company",
"BigQuery"
]
}
More details: Google BigQuery Connector
🔹 SWIRL 1.8 supports pipelining of Processors for pre-query, query, result and post-result transformation of queries, responses and results.
For example, the new Search post-result pipeline:
"post_result_processors": [
"DedupeByFieldPostResultProcessor",
"CosineRelevancyPostResultProcessor"
]
More details: Processing Pipelines
🔹 The new DedupeByFieldPostResultProcessor detects and removes duplicates on any field - 'url' by default.
🔹 The new DedupeBySimilarityPostResultProcessor detects and removes duplicates by similarity between - 'title' and 'body' field (by default), with a cut-off threshold of .95.
More details: Detecting and Removing Duplicate Results
Changes
🔹 swirl.py can now be invoked with --debug
python swirl.py --debug restart core
This configuration starts django using the built-in runserver, instead of daphne, and sets the logging level to DEBUG.
Known Issues
🔹 The PostgreSQL Connector no longer causes errors in the celery-worker log if PostgreSQL isn't installed
Please follow the updated installation instructions before attempting to install a SearchProvider that uses the PostgreSQL Connector. We hope to make this easier in a future release.
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch
Please report any issues with this to support.
Upgrading
Documentation Wiki
🔹 Quick Start 🔹 User Guide 🔹 Developer Guide 🔹 Admin Guide
Support
🔹 Join SWIRL SEARCH #support on Slack!
🔹 Create an Issue if something doesn't work, isn't clear, or should be documented
🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!