feat(org-detection): org detection agentic workflow

[ekremney]

Org Detection Agentic Workflow

This PR introduces a langchainjs based agent that detects the IMS organization for a site based on its domain and github login (retrieved from the x-fw-host header during site detection).

The agent works by gathering information step by step. It decides which tools to use based on the data it has and keeps collecting more data if needed. It stops only when it finds enough information to decide the correct IMS organization.

How the Agent Works

Decision-Making Process

• The agent starts by using the footer_retriever tool to extract content from the site’s <footer>. It scans the content for short phrases that might represent a company name and checks these candidates with the company_matcher tool. If a match is found, the agent finalizes its result with the matched organization.
• If no match is found in the footer, the agent retrieves the GitHub organization name using the github_org_name_retriever tool. It then checks this name with the company_matcher. If a match is identified, it finalizes the result.
• If neither the footer nor the GitHub details yield a match, the agent uses the link_extractor tool to gather all links from the site’s HTML. It then identifies which links (ie /about-us or /contact-us) are likely to lead to pages containing the company name. Based on this reasoning, the agent retrieves the main content of those pages using the main_content_retriever and checks it with the company_matcher. If a match is found during this step, the agent finalizes the result.
• If all these steps fail, the agent concludes that no organization could be identified.

Tools Available to the Agent

1. Footer Retriever
   An adapter for the Spacecat Content Scraper. It retrieves HTML content from the <footer> element of a website, helping the agent identify potential company names.
2. Company Matcher
   An adapter for the Spacecat Data Access layer. It matches the agent’s guesses (potential company names) against organizations stored in the Spacecat database using fuzzy matching.
3. GitHub Org Name Retriever
   Retrieves the GitHub organization name for a given login.
4. Main Content Retriever
   Similar to the Footer Retriever, this tool extracts text content from the <main> element of a page.
5. Link Extractor
   Extracts all links from raw HTML and converts them to absolute URLs for further analysis.

Full Org-Detection Flow

The agent kicks-in after site detection approval and follows the steps:

1. Checks if the detected site belongs to a customer (does not proceed for friend and family).
2. If a potential IMS organization is detected:
   • Sends a Slack message to announce org detection: Detected IMS organization ORG NAME with IMS org ID XXX@AdobeOrg for domain.com. Would you approve? @user.
3. Based on user feedback:
   • If approved: The detected imsOrg is assigned to the site's entity in the database.
   • If rejected: A follow-up Slack message is sent asking users to set the organization manually using a Slack command:

     @spacecat set imsorg [url] [imsOrgId]. Example:
     @spacecat set imsorg spacecat.com 000000000000000000000000@AdobeOrg
     

   • The backend fetches or creates the organization details, then assigns it to the site.

Successful flow

Unsuccessful flow

[github-actions]

This PR will trigger a minor release when merged.

[solaris007]

good stuff

some thoughts for the future:

• workflow should be configured declaratively
• same for tools
• prompts should not be in js code but rather static / declarative

[solaris007]

🎉 This PR is included in version 1.87.0 🎉

The release is available on:

• v1.87.0
• GitHub release

Your semantic-release bot 📦🚀