diff --git a/.github/workflows/website-deploy-prod.yml b/.github/workflows/website-deploy-prod.yml index 361fc70fc7..91b155ceb9 100644 --- a/.github/workflows/website-deploy-prod.yml +++ b/.github/workflows/website-deploy-prod.yml @@ -71,11 +71,14 @@ jobs: aws s3 sync . s3://${{ env.S3_BUCKET_NAME }} --delete aws s3 ls s3://${{ env.S3_BUCKET_NAME }} --recursive --human-readable --summarize - - name: ReIndex with Algolia - env: - ALGOLIA_SCRAPER_API_KEY: ${{ secrets.ALGOLIA_SCRAPER_API_KEY }} - run: | - ./website/algolia/reindex.sh + - name: Trigger Algolia Crawler + uses: algolia/algoliasearch-crawler-github-actions@v1 + with: + crawler-user-id: ${{ secrets.ALGOLIA_CRAWLER_USER_ID }} + crawler-api-key: ${{ secrets.ALGOLIA_CRAWLER_API_KEY }} + algolia-app-id: ${{ env.ALGOLIA_APP_ID }} + algolia-api-key: ${{ secrets.ALGOLIA_API_KEY }} + site-url: 'https://${{ env.DEPLOYMENT_HOST }}' - name: Trigger ReIndexing of atmos.tools and docs.cloudposse.com Docs run: | diff --git a/docs/algolia-search-indexing.md b/docs/algolia-search-indexing.md new file mode 100644 index 0000000000..9297646dae --- /dev/null +++ b/docs/algolia-search-indexing.md @@ -0,0 +1,153 @@ +# Algolia Search Indexing + +This document describes how Algolia search indexing is configured for the atmos.tools documentation site. + +## Overview + +The atmos.tools documentation uses [Algolia DocSearch](https://docsearch.algolia.com/) for search functionality. Search indexing is managed via the **Algolia Crawler**, which is triggered automatically on every deployment. + +## Architecture + +``` +┌─────────────────────┐ ┌──────────────────────┐ ┌─────────────────┐ +│ GitHub Actions │────▶│ Algolia Crawler │────▶│ Algolia Index │ +│ (website-deploy) │ │ (cloud-hosted) │ │ (atmos.tools) │ +└─────────────────────┘ └──────────────────────┘ └─────────────────┘ +``` + +### Components + +1. **GitHub Actions Workflow** (`.github/workflows/website-deploy-prod.yml`) + - Triggers the Algolia Crawler after deploying the website to S3. + - Uses the official `algolia/algoliasearch-crawler-github-actions` action. + +2. **Algolia Crawler** (dashboard.algolia.com) + - Cloud-hosted crawler that fetches and indexes the documentation. + - Uses the official Docusaurus v3 template for optimal indexing. + - Runs on-demand (CI-triggered) and weekly (scheduled backup). + +3. **Docusaurus Frontend** (`website/docusaurus.config.js`) + - Integrates with Algolia via the DocSearch plugin. + - Supports DocSearch v4 with Ask AI integration. + +## Configuration + +### GitHub Secrets Required + +The following secrets must be configured in the GitHub repository: + +| Secret | Description | Source | +|--------|-------------|--------| +| `ALGOLIA_CRAWLER_USER_ID` | Crawler authentication user ID | Algolia Crawler > Account Settings | +| `ALGOLIA_CRAWLER_API_KEY` | Crawler authentication API key | Algolia Crawler > Account Settings | +| `ALGOLIA_API_KEY` | API key for writing to index | Algolia Dashboard > API Keys | + +### API Key Permissions (ACL) + +The `ALGOLIA_API_KEY` must have the following ACL permissions: + +| Permission | Purpose | +|------------|---------| +| `addObject` | Add records to the index | +| `editSettings` | Modify index settings | +| `deleteIndex` | Delete/recreate the index during full reindex | +| `browse` | Optional - needed for partial updates | + +To create or edit an API key with these permissions: + +1. Go to **Algolia Dashboard** → **Settings** → **API Keys** → **All API Keys**. +2. Click **New API Key** (or edit an existing key). +3. Under **ACL**, select the permissions listed above. +4. Under **Indices**, restrict to `atmos.tools` or use `*` for all indices. +5. Save and copy the key to GitHub secrets. + +### Algolia Crawler Configuration + +The crawler is configured in the Algolia dashboard (dashboard.algolia.com) with these settings: + +- **Template**: Docusaurus v3 +- **Start URL**: `https://atmos.tools/` +- **Sitemap URL**: `https://atmos.tools/sitemap.xml` +- **Index Name**: `atmos.tools` +- **Schedule**: Weekly (as backup to CI-triggered crawls) + +### Frontend Configuration + +The Docusaurus frontend is configured in `website/docusaurus.config.js`: + +```javascript +algolia: { + appId: process.env.ALGOLIA_APP_ID || '32YOERUX83', + apiKey: process.env.ALGOLIA_SEARCH_API_KEY || '557985309adf0e4df9dcf3cb29c61928', + indexName: process.env.ALGOLIA_INDEX_NAME || 'atmos.tools', + contextualSearch: false, + askAi: { + assistantId: process.env.ALGOLIA_ASKAI_ASSISTANT_ID || 'xzgtsIXZSf7V', + // ... additional Ask AI config + } +} +``` + +## Triggering a Crawl + +### Automatic (CI/CD) - Primary Method + +Crawls are automatically triggered via the `algolia/algoliasearch-crawler-github-actions` GitHub Action on: +- Push to `main` branch +- Release published +- Manual workflow dispatch + +This is the primary indexing method. The GitHub Action is configured in `.github/workflows/website-deploy-prod.yml`. + +### Manual (Dashboard) + +For debugging or one-off reindexing: + +1. Log into dashboard.algolia.com. +2. Navigate to the Crawler section. +3. Select the `atmos.tools` crawler. +4. Click "Start crawl" or "Restart crawling". + +### Manual (API) + +For scripting or automation outside of GitHub Actions: + +```bash +curl -X POST "https://crawler.algolia.com/api/1/crawlers/{CRAWLER_ID}/reindex" \ + -H "Authorization: Basic $(echo -n '{USER_ID}:{API_KEY}' | base64)" +``` + +## Troubleshooting + +### Search Not Returning Results + +1. **Check index status**: Log into Algolia dashboard and verify the index has records. +2. **Check crawler logs**: Review the crawler run logs for errors. +3. **Verify sitemap**: Ensure `https://atmos.tools/sitemap.xml` is accessible and complete. +4. **Test selectors**: Use the URL Tester in the crawler dashboard to verify content extraction. + +### CI Workflow Failing + +1. **Verify secrets**: Ensure all required GitHub secrets are configured. +2. **Check credentials**: Verify the Crawler User ID and API Key are correct. +3. **Review action logs**: Check the GitHub Actions logs for specific error messages. + +### Low Record Count + +If the index has significantly fewer records than expected: + +1. **Check sitemap**: Verify all pages are included in the sitemap. +2. **Review crawler config**: Ensure the start URL and sitemap URL are correct. +3. **Check for blocked pages**: Review robots.txt for any blocked paths. +4. **Verify page linking**: Ensure all pages are linked from the sitemap or other indexed pages. + +## Migration History + +**January 2026**: Migrated from deprecated `algolia/docsearch-scraper` Docker image to the official Algolia Crawler with GitHub Actions integration. The legacy scraper was deprecated in February 2022 and was causing indexing failures. + +## References + +- [Algolia Crawler Documentation](https://www.algolia.com/doc/tools/crawler/getting-started/overview/) +- [Algolia Crawler GitHub Action](https://github.com/algolia/algoliasearch-crawler-github-actions) +- [DocSearch Documentation](https://docsearch.algolia.com/) +- [Docusaurus Search Documentation](https://docusaurus.io/docs/search) diff --git a/website/algolia/reindex.sh b/website/algolia/reindex.sh deleted file mode 100755 index d155aade9c..0000000000 --- a/website/algolia/reindex.sh +++ /dev/null @@ -1,23 +0,0 @@ -#!/usr/bin/env bash - -ALGOLIA_INDEX_NAME=${ALGOLIA_INDEX_NAME:-test} -DEPLOYMENT_HOST=${DEPLOYMENT_HOST:-'atmos.tools'} -ALGOLIA_APP_ID=${ALGOLIA_APP_ID:-'32YOERUX83'} - -[ -z "$ALGOLIA_SCRAPER_API_KEY" ] && echo "Need to set ALGOLIA_SCRAPER_API_KEY" && exit 1; - -# prepare algolia config -cat website/algolia/template.json \ - | jq '.index_name="'${ALGOLIA_INDEX_NAME}'"' \ - | jq '.start_urls[0]="https://'${DEPLOYMENT_HOST}'/"' \ - | jq '.sitemap_urls[0]="https://'${DEPLOYMENT_HOST}'/sitemap.xml"' \ - > algolia.index.json - -cat algolia.index.json - -# do actual scraping -docker run \ - --env APPLICATION_ID="${ALGOLIA_APP_ID}" \ - --env API_KEY="${ALGOLIA_SCRAPER_API_KEY}" \ - --env "CONFIG=$(cat algolia.index.json | jq -r tostring)" \ - algolia/docsearch-scraper diff --git a/website/algolia/template.json b/website/algolia/template.json deleted file mode 100644 index 873ac8d053..0000000000 --- a/website/algolia/template.json +++ /dev/null @@ -1,48 +0,0 @@ -{ - "index_name": "atmos.tools", - "start_urls": [ - "https://atmos.tools/" - ], - "sitemap_urls": [ - "https://atmos.tools/sitemap.xml" - ], - "sitemap_alternate_links": true, - "stop_urls": [], - "selectors": { - "lvl0": { - "selector": "(//ul[contains(@class,'menu__list')]//a[contains(@class, 'menu__link menu__link--sublist menu__link--active')]/text() | //nav[contains(@class, 'navbar')]//a[contains(@class, 'navbar__link--active')]/text())[last()]", - "type": "xpath", - "global": true, - "default_value": "Documentation" - }, - "lvl1": "header h1", - "lvl2": "article h2", - "lvl3": "article h3", - "lvl4": "article h4", - "lvl5": "article h5, article td:first-child", - "lvl6": "article h6", - "text": "article p, article li, article td:last-child" - }, - "strip_chars": " .,;:#", - "custom_settings": { - "separatorsToIndex": "_", - "attributesForFaceting": [ - "language", - "version", - "type", - "docusaurus_tag" - ], - "attributesToRetrieve": [ - "hierarchy", - "content", - "anchor", - "url", - "url_without_anchor", - "type" - ] - }, - "conversation_id": [ - "833762294" - ], - "nb_hits": 46251 -} diff --git a/website/blog/2026-01-07-ask-ai-search.mdx b/website/blog/2026-01-07-ask-ai-search.mdx new file mode 100644 index 0000000000..fbbb2066e6 --- /dev/null +++ b/website/blog/2026-01-07-ask-ai-search.mdx @@ -0,0 +1,57 @@ +--- +slug: ask-ai-search +title: "Ask AI: Conversational Search for Atmos Documentation" +authors: + - osterman +tags: + - feature + - documentation +--- + +Finding information in documentation shouldn't require knowing the exact terminology or page structure. With **Ask AI**, you can now ask natural language questions about Atmos and get intelligent, contextual answers—powered by [Algolia DocSearch v4](https://docsearch.algolia.com/docs/v4/askai/) and ChatGPT. + + + +## How It Works + +When you open the search dialog (press `/` or `Cmd+K`), you can seamlessly switch between traditional keyword search and a chat-style assistant that understands your questions. + +**Key features:** + +- **Natural language queries** - Ask questions like "How do I configure AWS SSO authentication?" or "What's the difference between stacks and components?" +- **Contextual answers** - Responses are generated from the actual documentation content, not generic AI knowledge +- **Source citations** - Every answer includes links to the specific documentation pages it drew from +- **Seamless experience** - Switch between keyword search and AI assistant without leaving the search dialog. + +## Under the Hood + +Ask AI uses Algolia's integration with large language models (LLMs) to understand your questions and find relevant information from the indexed documentation. The assistant is specifically trained on Atmos documentation, so it understands: + +- Atmos CLI commands and their flags +- Stack configuration patterns and YAML syntax +- Authentication providers and identity management +- Workflows, custom commands, and extensibility patterns +- Design patterns and best practices + +## Try It Now + +Visit [atmos.tools](https://atmos.tools) and press `/` or `Cmd+K` to open the search dialog. Try asking questions like: + +- "How do I set up AWS SSO with Atmos?" +- "What is the difference between imports and inheritance?" +- "How do I create a custom workflow?" +- "What YAML functions are available?" + +The AI assistant will provide detailed answers with direct links to the relevant documentation pages. + +## Technical Details + +This feature is enabled by upgrading to Docusaurus 3.9.2 and DocSearch v4, which adds native support for Algolia's Ask AI feature. The assistant is configured with an understanding of Atmos as a DevOps and cloud automation tool, ensuring responses are relevant and technically accurate. + +## Get Involved + +Have feedback on Ask AI? We'd love to hear how it's helping you navigate the documentation: + +- [Open an issue](https://github.com/cloudposse/atmos/issues) for bugs or feature requests +- [Join our Slack](https://slack.cloudposse.com/) to discuss with the community +- [Star the repo](https://github.com/cloudposse/atmos) if you find Atmos helpful diff --git a/website/docusaurus.config.js b/website/docusaurus.config.js index 48b775cbf3..ffe19a035e 100644 --- a/website/docusaurus.config.js +++ b/website/docusaurus.config.js @@ -337,7 +337,7 @@ const config = { [ 'docusaurus-plugin-sentry', { - DSN: 'b022344b0e7cc96f803033fff3b377ee@o56155.ingest.us.sentry.io/4507472203087872', + DSN: 'https://b022344b0e7cc96f803033fff3b377ee@o56155.ingest.us.sentry.io/4507472203087872', }, ], [ @@ -496,7 +496,15 @@ const config = { appId: process.env.ALGOLIA_APP_ID || '32YOERUX83', apiKey: process.env.ALGOLIA_SEARCH_API_KEY || '557985309adf0e4df9dcf3cb29c61928', // this is SEARCH ONLY API key and is not sensitive information indexName: process.env.ALGOLIA_INDEX_NAME || 'atmos.tools', - contextualSearch: false + contextualSearch: false, + // DocSearch v4 Ask AI integration + // https://docsearch.algolia.com/docs/v4/askai/ + askAi: { + assistantId: process.env.ALGOLIA_ASKAI_ASSISTANT_ID || 'xzgtsIXZSf7V', + appId: process.env.ALGOLIA_APP_ID || '32YOERUX83', + apiKey: process.env.ALGOLIA_SEARCH_API_KEY || '557985309adf0e4df9dcf3cb29c61928', + indexName: process.env.ALGOLIA_INDEX_NAME || 'atmos.tools', + } }, zoom: { selector: '.markdown :not(em) > img', diff --git a/website/package.json b/website/package.json index 221fc83c9f..08dd19bad4 100644 --- a/website/package.json +++ b/website/package.json @@ -17,13 +17,14 @@ "build:site": "npm run build" }, "dependencies": { - "@docusaurus/core": "^3.6.3", - "@docusaurus/plugin-client-redirects": "^3.6.3", - "@docusaurus/plugin-content-blog": "^3.6.3", - "@docusaurus/plugin-google-tag-manager": "^3.6.3", - "@docusaurus/preset-classic": "^3.6.3", - "@docusaurus/theme-common": "^3.6.3", - "@docusaurus/theme-mermaid": "^3.6.3", + "@docsearch/react": "^4.2.0", + "@docusaurus/core": "^3.9.2", + "@docusaurus/plugin-client-redirects": "^3.9.2", + "@docusaurus/plugin-content-blog": "^3.9.2", + "@docusaurus/plugin-google-tag-manager": "^3.9.2", + "@docusaurus/preset-classic": "^3.9.2", + "@docusaurus/theme-common": "^3.9.2", + "@docusaurus/theme-mermaid": "^3.9.2", "@excalidraw/excalidraw": "^0.17.6", "@fortawesome/fontawesome-svg-core": "^6.7.2", "@fortawesome/free-brands-svg-icons": "^6.7.2", @@ -58,7 +59,7 @@ "unified": "^11.0.5" }, "devDependencies": { - "@docusaurus/module-type-aliases": "^3.6.3", + "@docusaurus/module-type-aliases": "^3.9.2", "prettier": "^3.4.2" }, "browserslist": { diff --git a/website/pnpm-lock.yaml b/website/pnpm-lock.yaml index ab0badb6c7..112674aee6 100644 --- a/website/pnpm-lock.yaml +++ b/website/pnpm-lock.yaml @@ -8,26 +8,29 @@ importers: .: dependencies: + '@docsearch/react': + specifier: ^4.2.0 + version: 4.2.0(@algolia/client-search@5.41.0)(@types/react@19.2.2)(react-dom@18.3.1(react@18.3.1))(react@18.3.1)(search-insights@2.17.3) '@docusaurus/core': - specifier: ^3.6.3 + specifier: ^3.9.2 version: 3.9.2(@mdx-js/react@3.1.1(@types/react@19.2.2)(react@18.3.1))(react-dom@18.3.1(react@18.3.1))(react@18.3.1) '@docusaurus/plugin-client-redirects': - specifier: ^3.6.3 + specifier: ^3.9.2 version: 3.9.2(@mdx-js/react@3.1.1(@types/react@19.2.2)(react@18.3.1))(react-dom@18.3.1(react@18.3.1))(react@18.3.1) '@docusaurus/plugin-content-blog': - specifier: ^3.6.3 + specifier: ^3.9.2 version: 3.9.2(@docusaurus/plugin-content-docs@3.9.2(@mdx-js/react@3.1.1(@types/react@19.2.2)(react@18.3.1))(react-dom@18.3.1(react@18.3.1))(react@18.3.1))(@mdx-js/react@3.1.1(@types/react@19.2.2)(react@18.3.1))(react-dom@18.3.1(react@18.3.1))(react@18.3.1) '@docusaurus/plugin-google-tag-manager': - specifier: ^3.6.3 + specifier: ^3.9.2 version: 3.9.2(@mdx-js/react@3.1.1(@types/react@19.2.2)(react@18.3.1))(react-dom@18.3.1(react@18.3.1))(react@18.3.1) '@docusaurus/preset-classic': - specifier: ^3.6.3 + specifier: ^3.9.2 version: 3.9.2(@algolia/client-search@5.41.0)(@mdx-js/react@3.1.1(@types/react@19.2.2)(react@18.3.1))(@types/react@19.2.2)(react-dom@18.3.1(react@18.3.1))(react@18.3.1)(search-insights@2.17.3) '@docusaurus/theme-common': - specifier: ^3.6.3 + specifier: ^3.9.2 version: 3.9.2(@docusaurus/plugin-content-docs@3.9.2(@mdx-js/react@3.1.1(@types/react@19.2.2)(react@18.3.1))(react-dom@18.3.1(react@18.3.1))(react@18.3.1))(react-dom@18.3.1(react@18.3.1))(react@18.3.1) '@docusaurus/theme-mermaid': - specifier: ^3.6.3 + specifier: ^3.9.2 version: 3.9.2(@docusaurus/plugin-content-docs@3.9.2(@mdx-js/react@3.1.1(@types/react@19.2.2)(react@18.3.1))(react-dom@18.3.1(react@18.3.1))(react@18.3.1))(@mdx-js/react@3.1.1(@types/react@19.2.2)(react@18.3.1))(react-dom@18.3.1(react@18.3.1))(react@18.3.1) '@excalidraw/excalidraw': specifier: ^0.17.6 @@ -127,7 +130,7 @@ importers: version: 11.0.5 devDependencies: '@docusaurus/module-type-aliases': - specifier: ^3.6.3 + specifier: ^3.9.2 version: 3.9.2(react-dom@18.3.1(react@18.3.1))(react@18.3.1) prettier: specifier: ^3.4.2 @@ -8402,7 +8405,7 @@ snapshots: '@types/sax@1.2.7': dependencies: - '@types/node': 17.0.45 + '@types/node': 24.9.1 '@types/send@0.17.5': dependencies: diff --git a/website/src/data/roadmap.js b/website/src/data/roadmap.js index d89fcbae58..4af536bb51 100644 --- a/website/src/data/roadmap.js +++ b/website/src/data/roadmap.js @@ -385,15 +385,16 @@ export const roadmapConfig = { progress: 100, status: 'shipped', milestones: [ + { label: 'Ask AI conversational search', status: 'shipped', quarter: 'q1-2026', changelog: 'ask-ai-search', description: 'Conversational documentation search powered by Algolia DocSearch v4 and ChatGPT. Ask natural language questions and get intelligent, contextual answers.', category: 'featured', priority: 'high', benefits: 'Find answers faster with natural language queries. No need to guess the right keywords.' }, { label: 'Changelog introduction', status: 'shipped', quarter: 'q4-2025', docs: '/changelog', description: 'Detailed changelogs for every release with feature announcements and migration guides.', category: 'featured', priority: 'high', benefits: 'Know exactly what changed in each release. Plan upgrades with full visibility into breaking changes.' }, + { label: 'New learning section', status: 'shipped', quarter: 'q4-2025', docs: '/learn/concepts-overview', changelog: 'documentation-reorganization', description: 'Step-by-step tutorials for getting started with Atmos from scratch.', category: 'featured', priority: 'high', benefits: 'New users have a clear path from zero to productive. Concepts build on each other logically.' }, + { label: 'Added migration guides', status: 'shipped', quarter: 'q4-2025', docs: '/migration/terragrunt', changelog: 'migration-guides', description: 'Detailed guides for migrating from Terragrunt, Terraform workspaces, and native Terraform to Atmos.', category: 'featured', priority: 'high', benefits: 'Migrate from your current tool with step-by-step guidance. Understand the mapping between concepts.' }, + { label: 'Roadmap', status: 'shipped', quarter: 'q4-2025', docs: '/roadmap', changelog: 'product-roadmap', description: 'Public roadmap showing past accomplishments and future plans.', category: 'featured', priority: 'high', benefits: 'See what\'s coming and plan accordingly. Understand the direction of the project.' }, { label: 'Every `atmos.yaml` section documented', status: 'shipped', quarter: 'q3-2025', docs: '/cli/configuration', description: 'Complete reference for every configuration option in atmos.yaml with examples and defaults.', category: 'featured', priority: 'high', benefits: 'Find the exact option you need without reading source code. Every setting has examples.' }, { label: 'Every CLI command documented', status: 'shipped', quarter: 'q3-2025', docs: '/cli/commands', description: 'Comprehensive documentation for every CLI command with usage examples and screenshots.', category: 'featured', priority: 'high', benefits: 'Learn any command from its documentation. Screenshots show exactly what to expect.' }, { label: 'Cross-linking commands to configs', status: 'shipped', quarter: 'q3-2025', docs: '/cli/configuration', description: 'Navigate from any command to its related configuration and vice versa.', category: 'featured', priority: 'high', benefits: 'Jump between commands and their configuration options. Understand how settings affect behavior.' }, { label: 'Design patterns refresh', status: 'shipped', quarter: 'q3-2025', docs: '/design-patterns', description: 'Updated design patterns with real-world examples for common infrastructure scenarios.', category: 'featured', priority: 'high', benefits: 'Apply proven patterns instead of inventing solutions. Learn from real-world examples.' }, { label: 'Versioning strategy docs', status: 'shipped', quarter: 'q3-2025', docs: '/design-patterns/version-management', changelog: 'comprehensive-version-management-documentation', version: 'v1.199.0', description: 'Complete guide to version management including component versioning and upgrade strategies.', category: 'featured', priority: 'high', benefits: 'Upgrade components safely with a clear strategy. Understand version compatibility.' }, - { label: 'New learning section', status: 'shipped', quarter: 'q4-2025', docs: '/learn/concepts-overview', changelog: 'documentation-reorganization', description: 'Step-by-step tutorials for getting started with Atmos from scratch.', category: 'featured', priority: 'high', benefits: 'New users have a clear path from zero to productive. Concepts build on each other logically.' }, - { label: 'Added migration guides', status: 'shipped', quarter: 'q4-2025', docs: '/migration/terragrunt', changelog: 'migration-guides', description: 'Detailed guides for migrating from Terragrunt, Terraform workspaces, and native Terraform to Atmos.', category: 'featured', priority: 'high', benefits: 'Migrate from your current tool with step-by-step guidance. Understand the mapping between concepts.' }, - { label: 'Roadmap', status: 'shipped', quarter: 'q4-2025', docs: '/roadmap', changelog: 'product-roadmap', description: 'Public roadmap showing past accomplishments and future plans.', category: 'featured', priority: 'high', benefits: 'See what\'s coming and plan accordingly. Understand the direction of the project.' }, ], issues: [], prs: [],