feat: Add robots.txt for search engine indexing#1943
Conversation
…erification - Adds robots.txt to website/static/ for maximum search engine indexing - Includes Algolia crawler verification token - Explicitly allows all crawlers with User-agent: * and Allow: / - References sitemap for efficient crawler discovery 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Dependency Review✅ No vulnerabilities or license issues found.Scanned FilesNone |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1943 +/- ##
==========================================
+ Coverage 73.99% 74.01% +0.01%
==========================================
Files 769 769
Lines 69288 69288
==========================================
+ Hits 51273 51281 +8
+ Misses 14604 14597 -7
+ Partials 3411 3410 -1
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
📝 WalkthroughWalkthroughAdds Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Disabled knowledge base sources:
📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
⏰ Context from checks skipped due to timeout of 900000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In @website/static/robots.txt:
- Around line 1-8: robots.txt currently hardcodes
"https://atmos.tools/sitemap.xml", which breaks non-production deployments;
change robots.txt to produce an environment-aware sitemap URL by using the same
DEPLOYMENT_HOST env var used in the reindex script (or fallback to the
production host) so the Sitemap line points to `${DEPLOYMENT_HOST}/sitemap.xml`
in non-prod builds, or alternatively serve a template/SSR robots.txt that
injects DEPLOYMENT_HOST at runtime; update any build or deploy step that writes
website/static/robots.txt to use DEPLOYMENT_HOST accordingly and ensure the
fallback behavior for missing env var.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
website/static/robots.txt
🧰 Additional context used
📓 Path-based instructions (1)
website/**
📄 CodeRabbit inference engine (.cursor/rules/atmos-rules.mdc)
website/**: Update website documentation in thewebsite/directory when adding new features, ensure consistency between CLI help text and website documentation, and follow the website's documentation structure and style
Keep website code in thewebsite/directory, follow the existing website architecture and style, and test website changes locally before committing
Keep CLI documentation and website documentation in sync and document new features on the website with examples and use cases
Files:
website/static/robots.txt
⏰ Context from checks skipped due to timeout of 900000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: Acceptance Tests (linux)
- GitHub Check: Acceptance Tests (windows)
- GitHub Check: Summary
github-actions
bot
added
size/s
osterman
force-pushed
the
osterman/robots-algolia-verif
branch
from
9aad1d3 to
f22a0d7
Compare
osterman
force-pushed
the
osterman/robots-algolia-verif
branch
from
f22a0d7 to
b05d806
Compare
osterman
force-pushed
the
osterman/robots-algolia-verif
branch
from
b05d806 to
9c063fe
Compare
osterman
force-pushed
the
osterman/robots-algolia-verif
branch
from
9c063fe to
d944390
Compare
|
Important Cloud Posse Engineering Team Review RequiredThis pull request modifies files that require Cloud Posse's review. Please be patient, and a core maintainer will review your changes. To expedite this process, reach out to us on Slack in the |
osterman
added
the
no-release
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In @.github/workflows/website-preview-build.yml:
- Line 50: The DEPLOYMENT_HOST environment variable is being set unconditionally
using github.event.pull_request.number which is undefined on workflow_dispatch
runs; change the assignment so it is conditional and empty when there is no
pull_request number (so docusaurus.config.js can fall back). Replace the
existing DEPLOYMENT_HOST line with a conditional expression that yields an empty
string when github.event.pull_request.number is missing, for example using the
GitHub Actions expression: DEPLOYMENT_HOST: ${{ github.event.pull_request.number
&& format('pr-{0}.atmos-docs.ue2.dev.plat.cloudposse.org',
github.event.pull_request.number) }}, ensuring the variable is empty for manual
runs and populated only for PR-triggered runs.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (2)
.github/workflows/website-preview-build.ymlwebsite/docusaurus.config.js
🧰 Additional context used
📓 Path-based instructions (2)
.github/workflows/*.{yml,yaml}
📄 CodeRabbit inference engine (.cursor/rules/atmos-rules.mdc)
Configure CI to run unit tests, integration tests, golangci-lint, and coverage reporting on all pull requests
Files:
.github/workflows/website-preview-build.yml
website/**
📄 CodeRabbit inference engine (.cursor/rules/atmos-rules.mdc)
website/**: Update website documentation in thewebsite/directory when adding new features, ensure consistency between CLI help text and website documentation, and follow the website's documentation structure and style
Keep website code in thewebsite/directory, follow the existing website architecture and style, and test website changes locally before committing
Keep CLI documentation and website documentation in sync and document new features on the website with examples and use cases
Files:
website/docusaurus.config.js
🧠 Learnings (2)
📓 Common learnings
Learnt from: osterman
Repo: cloudposse/atmos PR: 1686
File: docs/prd/tool-dependencies-integration.md:58-64
Timestamp: 2025-12-13T06:07:37.766Z
Learning: cloudposse/atmos: For PRD docs (docs/prd/*.md), markdownlint issues like MD040/MD010/MD034 can be handled in a separate documentation cleanup commit and should not block the current PR.
📚 Learning: 2025-09-30T00:36:22.219Z
Learnt from: aknysh
Repo: cloudposse/atmos PR: 0
File: :0-0
Timestamp: 2025-09-30T00:36:22.219Z
Learning: In the Atmos website project using docusaurus-plugin-llms, the postbuild script intentionally copies llms.txt and llms-full.txt from build/ to static/ (reverse of typical Docusaurus flow). This is necessary because: (1) the plugin hardcodes output to build/ directory, (2) files must be in static/ for deployment and dev mode access, (3) the plugin doesn't support configuring output directory. The files are source-controlled in static/ and regenerated on each build.
Applied to files:
website/docusaurus.config.js
⏰ Context from checks skipped due to timeout of 900000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: Acceptance Tests (windows)
- GitHub Check: Summary
🔇 Additional comments (1)
website/docusaurus.config.js (1)
16-16: DEPLOYMENT_HOST fallback handles environment-aware URLs correctly.The
DEPLOYMENT_HOSTwith fallback toatmos.toolsis solid, and the dynamic URL at line 22 will be picked up by Docusaurus's built-in sitemap generation. Docusaurus 2 generates the sitemap using theurlfield from the config, so the dynamic URL will automatically be used without requiring explicit sitemap plugin configuration.
- Use DEPLOYMENT_HOST env var in docusaurus.config.js url setting - Fallback to atmos.tools (production) when DEPLOYMENT_HOST is not set - Docusaurus automatically generates sitemap with correct URLs - robots.txt uses production sitemap URL (standard for static files) This follows Docusaurus conventions rather than custom post-build scripts. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
osterman
force-pushed
the
osterman/robots-algolia-verif
branch
from
d944390 to
169138e
Compare
|
These changes were released in v1.204.0-rc.3. |
2 participants
what
why
references
Summary by CodeRabbit
✏️ Tip: You can customize this high-level summary in your review settings.