Skip to content

Add warning for when the LLMs.txt cannot be generated in our dashboard. #22329

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: trunk
Choose a base branch
from
Open

Add warning for when the LLMs.txt cannot be generated in our dashboard. #22329

wants to merge 1 commit into from

Conversation

thijsoo
Copy link
Contributor

@thijsoo thijsoo commented Jun 4, 2025

Context

  • We want to notify customers when we cannot generate a llms.txt for them.

Summary

This PR can be summarized in the following changelog entry:

  • Adds a dashboard notification when the llms.txt cannot be generated.

Relevant technical choices:

Test instructions

Test instructions for the acceptance test before the PR gets merged

This PR can be acceptance tested by following these steps:

  • Remove your llms.txt file if you have it and make sure our feature is disabled.
  • Go to our alert center and make sure there are no problems relating to the llms.txt feature.
  • Enable the feature and make sure it works.
  • Go to our alert center and make sure there are no problems relating to the llms.txt feature.
  • Edit your llms.txt file and add a character. This makes it so we no longer control the file.
  • Either disable and re-enable the feature or run the cron job to regenerate it.
  • Go to our alert center and make sure there is now problem relating to the llms.txt feature. Make sure it looks like what is in the issue.
  • Remove the llms.txt file and regenerate it and make sure the alert center is empty again.

Relevant test scenarios

  • Changes should be tested with the browser console open
  • Changes should be tested on different posts/pages/taxonomies/custom post types/custom taxonomies
  • Changes should be tested on different editors (Default Block/Gutenberg/Classic/Elementor/other)
  • Changes should be tested on different browsers
  • Changes should be tested on multisite

Test instructions for QA when the code is in the RC

  • QA should use the same steps as above.

QA can test this PR by following these steps:

Impact check

This PR affects the following parts of the plugin, which may require extra testing:

UI changes

  • This PR changes the UI in the plugin. I have added the 'UI change' label to this PR.

Other environments

  • This PR also affects Shopify. I have added a changelog entry starting with [shopify-seo], added test instructions for Shopify and attached the Shopify label to this PR.

Documentation

  • I have written documentation for this change. For example, comments in the Relevant technical choices, comments in the code, documentation on Confluence / shared Google Drive / Yoast developer portal, or other.

Quality assurance

  • I have tested this code to the best of my abilities.
  • During testing, I had activated all plugins that Yoast SEO provides integrations for.
  • I have added unit tests to verify the code works as intended.
  • If any part of the code is behind a feature flag, my test instructions also cover cases where the feature flag is switched off.
  • I have written this PR in accordance with my team's definition of done.
  • I have checked that the base branch is correctly set.

Innovation

  • No innovation project is applicable for this PR.
  • This PR falls under an innovation project. I have attached the innovation label.
  • I have added my hours to the WBSO document.

Fixes #

@thijsoo thijsoo added the changelog: non-user-facing Needs to be included in the 'Non-userfacing' category in the changelog label Jun 4, 2025
@thijsoo thijsoo force-pushed the 611-show-warning-in-yoast-dashboard-when-llmstxt-cant-be-created-nice-to-have branch from 40a60ac to 59228b9 Compare June 4, 2025 07:59
@thijsoo thijsoo force-pushed the 611-show-warning-in-yoast-dashboard-when-llmstxt-cant-be-created-nice-to-have branch from 59228b9 to 273f280 Compare June 4, 2025 08:02
@coveralls
Copy link

Pull Request Test Coverage Report for Build a7ad6847bd1adff7d83c03c70d60e2ae4883676c

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 10 of 42 (23.81%) changed or added relevant lines in 2 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall first build on 611-show-warning-in-yoast-dashboard-when-llmstxt-cant-be-created-nice-to-have at 53.018%
Changes Missing Coverage Covered Lines Changed/Added Lines %
src/llms-txt/application/file/file-failure-notification-presenter.php 0 15 0.0%
src/llms-txt/user-interface/file-failure-llms-txt-notification-integration.php 10 27 37.04%
Totals Coverage Status
Change from base Build 687a2c1e05a7d3a86cf158fba04e7aef45dcccd3: 53.0%
Covered Lines: 29722
Relevant Lines: 57006
💛 - Coveralls

leonidasmi
leonidasmi reviewed Jun 4, 2025
View reviewed changes
Copy link
Contributor

@leonidasmi leonidasmi left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CR: ❌

I think this implementation is blocked by this issue, so let's talk to see what we should do.

src/llms-txt/application/file/file-failure-notification-presenter.php
\esc_html__( 'Your llms.txt file couldn\'t be auto-generated', 'wordpress-seo' ),
\sprintf(
/* translators: 1: Link start tag to the WordPress Reading Settings page, 2: Link closing tag. */
\esc_html__( 'It looks like there is an llms.txt file already that wasn\'t created by Yoast, or the llms.txt file created by Yoast has been edited manually. We don\'t want to overwrite this file\'s content, so if you want to let Yoast keep auto-generating the llms.txt file, you can %1$smanually delete the existing one%2$s. Otherwise, consider disabling the Yoast feature.', 'wordpress-seo' ),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like we have a single message that describes only the not_managed_by_yoast_seo case. We need to be able to show the different message when on the filesystem_permissions case.

src/llms-txt/user-interface/file-failure-llms-txt-notification-integration.php
* @return bool
*/
private function should_show_file_failure_notification(): bool {
return \get_option( Populate_File_Command_Handler::GENERATION_FAILURE_OPTION, false ) !== false;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Populate_File_Command_Handler::GENERATION_FAILURE_OPTION is not an autoloaded option (nor it should be).

Which means that with this PR, we add a DB call to every admin page load, which is something we shouldn't do.

Unfortunately, the way the notification center is built, that way is the only way we ever added notices, so I'm not sure what's the solution here, but let's go back to the drawing board for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@leonidasmi leonidasmi leonidasmi left review comments

Assignees
No one assigned
Labels
changelog: non-user-facing Needs to be included in the 'Non-userfacing' category in the changelog
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@thijsoo @coveralls @leonidasmi