The skill fetching-salesforce-docs does not fetch the content of articles https://developer.salesforce.com/*

[georzaza]

Greetings!

What an awesome skill.
I have tested the python script(s) on domains help.salesforce.com and the results are great, thank you!

However, there is no real extraction happening for domains developer.salesforce.com.

Below is a sample execution along with the returned results.

python .\extract_salesforce_doc.py --url https://developer.salesforce.com/docs/atlas.en-us.api_meta.meta/api_meta/meta_intro.htm --pretty

{
"ok": true,
"url": "https://developer.salesforce.com/docs/atlas.en-us.api_meta.meta/api_meta/meta_intro.htm",
"httpStatus": 200,
"title": "Understanding Metadata API | Metadata API Developer Guide | Salesforce Developers",
"host": "developer.salesforce.com",
"hostKind": "official-salesforce",
"strategy": "body",
"selector": "body",
"likelyShell": false,
"stealthRequested": false,
"stealthAvailable": false,
"stealthUsed": false,
"text": "We use cookies to make your interactions with our website more meaningful. They help us better understand how our websites are used, so we can tailor content for you. For more information about the different cookies we are using, read the Privacy Statement. To change your cookie settings and preferences, see the Cookie Consent Manager.\nACCEPT ALL COOKIES",
"contentLinks": [
"https://www.salesforce.com/company/privacy/full_privacy/",
"https://onetrust.com/poweredbyonetrust",
"https://developer.salesforce.com/docs/atlas.en-us.api_meta.meta/api_meta/meta_intro.htm#maincontent",
"https://www.salesforce.com/",
"https://developer.salesforce.com/docs/atlas.en-us.api_meta.meta/api_meta/meta_intro.htm#",
"https://www.salesforce.com/industries/",
"...more links..."
],
"childLinks": [
"https://www.salesforce.com/company/privacy/full_privacy/",
"https://onetrust.com/poweredbyonetrust",
"https://developer.salesforce.com/docs/atlas.en-us.api_meta.meta/api_meta/meta_intro.htm#maincontent",
"https://www.salesforce.com/",
"https://developer.salesforce.com/docs/atlas.en-us.api_meta.meta/api_meta/meta_intro.htm#",
"https://www.salesforce.com/industries/",
"...more links..."
],
"candidateCount": 1,
"routedVia": "generic_official_salesforce_extractor"
}

[Digitload]

It looks like your extract_salesforce_doc.py script is hitting the cookie consent banner on developer.salesforce.com instead of the actual article content. This is a common issue when scraping sites that implement a Consent Management Platform (CMP).

Many websites, especially those with strict privacy regulations like GDPR, will present a cookie consent dialog or overlay that blocks or obscures the main content until a user interacts with it. Your script, by simply fetching the body content, is likely getting the initial HTML that includes this banner.

To fix this, your script needs to simulate a user accepting the cookies before attempting to extract the article content. This usually involves using a headless browser automation tool like Playwright or Selenium, which can interact with the page as a real user would.

Here's a conceptual example using Playwright in Python:

from playwright.sync_api import sync_playwright

def extract_salesforce_doc_with_consent(url):
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page()
        page.goto(url)

        # Wait for the cookie consent banner to appear (adjust selector as needed)
        # You'll need to inspect the developer.salesforce.com page to find the correct selector
        # for the "Accept All Cookies" button or the banner itself.
        # For example, if the button has text "ACCEPT ALL COOKIES":
        try:
            accept_button = page.locator('button:has-text("ACCEPT ALL COOKIES")')
            if accept_button.is_visible():
                accept_button.click()
                page.wait_for_load_state('networkidle') # Wait for page to settle after clicking
        except Exception as e:
            print(f"No cookie banner or failed to click: {e}")

        # Now that consent is handled, extract the main content
        # You might need a more specific selector than 'body' for the actual article content
        # e.g., 'article', '.main-content', etc.
        content_text = page.locator('body').inner_text() # Or a more specific content selector

        browser.close()
        return content_text

# Example usage:
# article_url = "https://developer.salesforce.com/docs/atlas.en-us.api_meta.meta/api_meta/meta_intro.htm"
# extracted_content = extract_salesforce_doc_with_consent(article_url)
# print(extracted_content)

You'll need to inspect the developer.salesforce.com page to find the exact CSS selector for the "ACCEPT ALL COOKIES" button or whatever element dismisses the banner. Once that's handled, your script should be able to access the actual article content.

[georzaza]

The html content i was looking for was already 'rendered' despite the cookie consent banner, so by adding a few targeted selectors i was able to extract the content.

That solution is also nice though, i might experiment with it if i get a chance.
Thank you.

[mendioroz87]

I reproduced this against the sample URL and opened a draft PR with a fix: #309

Root cause from the current legacy Atlas implementation: for some developer.salesforce.com/docs/atlas... pages, the browser shell loads successfully but the article web component does not leave enough readable article text in the DOM for selector-based extraction. The page does, however, request Salesforce-owned document payload endpoints:

• /docs/get_document/
• /docs/get_document_content/

The PR captures those responses during the Playwright page load, verifies that the response article ID matches the requested Atlas article, and then parses the returned article HTML into normalized text and resolved links. That identity check is important because /docs/get_document/ can return a guide/default article that is not the requested page; the extractor should not silently use that wrong payload.

Validation I ran locally:

• Issue sample URL returns ok: true, strategy: developer-doc-api, selector /docs/get_document/, and real Metadata API article text.
• Versioned CustomTab URL returns ok: true, strategy: developer-doc-api, selector /docs/get_document_content/, and the correct CustomTab article.
• Modern LWC docs URL still returns ok: true via the existing main DOM strategy.
• python -m py_compile skills\platform-docs-get\scripts\extract_salesforce_doc.py
• npm run validate:skills passed for all 90 skills, with existing warning-only messages unrelated to this fix.

[h-kapoor-sfdc]

@georzaza @mendioroz87 Thanks for the report. The extraction logic for developer.salesforce.com docs has been fixed, the extractor now reads content from the page's nested shadow DOM and ignores the cookie-consent overlay, instead of falling back to the banner text.
This shipped to production in last week's release, so it's already live. Please git pull (or reinstall via npx skills add forcedotcom/sf-skills) and re-run against your URL.