article-sample-with-table.html

<article class="post-full mw-100 ph3 ph0-l fs-20px">
    <h1 class="f6 f7-l fw4 gray1 pt1 pt3-l mb1">
        The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals
    </h1>
    <p class="f3 fw5 gray5 db di-l mt2">2025-08-29</p>
    <ul class="author-lists pl0 mt4 flex">
        <li class="list pr2 mb1-ns flex items-center">
            <a href="/author/joao-tome/" class="static-avatar pr1">
                <img
                    class="author-profile-image br-100 mr2"
                    src="https://blog.cloudflare.com/cdn-cgi/image/format=auto,dpr=3,width=64,height=64,gravity=face,fit=crop,zoom=0.5/https://cf-assets.www.cloudflare.com/zkvhlag99gkb/KOYiYhfI8O9WNWxB8IWk7/e1e24f4df878f45e812bdd4a893b026e/joao-tome.jpeg"
                    alt="João Tomé"
                    width="62"
                    height="62"
            /></a>
            <div class="author-name-tooltip">
                <a href="/author/joao-tome/" class="fw4 f3 black mr3 no-underline">João Tomé</a>
            </div>
        </li>
    </ul>
    <section class="post-full-content">
        <div class="mb2 gray5">9 min read</div>
        <img
            class="mr2"
            src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/5I7xKXRl6rCiu0BB1Xv5h2/41b1243026611f167f2bc6ad7b5e3a4e/unnamed__26_.png"
            alt=""
        />
        <div class="post-content lh-copy gray1">
            <p>
                In 2025, Generative AI is reshaping how people and companies use the Internet.
                Search engines once drove traffic to content creators through links. Now, AI
                training crawlers — the engines behind commonly-used LLMs — are consuming vast
                amounts of web data, while sending far fewer users back. We covered this shift,
                along with related
                <a
                    href="https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/"
                >
                    <u>trends</u>
                </a>
                and Cloudflare
                <a href="https://blog.cloudflare.com/tag/pay-per-crawl/">
                    <u>features</u>
                </a>
                (like pay per crawl) in early July. Studies from Pew Research Center (<a
                    href="https://www.pewresearch.org/short-reads/2025/04/28/americans-largely-foresee-ai-having-negative-effects-on-news-journalists/"
                >
                    <u>1</u> </a
                >,
                <a
                    href="https://www.pewresearch.org/short-reads/2025/07/22/google-users-are-less-likely-to-click-on-links-when-an-ai-summary-appears-in-the-results/"
                >
                    <u>2</u> </a
                >) and
                <a
                    href="https://pressgazette.co.uk/media-audience-and-business-data/google-ai-overviews-publishers-report-clickthroughs-authoritas-report/"
                >
                    <u>Authoritas</u>
                </a>
                already point to AI overviews — Google’s new AI-generated summaries shown at the top
                of search results — contributing to sharp declines in news website traffic. For a
                news site, this means lots of bot hits, but far fewer real readers clicking through
                — which in turn means fewer people clicking on ads or chances to convert to
                subscriptions.
            </p>
            <p>
                Cloudflare's data shows the same pattern. Crawling by search engines and AI services
                surged in the first half of 2025 — up 24% year-over-year in June — before slowing to
                just 4% year-over-year growth in July. How is the space evolving? Which crawling
                purposes are most common, and how is that changing? Spoiler: training-related
                crawling is leading the way. In this post, we track AI and search bot crawl
                activity, what purposes dominate, and which platforms contribute the least referral
                traffic back to creators.
            </p>
            <div class="anchor relative flex">
                <h3 id="key-takeaways">Key takeaways</h3>
                <a href="#key-takeaways" aria-hidden="true" class="relative sm:absolute sm:-left-5">
                    <svg width="16" height="16" viewBox="0 0 24 24">
                        <path
                            fill="currentcolor"
                            d="m12.11 15.39-3.88 3.88a2.52 2.52 0 0 1-3.5 0 2.47 2.47 0 0 1 0-3.5l3.88-3.88a1 1 0 0 0-1.42-1.42l-3.88 3.89a4.48 4.48 0 0 0 6.33 6.33l3.89-3.88a1 1 0 1 0-1.42-1.42Zm8.58-12.08a4.49 4.49 0 0 0-6.33 0l-3.89 3.88a1 1 0 0 0 1.42 1.42l3.88-3.88a2.52 2.52 0 0 1 3.5 0 2.47 2.47 0 0 1 0 3.5l-3.88 3.88a1 1 0 1 0 1.42 1.42l3.88-3.89a4.49 4.49 0 0 0 0-6.33ZM8.83 15.17a1 1 0 0 0 1.1.22 1 1 0 0 0 .32-.22l4.92-4.92a1 1 0 0 0-1.42-1.42l-4.92 4.92a1 1 0 0 0 0 1.42Z"
                        ></path>
                    </svg>
                </a>
            </div>
            <ul>
                <li>
                    <p>
                        Training crawling grows: Training now drives nearly 80% of AI bot activity,
                        up from 72% a year ago.
                    </p>
                </li>
                <li>
                    <p>
                        Publisher referrals drop: Google referrals to news sites fell, with March
                        2025 down ~9% compared to January.
                    </p>
                </li>
                <li>
                    <p>
                        AI &amp; search crawling increase: Crawling rose 32% year-over-year in April
                        2025, before slowing to 4% year-over-year growth in July.
                    </p>
                </li>
                <li>
                    <p>
                        AI-only crawler shifts: OpenAI’s GPTBot more than doubled in share of AI
                        crawling traffic (4.7% to 11.7%),&nbsp;Anthropic’s ClaudeBot rose (6% to
                        ~10%), while ByteDance’s Bytespider fell from 14.1% to 2.4%.
                    </p>
                </li>
                <li>
                    <p>
                        Crawl-to-refer imbalance (how many pages a bot crawls per page that a user
                        clicks back to): Anthropic increased referrals but still leads with 38,000
                        crawls per visitor in July (down from 286,000:1 in January). Perplexity
                        decreased referrals in 2025 — with more crawling but fewer referrals at 194
                        crawls per visitor in July.
                    </p>
                </li>
            </ul>
            <p>
                Several of the trends in this blog use
                <a href="https://radar.cloudflare.com/ai-insights">
                    <u>Cloudflare Radar’s new AI Insights</u>
                </a>
                features, explained in more detail in the post: “<a
                    href="https://blog.cloudflare.com/ai-crawler-traffic-by-purpose-and-industry"
                >
                    <b>
                        <u
                            >A deeper look at AI crawlers: breaking down traffic by purpose and
                            industry</u
                        >
                    </b> </a
                >.”
            </p>
            <div class="anchor relative flex">
                <h2 id="google-referrals-fall-as-ai-overviews-expand">
                    Google referrals fall as AI Overviews expand
                </h2>
                <a
                    href="#google-referrals-fall-as-ai-overviews-expand"
                    aria-hidden="true"
                    class="relative sm:absolute sm:-left-5"
                >
                    <svg width="16" height="16" viewBox="0 0 24 24">
                        <path
                            fill="currentcolor"
                            d="m12.11 15.39-3.88 3.88a2.52 2.52 0 0 1-3.5 0 2.47 2.47 0 0 1 0-3.5l3.88-3.88a1 1 0 0 0-1.42-1.42l-3.88 3.89a4.48 4.48 0 0 0 6.33 6.33l3.89-3.88a1 1 0 1 0-1.42-1.42Zm8.58-12.08a4.49 4.49 0 0 0-6.33 0l-3.89 3.88a1 1 0 0 0 1.42 1.42l3.88-3.88a2.52 2.52 0 0 1 3.5 0 2.47 2.47 0 0 1 0 3.5l-3.88 3.88a1 1 0 1 0 1.42 1.42l3.88-3.89a4.49 4.49 0 0 0 0-6.33ZM8.83 15.17a1 1 0 0 0 1.1.22 1 1 0 0 0 .32-.22l4.92-4.92a1 1 0 0 0-1.42-1.42l-4.92 4.92a1 1 0 0 0 0 1.42Z"
                        ></path>
                    </svg>
                </a>
            </div>
            <p>
                Referral traffic from search is already shifting, as we noted above and as
                <a href="http://studies">
                    <u>studies</u>
                </a>
                have shown. In our dataset of news-related customers (spanning the Americas, Europe,
                and Asia), Google’s referrals have been clearly declining since February 2025. This
                drop is unusual, since overall Internet traffic (and referrals as well) historically
                has only dipped during July and August — the summer months when the Northern
                Hemisphere is largely on break from school or work. The sharpest and least seasonal
                decline came in March. Despite being a 31-day month, March had almost the same
                referral volume as the shorter, 28-day February.
            </p>
            <figure class="kg-card kg-image-card">
                <img
                    src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/1ZWlDsTAtPveEo2Kq8nzu9/ebd655d9ea51f35cfae1f4d09cfecc76/1.png"
                    alt=""
                    class="kg-image"
                    width="1999"
                    height="528"
                    loading="lazy"
                />
            </figure>
            <p>
                Looking at longer comparisons: March 2025 referral traffic from Google was 9% lower
                than January, the same drop seen in June. April was worse, down 15% compared with
                January.
            </p>
            <p>
                This drop seems to coincide with some of Google’s changes. AI Overviews launched in
                the U.S. in
                <a href="https://blog.google/products/search/generative-ai-google-search-may-2024/">
                    <u>May 2024</u> </a
                >, but in March 2025, Google upgraded AI Overviews with Gemini 2.0, introduced AI
                Mode in Labs, and
                <a
                    href="https://blog.google/feed/were-bringing-the-helpfulness-of-ai-overviews-to-more-countries-in-europe/"
                >
                    <u>expanded</u>
                </a>
                Overviews to more European countries. By May 2025, AI Mode rolled out broadly in the
                U.S. with Gemini 2.5, adding conversational search, Deep Search, and personalized
                recommendations.
            </p>
            <p>
                The search-to-news site pipeline seems to be weakening, replaced in part by
                AI-driven results.
            </p>
            <p>
                Looking at a daily perspective, we can also spot a clear U.S.-election-related peak
                in referrals from Google to the cohort of known news sites on November 5–6, 2024.
            </p>
            <figure class="kg-card kg-image-card">
                <img
                    src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/1Gtq4mnTg8KdVWaUkpH51A/86e7f7dfeb31f846df4ae8486c25b4aa/2.png"
                    alt=""
                    class="kg-image"
                    width="1999"
                    height="532"
                    loading="lazy"
                />
            </figure>
            <div class="anchor relative flex">
                <h2 id="ai-and-search-crawling-spring-surge-24-summer-slowdown">
                    AI and search crawling: spring surge (+24%), summer slowdown
                </h2>
                <a
                    href="#ai-and-search-crawling-spring-surge-24-summer-slowdown"
                    aria-hidden="true"
                    class="relative sm:absolute sm:-left-5"
                >
                    <svg width="16" height="16" viewBox="0 0 24 24">
                        <path
                            fill="currentcolor"
                            d="m12.11 15.39-3.88 3.88a2.52 2.52 0 0 1-3.5 0 2.47 2.47 0 0 1 0-3.5l3.88-3.88a1 1 0 0 0-1.42-1.42l-3.88 3.89a4.48 4.48 0 0 0 6.33 6.33l3.89-3.88a1 1 0 1 0-1.42-1.42Zm8.58-12.08a4.49 4.49 0 0 0-6.33 0l-3.89 3.88a1 1 0 0 0 1.42 1.42l3.88-3.88a2.52 2.52 0 0 1 3.5 0 2.47 2.47 0 0 1 0 3.5l-3.88 3.88a1 1 0 1 0 1.42 1.42l3.88-3.89a4.49 4.49 0 0 0 0-6.33ZM8.83 15.17a1 1 0 0 0 1.1.22 1 1 0 0 0 .32-.22l4.92-4.92a1 1 0 0 0-1.42-1.42l-4.92 4.92a1 1 0 0 0 0 1.42Z"
                        ></path>
                    </svg>
                </a>
            </div>
            <p>
                <a
                    href="https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/"
                >
                    <u>In June</u> </a
                >, we talked about search and AI crawler growth, and our picture of the trend is now
                more complete with more data. To focus only on AI and search crawlers, and to remove
                the bias of customer growth, we analyzed a fixed set of customers from specific
                weeks, a method we’ve also used in the
                <a href="http://radar.cloudflare.com/year-in-review/">
                    <u>Cloudflare Radar Year in Review</u> </a
                >.
            </p>
            <p>
                What the data shows: crawling spiked twice: first in November 2024, then again
                between March and April 2025. April 2025 alone was up 32% compared with May 2024,
                the first full month where we have comparable data. After that surge, growth
                stabilized. In June 2025, crawling traffic was still 24% higher year-over-year, but
                by July the increase was down to just 4%. That shift highlights how quickly crawler
                activity can accelerate and then cool down.
            </p>
            <p>
                As the chart below shows, crawling traffic rose sharply in March and April. It
                remained high but slightly lower in May, before starting to drop in June. The
                seasonal dip is similar to what we see in overall Internet traffic during the
                Northern Hemisphere’s summer months (August and September are often the quietest),
                though in the case of crawlers, this is likely due to reduced overall web activity
                rather than bots themselves taking a “break.” Historically, activity tends to rise
                again in November — as it did in 2024 for AI and search bot traffic — when people
                spend more time online for shopping and seasonal habits (a pattern we’ve seen in
                <a
                    href="https://blog.cloudflare.com/from-deals-to-ddos-exploring-cyber-week-2024-internet-trends/"
                >
                    <u>past years</u> </a
                >).
            </p>
            <figure class="kg-card kg-image-card">
                <img
                    src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/1SKJcH4r7smlgCBC9vjULt/1311a9ded068a142122630af5afc3766/3.png"
                    alt=""
                    class="kg-image"
                    width="1999"
                    height="692"
                    loading="lazy"
                />
            </figure>
            <p>
                Googlebot is
                <a
                    href="https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/"
                >
                    <u>still</u>
                </a>
                the anchor, accounting for 39% of all AI and search crawler traffic, but the fastest
                growth now comes from AI-specific crawlers, though bots related to Amazon and
                ByteDance (Bytespider) have lost significant ground. GPTBot’s share grew from 4.7%
                in July 2024 to 11.7% in July 2025. ClaudeBot also increased, from 6% to nearly 10%,
                while Meta’s crawler jumped from 0.9% to 7.5%. By contrast, Amazonbot dropped from
                10.2% to 5.9%, and ByteDance’s Bytespider dropped from 14.1% to just 2.4%.
            </p>
            <p>
                The table below shows how market shares have shifted between July 2024 and July
                2025:
            </p>
            <table>
                <tbody>
                    <tr>
                        <td>
                            <p></p>
                        </td>
                        <td>
                            <p>
                                <b>Bot name</b>
                            </p>
                        </td>
                        <td>
                            <p>
                                <b>% share July 2024</b>
                            </p>
                        </td>
                        <td>
                            <p>
                                <b>% share July 2025</b>
                            </p>
                        </td>
                        <td>
                            <p>
                                <b>Δ percentage-point change</b>
                            </p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>1</b>
                            </p>
                        </td>
                        <td>
                            <p>Googlebot</p>
                        </td>
                        <td>
                            <p>37.5</p>
                        </td>
                        <td>
                            <p>39</p>
                        </td>
                        <td>
                            <p>1.5</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>2</b>
                            </p>
                        </td>
                        <td>
                            <p>GPTBot</p>
                        </td>
                        <td>
                            <p>4.7</p>
                        </td>
                        <td>
                            <p>11.7</p>
                        </td>
                        <td>
                            <p>7</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>3</b>
                            </p>
                        </td>
                        <td>
                            <p>ClaudeBot</p>
                        </td>
                        <td>
                            <p>6</p>
                        </td>
                        <td>
                            <p>9.9</p>
                        </td>
                        <td>
                            <p>3.9</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>4</b>
                            </p>
                        </td>
                        <td>
                            <p>Bingbot</p>
                        </td>
                        <td>
                            <p>8.7</p>
                        </td>
                        <td>
                            <p>9.3</p>
                        </td>
                        <td>
                            <p>0.6</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>5</b>
                            </p>
                        </td>
                        <td>
                            <p>Meta-ExternalAgent</p>
                        </td>
                        <td>
                            <p>0.9</p>
                        </td>
                        <td>
                            <p>7.5</p>
                        </td>
                        <td>
                            <p>6.5</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>6</b>
                            </p>
                        </td>
                        <td>
                            <p>Amazonbot</p>
                        </td>
                        <td>
                            <p>10.2</p>
                        </td>
                        <td>
                            <p>5.9</p>
                        </td>
                        <td>
                            <p>-4.3</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>7</b>
                            </p>
                        </td>
                        <td>
                            <p>Googlebot-Image</p>
                        </td>
                        <td>
                            <p>4.1</p>
                        </td>
                        <td>
                            <p>3.3</p>
                        </td>
                        <td>
                            <p>-0.8</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>8</b>
                            </p>
                        </td>
                        <td>
                            <p>Yandex</p>
                        </td>
                        <td>
                            <p>5</p>
                        </td>
                        <td>
                            <p>2.9</p>
                        </td>
                        <td>
                            <p>-2.1</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>9</b>
                            </p>
                        </td>
                        <td>
                            <p>GoogleOther</p>
                        </td>
                        <td>
                            <p>4.6</p>
                        </td>
                        <td>
                            <p>2.7</p>
                        </td>
                        <td>
                            <p>-1.8</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>10</b>
                            </p>
                        </td>
                        <td>
                            <p>Bytespider</p>
                        </td>
                        <td>
                            <p>14.1</p>
                        </td>
                        <td>
                            <p>2.4</p>
                        </td>
                        <td>
                            <p>-11.6</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>11</b>
                            </p>
                        </td>
                        <td>
                            <p>Applebot</p>
                        </td>
                        <td>
                            <p>1.8</p>
                        </td>
                        <td>
                            <p>1.5</p>
                        </td>
                        <td>
                            <p>-0.3</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>12</b>
                            </p>
                        </td>
                        <td>
                            <p>ChatGPT-User</p>
                        </td>
                        <td>
                            <p>0.1</p>
                        </td>
                        <td>
                            <p>0.9</p>
                        </td>
                        <td>
                            <p>0.9</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>13</b>
                            </p>
                        </td>
                        <td>
                            <p>OAI-SearchBot</p>
                        </td>
                        <td>
                            <p>0</p>
                        </td>
                        <td>
                            <p>0.9</p>
                        </td>
                        <td>
                            <p>0.9</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>14</b>
                            </p>
                        </td>
                        <td>
                            <p>Baiduspider</p>
                        </td>
                        <td>
                            <p>0.5</p>
                        </td>
                        <td>
                            <p>0.5</p>
                        </td>
                        <td>
                            <p>0</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>15</b>
                            </p>
                        </td>
                        <td>
                            <p>Googlebot-Mobile</p>
                        </td>
                        <td>
                            <p>0.2</p>
                        </td>
                        <td>
                            <p>0.4</p>
                        </td>
                        <td>
                            <p>0.2</p>
                        </td>
                    </tr>
                </tbody>
            </table>
            <div class="anchor relative flex">
                <h2 id="ai-only-crawlers-openai-rises-bytedance-falls">
                    AI-only crawlers: OpenAI rises, ByteDance falls
                </h2>
                <a
                    href="#ai-only-crawlers-openai-rises-bytedance-falls"
                    aria-hidden="true"
                    class="relative sm:absolute sm:-left-5"
                >
                    <svg width="16" height="16" viewBox="0 0 24 24">
                        <path
                            fill="currentcolor"
                            d="m12.11 15.39-3.88 3.88a2.52 2.52 0 0 1-3.5 0 2.47 2.47 0 0 1 0-3.5l3.88-3.88a1 1 0 0 0-1.42-1.42l-3.88 3.89a4.48 4.48 0 0 0 6.33 6.33l3.89-3.88a1 1 0 1 0-1.42-1.42Zm8.58-12.08a4.49 4.49 0 0 0-6.33 0l-3.89 3.88a1 1 0 0 0 1.42 1.42l3.88-3.88a2.52 2.52 0 0 1 3.5 0 2.47 2.47 0 0 1 0 3.5l-3.88 3.88a1 1 0 1 0 1.42 1.42l3.88-3.89a4.49 4.49 0 0 0 0-6.33ZM8.83 15.17a1 1 0 0 0 1.1.22 1 1 0 0 0 .32-.22l4.92-4.92a1 1 0 0 0-1.42-1.42l-4.92 4.92a1 1 0 0 0 0 1.42Z"
                        ></path>
                    </svg>
                </a>
            </div>
            <p>
                Looking only at AI bot traffic (as tracked on our
                <a
                    href="https://radar.cloudflare.com/explorer?dataSet=ai.bots&amp;groupBy=user_agent&amp;dt=2025-07-01_2025-07-31&amp;timeCompare=2024-07-01"
                >
                    <u>Radar AI page</u> </a
                >), the trend is clear. Since January 2025, GPTBot has steadily increased its
                crawling volume, driven mainly by training-related activity. ClaudeBot crawling
                accelerated in June, while Amazonbot and Bytespider activity slowed.
            </p>
            <p>
                The
                <a
                    href="https://radar.cloudflare.com/explorer?dataSet=ai.bots&amp;groupBy=user_agent&amp;dt=2025-07-01_2025-07-31&amp;timeCompare=2024-07-01"
                >
                    <u>chart</u>
                </a>
                below shows how GPTBot surged over the past 12 months, overtaking Amazonbot and
                Bytespider, which both fell sharply:
            </p>
            <figure class="kg-card kg-image-card">
                <img
                    src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/5XRamYFTPqDrQ0bMQSG4C7/e741692f7019a4842b5d82bf4ab64106/4.png"
                    alt=""
                    class="kg-image"
                    width="1600"
                    height="1434"
                    loading="lazy"
                />
            </figure>
            <p>
                A comparison between July 2024 and July 2025 makes the shift even more obvious.
                GPTBot gained 16 percentage points, Meta’s crawler rose by more than 15, and
                ClaudeBot grew by 8. On the shrinking side, Amazonbot dropped 12 percentage points
                and Bytespider dropped over 31 percentage points.
            </p>
            <table>
                <tbody>
                    <tr>
                        <td>
                            <p></p>
                        </td>
                        <td>
                            <p>
                                <b>AI-only bots</b>
                            </p>
                        </td>
                        <td>
                            <p>July 2024 %</p>
                        </td>
                        <td>
                            <p>July 2025 %</p>
                        </td>
                        <td>
                            <p>Δ percentage-point change</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>1</p>
                        </td>
                        <td>
                            <p>GPTBot</p>
                        </td>
                        <td>
                            <p>11.9</p>
                        </td>
                        <td>
                            <p>28.1</p>
                        </td>
                        <td>
                            <p>16.1</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>2</p>
                        </td>
                        <td>
                            <p>ClaudeBot</p>
                        </td>
                        <td>
                            <p>15</p>
                        </td>
                        <td>
                            <p>23.3</p>
                        </td>
                        <td>
                            <p>8.3</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>3</p>
                        </td>
                        <td>
                            <p>Meta-ExternalAgent</p>
                        </td>
                        <td>
                            <p>2.4</p>
                        </td>
                        <td>
                            <p>17.7</p>
                        </td>
                        <td>
                            <p>15.3</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>4</p>
                        </td>
                        <td>
                            <p>Amazonbot</p>
                        </td>
                        <td>
                            <p>26.4</p>
                        </td>
                        <td>
                            <p>14.1</p>
                        </td>
                        <td>
                            <p>-12.3</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>5</p>
                        </td>
                        <td>
                            <p>Bytespider</p>
                        </td>
                        <td>
                            <p>37.3</p>
                        </td>
                        <td>
                            <p>5.8</p>
                        </td>
                        <td>
                            <p>-31.5</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>6</p>
                        </td>
                        <td>
                            <p>Applebot</p>
                        </td>
                        <td>
                            <p>4.9</p>
                        </td>
                        <td>
                            <p>3.7</p>
                        </td>
                        <td>
                            <p>-1.2</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>7</p>
                        </td>
                        <td>
                            <p>ChatGPT-User</p>
                        </td>
                        <td>
                            <p>0.2</p>
                        </td>
                        <td>
                            <p>2.4</p>
                        </td>
                        <td>
                            <p>2.2</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>8</p>
                        </td>
                        <td>
                            <p>OAI-SearchBot</p>
                        </td>
                        <td>
                            <p>0</p>
                        </td>
                        <td>
                            <p>2.2</p>
                        </td>
                        <td>
                            <p>2.2</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>9</p>
                        </td>
                        <td>
                            <p>TikTokSpider</p>
                        </td>
                        <td>
                            <p>0</p>
                        </td>
                        <td>
                            <p>0.7</p>
                        </td>
                        <td>
                            <p>0.7</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>10</p>
                        </td>
                        <td>
                            <p>imgproxy</p>
                        </td>
                        <td>
                            <p>0</p>
                        </td>
                        <td>
                            <p>0.7</p>
                        </td>
                        <td>
                            <p>0.7</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>11</p>
                        </td>
                        <td>
                            <p>PerplexityBot</p>
                        </td>
                        <td>
                            <p>0</p>
                        </td>
                        <td>
                            <p>0.4</p>
                        </td>
                        <td>
                            <p>0.4</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>12</p>
                        </td>
                        <td>
                            <p>Google-CloudVertexBot</p>
                        </td>
                        <td>
                            <p>0</p>
                        </td>
                        <td>
                            <p>0.3</p>
                        </td>
                        <td>
                            <p>0.3</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>13</p>
                        </td>
                        <td>
                            <p>AI2Bot</p>
                        </td>
                        <td>
                            <p>0</p>
                        </td>
                        <td>
                            <p>0.2</p>
                        </td>
                        <td>
                            <p>0.2</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>14</p>
                        </td>
                        <td>
                            <p>Timpibot</p>
                        </td>
                        <td>
                            <p>0.6</p>
                        </td>
                        <td>
                            <p>0.1</p>
                        </td>
                        <td>
                            <p>-0.5</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>15</p>
                        </td>
                        <td>
                            <p>CCBot</p>
                        </td>
                        <td>
                            <p>0.1</p>
                        </td>
                        <td>
                            <p>0.1</p>
                        </td>
                        <td>
                            <p>0</p>
                        </td>
                    </tr>
                </tbody>
            </table>
            <figure class="kg-card kg-image-card">
                <img
                    src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/71p4CgiUXwYrb9LIsJCruI/44dd4b232a715b852417853e7026fbcb/5.png"
                    alt=""
                    class="kg-image"
                    width="1600"
                    height="1101"
                    loading="lazy"
                />
            </figure>
            <p>
                We covered the functionality of these bots in our
                <a
                    href="https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/#ai-only-crawlers-perspective"
                >
                    <u>June blog post</u> </a
                >.
            </p>
            <div class="anchor relative flex">
                <h2 id="crawling-by-purpose-training-dominates">
                    Crawling by purpose: training dominates
                </h2>
                <a
                    href="#crawling-by-purpose-training-dominates"
                    aria-hidden="true"
                    class="relative sm:absolute sm:-left-5"
                >
                    <svg width="16" height="16" viewBox="0 0 24 24">
                        <path
                            fill="currentcolor"
                            d="m12.11 15.39-3.88 3.88a2.52 2.52 0 0 1-3.5 0 2.47 2.47 0 0 1 0-3.5l3.88-3.88a1 1 0 0 0-1.42-1.42l-3.88 3.89a4.48 4.48 0 0 0 6.33 6.33l3.89-3.88a1 1 0 1 0-1.42-1.42Zm8.58-12.08a4.49 4.49 0 0 0-6.33 0l-3.89 3.88a1 1 0 0 0 1.42 1.42l3.88-3.88a2.52 2.52 0 0 1 3.5 0 2.47 2.47 0 0 1 0 3.5l-3.88 3.88a1 1 0 1 0 1.42 1.42l3.88-3.89a4.49 4.49 0 0 0 0-6.33ZM8.83 15.17a1 1 0 0 0 1.1.22 1 1 0 0 0 .32-.22l4.92-4.92a1 1 0 0 0-1.42-1.42l-4.92 4.92a1 1 0 0 0 0 1.42Z"
                        ></path>
                    </svg>
                </a>
            </div>
            <p>
                Training is the clear leader.<i>
                    (We classify purpose based on operator disclosures and industry sources, a
                    method we explained in this
                </i>
                <a href="https://blog.cloudflare.com/ai-crawler-traffic-by-purpose-and-industry">
                    <i>
                        <u>AI Week blog</u>
                    </i>
                </a>
                <i>.)</i> Over the past 12 months, 80% of AI crawling was for training, compared
                with 18% for search and just 2% for user actions. In the last six months, the share
                for training rose further to 82%, while search dropped to 15% and user actions
                increased slightly to 3%.
            </p>
            <p>
                The
                <a href="https://radar.cloudflare.com/ai-insights#crawl-purpose">
                    <u>chart</u>
                </a>
                below shows how training-related crawling steadily grew over the past year, far
                outpacing other purposes:
            </p>
            <figure class="kg-card kg-image-card">
                <img
                    src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/10lBzdfhgLKiWrEAIcs691/8b11d8d733c48938a7235dc07f65a83a/6.png"
                    alt=""
                    class="kg-image"
                    width="1600"
                    height="876"
                    loading="lazy"
                />
            </figure>
            <p>
                The year-over-year comparison reinforces this trend. In July 2024, training
                accounted for 72% of AI crawling. By July 2025, it had risen to 79%. Over the same
                period, search fell from 26% to 17%, while user actions grew modestly from 2% to
                3.2%.
            </p>
            <figure class="kg-card kg-image-card">
                <img
                    src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/2OcV2pA5nOBpOrl8pKPotL/4901f128d5feaba82357972509ba09f2/7.png"
                    alt=""
                    class="kg-image"
                    width="1578"
                    height="460"
                    loading="lazy"
                />
            </figure>
            <div class="anchor relative flex">
                <h2
                    id="crawl-to-refer-ratios-shifts-tens-of-thousands-of-bot-crawls-per-human-click"
                >
                    Crawl-to-refer ratios shifts: tens of thousands of bot crawls per human click
                </h2>
                <a
                    href="#crawl-to-refer-ratios-shifts-tens-of-thousands-of-bot-crawls-per-human-click"
                    aria-hidden="true"
                    class="relative sm:absolute sm:-left-5"
                >
                    <svg width="16" height="16" viewBox="0 0 24 24">
                        <path
                            fill="currentcolor"
                            d="m12.11 15.39-3.88 3.88a2.52 2.52 0 0 1-3.5 0 2.47 2.47 0 0 1 0-3.5l3.88-3.88a1 1 0 0 0-1.42-1.42l-3.88 3.89a4.48 4.48 0 0 0 6.33 6.33l3.89-3.88a1 1 0 1 0-1.42-1.42Zm8.58-12.08a4.49 4.49 0 0 0-6.33 0l-3.89 3.88a1 1 0 0 0 1.42 1.42l3.88-3.88a2.52 2.52 0 0 1 3.5 0 2.47 2.47 0 0 1 0 3.5l-3.88 3.88a1 1 0 1 0 1.42 1.42l3.88-3.89a4.49 4.49 0 0 0 0-6.33ZM8.83 15.17a1 1 0 0 0 1.1.22 1 1 0 0 0 .32-.22l4.92-4.92a1 1 0 0 0-1.42-1.42l-4.92 4.92a1 1 0 0 0 0 1.42Z"
                        ></path>
                    </svg>
                </a>
            </div>
            <p>
                The crawl-to-refer ratio measures how many pages a platform crawls compared with how
                often it drives users to a website. In practice, a high ratio means heavy crawling
                but little referral traffic. For example, for every visitor Anthropic refers back to
                a website, its crawlers have already visited tens of thousands of pages.
            </p>
            <p>
                Why does this metric matter? It highlights the imbalance between how much content AI
                systems consume and how little traffic they return. For publishers, it can feel like
                giving away the raw material for free. With that in mind, here’s how different
                platforms compare from January to July 2025.
            </p>
            <p>
                Anthropic remains the most crawl-heavy platform. Even after an 87% decline this
                year, it still crawled 38,000 pages for every referred page visit in July 2025 — the
                highest imbalance among major AI players. Referrals may be improving, though, after
                Anthropic added
                <a href="https://www.anthropic.com/news/web-search">
                    <u>web search to Claude in March 2025</u>
                </a>
                (initially for U.S. paid users) and expanded it globally by
                <a href="https://www.brightedge.com/claude-search">
                    <u>May to all users, including the free tier</u> </a
                >. The feature introduced direct citations with clickable URLs, creating new
                referral pathways.
            </p>
            <p>
                The full dataset is below, showing January–July 2025 ratios by platform ordered by
                the highest ratio average: (Note: a rising ratio means <i>more</i> bot crawling per
                human click sent back, while a falling ratio means <i>less</i> bot crawling per
                human click sent back)

                <b>Crawl-to-refer ratio (from </b>
                <a
                    href="https://radar.cloudflare.com/ai-insights?dateStart=2025-07-01&amp;dateEnd=2025-07-31#crawl-to-refer-ratio"
                >
                    <b>
                        <u>Cloudflare Radar’s data</u>
                    </b>
                </a>
                <b>)</b>
            </p>
            <table>
                <tbody>
                    <tr>
                        <td>
                            <p>
                                <b>Service</b>
                            </p>
                        </td>
                        <td>
                            <p>
                                <b>Jan</b>
                            </p>
                        </td>
                        <td>
                            <p>
                                <b>Feb</b>
                            </p>
                        </td>
                        <td>
                            <p>
                                <b>Mar</b>
                            </p>
                        </td>
                        <td>
                            <p>
                                <b>Apr</b>
                            </p>
                        </td>
                        <td>
                            <p>
                                <b>May</b>
                            </p>
                        </td>
                        <td>
                            <p>
                                <b>Jun</b>
                            </p>
                        </td>
                        <td>
                            <p>
                                <b>Jul</b>
                            </p>
                        </td>
                        <td>
                            <p>
                                <b>Average</b>
                            </p>
                        </td>
                        <td>
                            <p>
                                <b>% Change Jan-Jul</b>
                            </p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>Anthropic</b>
                            </p>
                        </td>
                        <td>
                            <p>286,930.1</p>
                        </td>
                        <td>
                            <p>271,748.2</p>
                        </td>
                        <td>
                            <p>121,612.7</p>
                        </td>
                        <td>
                            <p>130,330.2</p>
                        </td>
                        <td>
                            <p>114,313</p>
                        </td>
                        <td>
                            <p>71,282.8</p>
                        </td>
                        <td>
                            <p>38,065.7</p>
                        </td>
                        <td>
                            <p>147,754.7</p>
                        </td>
                        <td>
                            <p>-86.7%</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>OpenAI</b>
                            </p>
                        </td>
                        <td>
                            <p>1,217.4</p>
                        </td>
                        <td>
                            <p>1,774.5</p>
                        </td>
                        <td>
                            <p>2,217</p>
                        </td>
                        <td>
                            <p>1200</p>
                        </td>
                        <td>
                            <p>995.6</p>
                        </td>
                        <td>
                            <p>1,655.9</p>
                        </td>
                        <td>
                            <p>1,091.4</p>
                        </td>
                        <td>
                            <p>1,437.8</p>
                        </td>
                        <td>
                            <p>-10.4%</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>Perplexity</b>
                            </p>
                        </td>
                        <td>
                            <p>54.6</p>
                        </td>
                        <td>
                            <p>55.3</p>
                        </td>
                        <td>
                            <p>201.3</p>
                        </td>
                        <td>
                            <p>300.9</p>
                        </td>
                        <td>
                            <p>199.1</p>
                        </td>
                        <td>
                            <p>200.6</p>
                        </td>
                        <td>
                            <p>194.8</p>
                        </td>
                        <td>
                            <p>172.4</p>
                        </td>
                        <td>
                            <p>256.7%</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>Microsoft</b>
                            </p>
                        </td>
                        <td>
                            <p>38.5</p>
                        </td>
                        <td>
                            <p>44.2</p>
                        </td>
                        <td>
                            <p>42.3</p>
                        </td>
                        <td>
                            <p>43.3</p>
                        </td>
                        <td>
                            <p>45.1</p>
                        </td>
                        <td>
                            <p>42</p>
                        </td>
                        <td>
                            <p>40.7</p>
                        </td>
                        <td>
                            <p>42.3</p>
                        </td>
                        <td>
                            <p>5.7%</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>Yandex</b>
                            </p>
                        </td>
                        <td>
                            <p>15.5</p>
                        </td>
                        <td>
                            <p>13.1</p>
                        </td>
                        <td>
                            <p>13.1</p>
                        </td>
                        <td>
                            <p>15.7</p>
                        </td>
                        <td>
                            <p>14.7</p>
                        </td>
                        <td>
                            <p>15.9</p>
                        </td>
                        <td>
                            <p>21.4</p>
                        </td>
                        <td>
                            <p>15.6</p>
                        </td>
                        <td>
                            <p>38.3%</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>Google</b>
                            </p>
                        </td>
                        <td>
                            <p>3.8</p>
                        </td>
                        <td>
                            <p>6.3</p>
                        </td>
                        <td>
                            <p>14.6</p>
                        </td>
                        <td>
                            <p>22.5</p>
                        </td>
                        <td>
                            <p>16.7</p>
                        </td>
                        <td>
                            <p>13.1</p>
                        </td>
                        <td>
                            <p>5.4</p>
                        </td>
                        <td>
                            <p>11.8</p>
                        </td>
                        <td>
                            <p>43%</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>ByteDance</b>
                            </p>
                        </td>
                        <td>
                            <p>18</p>
                        </td>
                        <td>
                            <p>16.4</p>
                        </td>
                        <td>
                            <p>3.5</p>
                        </td>
                        <td>
                            <p>2.3</p>
                        </td>
                        <td>
                            <p>1.6</p>
                        </td>
                        <td>
                            <p>1.6</p>
                        </td>
                        <td>
                            <p>0.9</p>
                        </td>
                        <td>
                            <p>6.3</p>
                        </td>
                        <td>
                            <p>-95%</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>Baidu</b>
                            </p>
                        </td>
                        <td>
                            <p>0.6</p>
                        </td>
                        <td>
                            <p>0.7</p>
                        </td>
                        <td>
                            <p>0.8</p>
                        </td>
                        <td>
                            <p>1.5</p>
                        </td>
                        <td>
                            <p>1.2</p>
                        </td>
                        <td>
                            <p>1</p>
                        </td>
                        <td>
                            <p>0.9</p>
                        </td>
                        <td>
                            <p>1</p>
                        </td>
                        <td>
                            <p>44.5%</p>
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <p>
                                <b>DuckDuckGo</b>
                            </p>
                        </td>
                        <td>
                            <p>0.1</p>
                        </td>
                        <td>
                            <p>0.2</p>
                        </td>
                        <td>
                            <p>0.2</p>
                        </td>
                        <td>
                            <p>0.2</p>
                        </td>
                        <td>
                            <p>0.3</p>
                        </td>
                        <td>
                            <p>0.3</p>
                        </td>
                        <td>
                            <p>0.3</p>
                        </td>
                        <td>
                            <p>0.2</p>
                        </td>
                        <td>
                            <p>116.3%</p>
                        </td>
                    </tr>
                </tbody>
            </table>
            <p>Looking at the changes from January to July 2025:</p>
            <ul>
                <li>
                    <p>
                        <b>Anthropic</b> recorded the steepest decrease in bot to human traffic,
                        down <b>86.7%</b>. From 286,930 bots per human in January, to 38,065 bots
                        per human in July, the change shows a dramatic increase in referrals.
                        Despite the change, it remains by far the most crawl-heavy platform, with
                        tens of thousands of pages still crawled for every referral.
                    </p>
                </li>
                <li>
                    <p>
                        <b>Perplexity</b> moved in the opposite direction, with bot crawling
                        increasing <b>+256.7%</b> relative to human visitors; climbing from
                        <b>54 bots per human</b> in January to <b>195 bots per human</b> in July.
                        While the ratio is still far below Anthropic, the increase shows it is
                        crawling more heavily, relative to the traffic it refers, than it did
                        earlier.
                    </p>
                </li>
                <li>
                    <p>
                        <b>OpenAI</b> ratio dropped slightly, from 1,217 bots per human in January
                        to 1,091 in July (-10%). The shift is smaller than Anthropic’s but suggests
                        OpenAI is sending a bit more referral traffic relative to its crawling.
                    </p>
                </li>
                <li>
                    <p>
                        <b>Microsoft</b> stayed steady, with its ratio moving only slightly, from
                        38.5 bots per human in January to 40.7 in July (+6%). This consistency
                        suggests stable behavior from Bing-linked services.
                    </p>
                </li>
                <li>
                    <p>
                        <b>Yandex</b> increased from 15.5 bots per human in January to 21.4 in July
                        (+38%). The overall ratio is far smaller than Anthropic’s or Perplexity’s,
                        but it shows Yandex is crawling more heavily relative to the traffic it
                        sends back.
                    </p>
                </li>
            </ul>
            <p>
                Alongside measuring crawling volumes and referral traffic (now also visible on the<a
                    href="https://radar.cloudflare.com/ai-insights#ai-bot-best-practices"
                >
                    <u> AI Insights page of Cloudflare Radar</u> </a
                >), it’s worth looking at whether AI operators follow good practices when deploying
                their bots. Cloudflare data shows that most leading AI crawlers are on our
                <a href="https://radar.cloudflare.com/bots#verified-bots">
                    <u>verified bots</u>
                </a>
                list, meaning their IP addresses match published ranges and they respect robots.txt.
                But adoption of newer standards like
                <a
                    href="https://developers.cloudflare.com/bots/concepts/bot/verified-bots/web-bot-auth/"
                >
                    <u>WebBotAuth</u>
                </a>
                — which uses cryptographic signatures in HTTP messages to confirm a request comes
                from a specific bot, and is especially relevant today — is still missing.&nbsp;
            </p>
            <p>
                Google, Meta, and OpenAI run distinct bots for different purposes, while Anthropic
                lags in verification. That makes it easier for bad actors to spoof its crawler and
                ignore robots.txt, since without verification, it’s hard to distinguish real from
                fake traffic — leaving its compliance effectively unclear. (A longer list of AI bots
                is available
                <a href="https://radar.cloudflare.com/ai-insights#ai-bot-best-practices">
                    <u>here</u> </a
                >).
            </p>
            <figure class="kg-card kg-image-card">
                <img
                    src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/4EvNGFKp6pGQUP84P33qJG/b646c0aad05d68d3f9c4a37d08bd483f/8.png"
                    alt=""
                    class="kg-image"
                    width="1600"
                    height="900"
                    loading="lazy"
                />
            </figure>
            <div class="anchor relative flex">
                <h2 id="conclusion-and-whats-next">Conclusion and what’s next</h2>
                <a
                    href="#conclusion-and-whats-next"
                    aria-hidden="true"
                    class="relative sm:absolute sm:-left-5"
                >
                    <svg width="16" height="16" viewBox="0 0 24 24">
                        <path
                            fill="currentcolor"
                            d="m12.11 15.39-3.88 3.88a2.52 2.52 0 0 1-3.5 0 2.47 2.47 0 0 1 0-3.5l3.88-3.88a1 1 0 0 0-1.42-1.42l-3.88 3.89a4.48 4.48 0 0 0 6.33 6.33l3.89-3.88a1 1 0 1 0-1.42-1.42Zm8.58-12.08a4.49 4.49 0 0 0-6.33 0l-3.89 3.88a1 1 0 0 0 1.42 1.42l3.88-3.88a2.52 2.52 0 0 1 3.5 0 2.47 2.47 0 0 1 0 3.5l-3.88 3.88a1 1 0 1 0 1.42 1.42l3.88-3.89a4.49 4.49 0 0 0 0-6.33ZM8.83 15.17a1 1 0 0 0 1.1.22 1 1 0 0 0 .32-.22l4.92-4.92a1 1 0 0 0-1.42-1.42l-4.92 4.92a1 1 0 0 0 0 1.42Z"
                        ></path>
                    </svg>
                </a>
            </div>
            <p>
                If training-related crawling continues to dominate while referrals stay flat,
                creators face a paradox: feeding AI systems without gaining traffic in return. Many
                want their content to appear in chatbot answers, but without monetization or
                cooperation, the incentive to produce quality work declines.
            </p>
            <p>
                The Web now stands at a fork in the road. Either a new balance emerges — one where
                the new AI era helps sustain publishers and creators — or AI turns the open web into
                a one-way training set, extracting value with little flowing back.
            </p>
            <p>
                You can learn more about some of these data trends on Cloudflare Radar’s updated<a
                    href="https://radar.cloudflare.com/ai-insights"
                >
                    <u> AI Insights page</u> </a
                >.
            </p>
        </div>
    </section>
    <section class="post-full-content mw7 center mb4 flex flex-row flex-wrap">
        <div class="post-content lh-copy gray1 bt b--gray8 pt4 w-100">
            Cloudflare's connectivity cloud protects
            <a target="_blank" href="https://www.cloudflare.com/network-services/" rel="noreferrer"
                >entire corporate networks</a
            >, helps customers build
            <a target="_blank" href="https://workers.cloudflare.com/" rel="noreferrer"
                >Internet-scale applications efficiently</a
            >, accelerates any
            <a
                target="_blank"
                href="https://www.cloudflare.com/performance/accelerate-internet-applications/"
                rel="noreferrer"
                >website or Internet application</a
            >,<a target="_blank" href="https://www.cloudflare.com/ddos/" rel="noreferrer"
                >wards off DDoS attacks</a
            >, keeps
            <a
                target="_blank"
                href="https://www.cloudflare.com/application-security/"
                rel="noreferrer"
                >hackers at bay</a
            >, and can help you on
            <a
                target="_blank"
                href="https://www.cloudflare.com/products/zero-trust/"
                rel="noreferrer"
                >your journey to Zero Trust</a
            >.<br />
            <br />Visit
            <a target="_blank" href="https://one.one.one.one/" rel="noreferrer">1.1.1.1</a> from any
            device to get started with our free app that makes your Internet faster and safer.<br />
            <br />To learn more about our mission to help build a better Internet,
            <a
                target="_blank"
                href="https://www.cloudflare.com/learning/what-is-cloudflare/"
                rel="noreferrer"
                >start here</a
            >. If you're looking for a new career direction, check out
            <a target="_blank" href="http://www.cloudflare.com/careers" rel="noreferrer"
                >our open positions</a
            >.
        </div>
    </section>
    <div class="pv2 ph0-l mw7 center" id="social-buttons">
        <div class="mt5-l mt2 mb4 f2 flex-row-ns flex-column flex flex-wrap">
            <a
                id="social-button-hn"
                title="Discuss on Hacker News"
                href="https://news.ycombinator.com/submitlink?u=https://blog.cloudflare.com/crawlers-click-ai-bots-training"
                target="_blank"
                rel="noreferrer"
                class="mr2-ns mb0-l white link b pv3 ph3 mb3"
                style="background-color: rgb(0, 85, 220)"
            >
                <svg
                    version="1.1"
                    id="Layer_1"
                    xmlns="http://www.w3.org/2000/svg"
                    x="0px"
                    y="0px"
                    viewBox="0 0 512 512"
                    class="mr2"
                >
                    <g>
                        <path
                            d="M31,31v450h450V31H31z M270.1,287.6v94.9h-28.1v-94.9L165,143.5h31.9L256,254.3l59.1-110.8H347
C347,143.5,270.1,287.6,270.1,287.6z"
                        ></path>
                    </g>
                </svg>
                <span class="v-mid">Discuss on Hacker News</span>
            </a>
        </div>
    </div>
    <iframe
        sandbox="allow-scripts allow-popups allow-popups-to-escape-sandbox"
        title="cloudflare-tv-live-link"
        id="cloudflare-tv-embed"
        src="https://cloudflare.tv/embed/live.html"
        loading="lazy"
    ></iframe>
    <a href="/tag/ai-week/" class="dib pl2 pr2 pt1 pb1 mb2 bg-gray8 blue3 f2 mr1 no-underline"
        >AI Week</a
    >
    <a href="/tag/ai/" class="dib pl2 pr2 pt1 pb1 mb2 bg-gray8 blue3 f2 mr1 no-underline">AI</a>
    <a
        href="/tag/cloudflare-radar/"
        class="dib pl2 pr2 pt1 pb1 mb2 bg-gray8 blue3 f2 mr1 no-underline"
        >Radar</a
    >
    <a
        href="/tag/internet-trends/"
        class="dib pl2 pr2 pt1 pb1 mb2 bg-gray8 blue3 f2 mr1 no-underline"
        >Internet Trends</a
    >
    <a href="/tag/traffic/" class="dib pl2 pr2 pt1 pb1 mb2 bg-gray8 blue3 f2 mr1 no-underline"
        >Traffic</a
    >
    <a href="/tag/bots/" class="dib pl2 pr2 pt1 pb1 mb2 bg-gray8 blue3 f2 mr1 no-underline">Bots</a>
</article>