Summary
Contao's crawler tries to prevent confidential HTTP client options from being sent to external domains by creating a scoped client: full options for root page origins, cleaned options for everything else. The cleaner removes Cookie and Authorization headers, but it removes the non-Symfony option names basic_auth and bearer_auth instead of Symfony HttpClient's real auth_basic and auth_bearer options.
When contao.crawl.default_http_client_options contains Basic or Bearer authentication for a protected staging/production site, those credentials remain in the "clean" client used for external links or configured additional URIs. An attacker who can get an external URL crawled, for example through a link on a crawled page while the broken-link checker is enabled, can receive the crawler credentials.
Technical Detail
Root Cause
// core-bundle/src/Crawl/Escargot/Factory.php:175-209 @ e550b92a01ef625bd546e6c3956dd200af05ebf0
private function createHttpClient(array $options = []): HttpClientInterface
{
$options = array_merge_recursive(
[
'headers' => [
'accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'user-agent' => self::USER_AGENT,
],
'max_duration' => 10,
],
array_merge_recursive($this->getDefaultHttpClientOptions(), $options),
);
$cleanOptions = $this->cleanOptionsFromConfidentialData($options);
if ($options === $cleanOptions) {
return ($this->httpClientFactory)($options);
}
$scopedOptionsByRegex = [];
foreach ($this->getRootPageUriCollection()->all() as $rootPageUri) {
$scopedOptionsByRegex[preg_quote($this->getOriginFromUri($rootPageUri))] = $options;
}
return new ScopingHttpClient(($this->httpClientFactory)($cleanOptions), $scopedOptionsByRegex);
}
// core-bundle/src/Crawl/Escargot/Factory.php:226-247 @ e550b92a01ef625bd546e6c3956dd200af05ebf0
foreach ($options as $k => $v) {
if ('headers' === $k) {
foreach ($v as $header => $value) {
if (\in_array(strtolower($header), ['authorization', 'cookie'], true)) {
continue;
}
$cleanOptions['headers'][$header] = $value;
}
continue;
}
if ('basic_auth' === $k || 'bearer_auth' === $k) {
continue;
}
$cleanOptions[$k] = $v;
}
Symfony HttpClient authentication options are auth_basic and auth_bearer; Contao's own manual documents auth_basic for crawler Basic Authentication. Because the cleaner only strips basic_auth and bearer_auth, the "clean" default client for non-root-page hosts still carries the real auth options. The existing factory test intends to assert that Authorization is not sent to www.foreign-domain.com, but its mock client factory ignores the $defaultOptions argument, so it does not catch auth options that survive into HttpClient::create($cleanOptions).
Suggested Mitigation
Strip the actual Symfony HttpClient authentication option keys from the clean client. Include NTLM as a defensive extension because Symfony documents it as another auth option.
- if ('basic_auth' === $k || 'bearer_auth' === $k) {
+ if (\in_array($k, ['auth_basic', 'auth_bearer', 'auth_ntlm', 'basic_auth', 'bearer_auth'], true)) {
continue;
}
Also update the factory test so the mock factory records or preserves $defaultOptions; otherwise the test does not verify what HttpClient::create($cleanOptions) receives in production.
Impact
- Direct primitive: disclosure of crawler Basic/Bearer credentials to an external host reached by the crawler.
- Chain potential: if those credentials protect a staging or pre-publication environment, an attacker can use them to access that environment. The impact depends on what the leaked credential unlocks.
- Realistic exploitation: a content editor adds a link to
https://attacker.example/probe on a page that the crawler visits. When an administrator or scheduled maintenance run starts the broken-link checker with crawler Basic/Bearer authentication configured, the request to the attacker URL includes the generated Authorization header.
References
Summary
Contao's crawler tries to prevent confidential HTTP client options from being sent to external domains by creating a scoped client: full options for root page origins, cleaned options for everything else. The cleaner removes
CookieandAuthorizationheaders, but it removes the non-Symfony option namesbasic_authandbearer_authinstead of Symfony HttpClient's realauth_basicandauth_beareroptions.When
contao.crawl.default_http_client_optionscontains Basic or Bearer authentication for a protected staging/production site, those credentials remain in the "clean" client used for external links or configured additional URIs. An attacker who can get an external URL crawled, for example through a link on a crawled page while the broken-link checker is enabled, can receive the crawler credentials.Technical Detail
Root Cause
Symfony HttpClient authentication options are
auth_basicandauth_bearer; Contao's own manual documentsauth_basicfor crawler Basic Authentication. Because the cleaner only stripsbasic_authandbearer_auth, the "clean" default client for non-root-page hosts still carries the real auth options. The existing factory test intends to assert thatAuthorizationis not sent towww.foreign-domain.com, but its mock client factory ignores the$defaultOptionsargument, so it does not catch auth options that survive intoHttpClient::create($cleanOptions).Suggested Mitigation
Strip the actual Symfony HttpClient authentication option keys from the clean client. Include NTLM as a defensive extension because Symfony documents it as another auth option.
Also update the factory test so the mock factory records or preserves
$defaultOptions; otherwise the test does not verify whatHttpClient::create($cleanOptions)receives in production.Impact
https://attacker.example/probeon a page that the crawler visits. When an administrator or scheduled maintenance run starts the broken-link checker with crawler Basic/Bearer authentication configured, the request to the attacker URL includes the generatedAuthorizationheader.References