IVY-1280 Support preemptive authentication

Author: twogee

asciidoc/release-notes.adoc

@@ -38,7 +38,7 @@ Key features of this 2.5.0 release are:
 
     * The minimum runtime Java version required is now Java 7
     * Ivy now uses BouncyCastle 1.58. Due to the non backward compatibility of that library, earlier versions are not supported.
-    * Ivy now uses HttpComponents HttpClient 4.5.x version when dealing with HTTP backed resolvers. Users are expected to have this version of the library (and its dependencies) in their runtime classpath if they want to use such resolvers. The previous (similarly named but not the same) commons-httpclient library is no longer used or supported. (jira:IVY-1563[])
+    * Ivy now uses HttpComponents HttpClient 4.5.x version with HTTP backed resolvers. Users are expected to have this version of the library (and its dependencies) in their runtime classpath if they want to use such resolvers. The previous (similarly named but not the same) commons-httpclient library is no longer used or supported. (jira:IVY-1563[])
 
 
 == List of Changes in this Release
@@ -86,6 +86,7 @@ For details about the following changes, check our JIRA install at link:https://
 - IMPROVEMENT: Support timestamped SNAPSHOT versions from Maven repository (jira:IVY-1476[])
 - IMPROVEMENT: Update Commons VFS to 2.1
 - IMPROVEMENT: Ivy now supports activating of Maven profiles, in `pom.xml`, by `jdk`, `os`, `property` and `file` (jira:IVY-1558[]) and (jira:IVY-1577[])
+- IMPROVEMENT: Support link:https://hc.apache.org/httpcomponents-client-4.5.x/tutorial/html/authentication.html[preemptive authentication]. (jira:IVY-1280[])
 
 - NEW: Lets SSH-based resolvers use an `~/.ssh/config` file to find username/hostname/keyfile options (Thanks to Colin Stanfill)
 - NEW: Add ivy.maven.lookup.sources and ivy.maven.lookup.javadoc variables to control the lookup of the additional artifacts. Defaults to true, for backward compatibility (jira:IVY-1529[])
@@ -188,6 +189,7 @@ Here is the list of people who have contributed source code and documentation up
 * Antoine Levy-Lambert
 * Tony Likhite
 * Andrey Lomakin
+* Aurélien Lourot
 * William Lyvers
 * Sakari Maaranen
 * Jan Materne

asciidoc/settings/settings.adoc

@@ -50,6 +50,7 @@ So if there is a setting in the resolver, it always wins against all other setti
 |validate|Indicates if Ivy files should be validated against ivy.xsd or not.|No, defaults to true
 |useRemoteConfig|true to configure ivyrep and ibiblio resolver from a remote settings file (updated with changes in those repository structure if any) (*__since 1.2__*)|No, defaults to false
 |httpRequestMethod|specifies the HTTP method to use to retrieve information about an URL. Possible values are 'GET' and 'HEAD'. This setting can be used to solve problems with firewalls and proxies. (*__since 2.0__*)|No, defaults to 'HEAD'
+|preemptiveAuth|true to use link:https://hc.apache.org/httpcomponents-client-4.5.x/tutorial/html/authentication.html[preemptive authentication] whenever possible (only supported with link:https://hc.apache.org/httpcomponents-client-4.5.x[HttpClient] as URL Handler) (*__since 2.5__*)|No, defaults to false
 |[line-through]#defaultCache#|a path to a directory to use as default basedir for both resolution and repository cache(s). +
 __Deprecated, we recommend using defaultCacheDir on the link:../settings/caches.html[caches] tag instead__|No, defaults to .ivy2/cache in user home
 |[line-through]#checkUpToDate#|Indicates if date should be checked before retrieving artifacts from cache. +

doc/settings/settings.html

@@ -61,6 +61,8 @@ <h1>Attributes</h1>
         <td>No, defaults to false</td></tr>
     <tr><td>httpRequestMethod</td><td>specifies the HTTP method to use to retrieve information about an URL. Possible values are 'GET' and 'HEAD'. This setting can be used to solve problems with firewalls and proxies. (<span class="since">since 2.0</span>)</td>
         <td>No, defaults to 'HEAD'</td></tr>
+    <tr><td>preemptiveAuth</td><td>true to use <a href="https://hc.apache.org/httpcomponents-client-4.5.x/tutorial/html/authentication.html">preemptive authentication</a> whenever possible (only supported with <a href="https://hc.apache.org/httpcomponents-client-4.5.x">HttpClient</a> as URL Handler) (<span class="since">since 2.5</span>)</td>
+        <td>No, defaults to false</td></tr>
     <tr><td><s>defaultCache</s></td><td>a path to a directory to use as default basedir for both resolution and repository cache(s).
 	    <i>Deprecated, we recommend using defaultCacheDir on the [[settings/caches]] tag instead</i></td>
         <td>No, defaults to .ivy2/cache in user home</td></tr>

src/java/org/apache/ivy/core/settings/XmlSettingsParser.java

@@ -380,6 +380,11 @@ private void settingsStarted(String qName, Map<String, String> attributes) {
             throw new IllegalArgumentException(
                     "Invalid httpRequestMethod specified, must be one of {'HEAD', 'GET'}");
         }
+
+        String preemptiveAuth = attributes.get("preemptiveAuth");
+        if (preemptiveAuth != null) {
+            URLHandlerRegistry.getHttp().setPreemptiveAuth(Boolean.valueOf(preemptiveAuth));
+        }
     }
 
     private void includeStarted(Map<String, String> attributes) throws IOException, ParseException {

src/java/org/apache/ivy/util/url/AbstractURLHandler.java

@@ -40,6 +40,8 @@ public abstract class AbstractURLHandler implements URLHandler {
     // the request method to use. TODO: don't use a static here
     private static int requestMethod = REQUEST_METHOD_HEAD;
 
+    private boolean preemptiveAuth = false;
+
     @Override
     public boolean isReachable(final URL url) {
         return getURLInfo(url).isReachable();
@@ -105,6 +107,14 @@ public int getRequestMethod() {
         return requestMethod;
     }
 
+    public void setPreemptiveAuth(boolean preemptive) {
+        this.preemptiveAuth = preemptive;
+    }
+
+    public boolean getPreemptiveAuth() {
+        return this.preemptiveAuth;
+    }
+
     protected String normalizeToString(URL url) throws IOException {
         if (!"http".equals(url.getProtocol()) && !"https".equals(url.getProtocol())) {
             return url.toExternalForm();

src/java/org/apache/ivy/util/url/HttpClientHandler.java

@@ -19,28 +19,33 @@
 
 import org.apache.http.Header;
 import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
 import org.apache.http.HttpResponse;
 import org.apache.http.HttpStatus;
 import org.apache.http.auth.AuthSchemeProvider;
 import org.apache.http.auth.AuthScope;
 import org.apache.http.auth.Credentials;
 import org.apache.http.auth.NTCredentials;
 import org.apache.http.client.CredentialsProvider;
+import org.apache.http.client.AuthCache;
 import org.apache.http.client.config.AuthSchemes;
 import org.apache.http.client.config.RequestConfig;
 import org.apache.http.client.methods.CloseableHttpResponse;
 import org.apache.http.client.methods.HttpGet;
 import org.apache.http.client.methods.HttpHead;
 import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.protocol.HttpClientContext;
 import org.apache.http.config.Lookup;
 import org.apache.http.config.RegistryBuilder;
 import org.apache.http.conn.HttpClientConnectionManager;
 import org.apache.http.conn.routing.HttpRoutePlanner;
 import org.apache.http.entity.ContentType;
 import org.apache.http.entity.FileEntity;
+import org.apache.http.impl.auth.BasicScheme;
 import org.apache.http.impl.auth.BasicSchemeFactory;
 import org.apache.http.impl.auth.DigestSchemeFactory;
 import org.apache.http.impl.auth.NTLMSchemeFactory;
+import org.apache.http.impl.client.BasicAuthCache;
 import org.apache.http.impl.client.CloseableHttpClient;
 import org.apache.http.impl.client.HttpClients;
 import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
@@ -186,7 +191,7 @@ public void upload(final File src, final URL dest, final CopyProgressListener li
         final HttpPut put = new HttpPut(normalizeToString(dest));
         put.setConfig(requestConfig);
         put.setEntity(new FileEntity(src));
-        try (final CloseableHttpResponse response = this.httpClient.execute(put)) {
+        try (final CloseableHttpResponse response = this.httpClient.execute(put, getHttpClientContext(dest))) {
             validatePutStatusCode(dest, response.getStatusLine().getStatusCode(), response.getStatusLine().getReasonPhrase());
         }
     }
@@ -326,7 +331,7 @@ private CloseableHttpResponse doGet(final URL url, final int connectionTimeout,
         final HttpGet httpGet = new HttpGet(normalizeToString(url));
         httpGet.setConfig(requestConfig);
         httpGet.addHeader("Accept-Encoding", "gzip,deflate");
-        return this.httpClient.execute(httpGet);
+        return this.httpClient.execute(httpGet, getHttpClientContext(url));
     }
 
     private CloseableHttpResponse doHead(final URL url, final int connectionTimeout, final int readTimeout) throws IOException {
@@ -338,13 +343,34 @@ private CloseableHttpResponse doHead(final URL url, final int connectionTimeout,
                 .build();
         final HttpHead httpHead = new HttpHead(normalizeToString(url));
         httpHead.setConfig(requestConfig);
-        return this.httpClient.execute(httpHead);
+        return this.httpClient.execute(httpHead, getHttpClientContext(url));
     }
 
     private boolean hasCredentialsConfigured(final URL url) {
         return CredentialsStore.INSTANCE.hasCredentials(url.getHost());
     }
 
+    private HttpClientContext getHttpClientContext(URL dest) {
+        if (!getPreemptiveAuth()) {
+            return null;
+        }
+
+        // Preemptive authentication, see
+        // https://hc.apache.org/httpcomponents-client-4.5.x/tutorial/html/authentication.html
+        // Without preemptive authentication, Ivy will first try non-authenticated, get a 4xx, retry
+        // authenticated and finally get a 2xx. This becomes an issue in an environment where Ivy is
+        // used a lot, as firewalls might see large amounts of 4xx as a brute-force attack and close
+        // the connections.
+
+        final BasicScheme basicAuth = new BasicScheme();
+        final HttpHost target = new HttpHost(dest.getHost(), dest.getPort(), dest.getProtocol());
+        final AuthCache authCache = new BasicAuthCache();
+        authCache.put(target, basicAuth);
+        final HttpClientContext localContext = HttpClientContext.create();
+        localContext.setAuthCache(authCache);
+        return localContext;
+    }
+
     @Override
     public void close() throws Exception {
         if (this.httpClient != null) {

src/java/org/apache/ivy/util/url/URLHandler.java

@@ -177,4 +177,6 @@ public String getBodyCharset() {
     void upload(File src, URL dest, CopyProgressListener l) throws IOException;
 
     void setRequestMethod(int requestMethod);
+
+    void setPreemptiveAuth(boolean preemptive);
 }

src/java/org/apache/ivy/util/url/URLHandlerDispatcher.java

@@ -167,6 +167,13 @@ public void setRequestMethod(int requestMethod) {
         }
     }
 
+    public void setPreemptiveAuth(boolean preemptive) {
+        defaultHandler.setPreemptiveAuth(preemptive);
+        for (URLHandler handler : handlers.values()) {
+            handler.setPreemptiveAuth(preemptive);
+        }
+    }
+
     public void setDownloader(String protocol, URLHandler downloader) {
         handlers.put(protocol, downloader);
     }