Feature Request: Add support for authentication for Elasticsearch

[karatekaneen]

TL;DR: It would be nice to be able to pass headers to Elasticsearch

Description

When trying out the changes from #92 I provisioned an Elasticsearch instance with auth enabled and it did not work as expected. Since there are no way to add custom headers I tried to use the URL basic authentication instead but it did not work.

Values I tried:

• https://elastic.company.internal:9200 - "failed: [security_exception] missing authentication credentials for REST request"
• https://username:password@elastic.company.internal:9200 - "invalid IPv6 address literal"
• username:password@elastic.company.internal:9200 - "invalid IPv6 address literal"

Stacktrace

ERROR [2024-04-23 13:24:42,129] com.scienceminer.glutton.indexing.ElasticSearchIndexer: Create index failed
! java.net.UnknownHostException: elastic:password@elastic-1.company.internal: invalid IPv6 address literal
! at java.base/java.net.InetAddress.invalidIPv6LiteralException(InetAddress.java:1391)
! at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1361)
! at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1307)
! at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
! at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$InternalAddressResolver.resolveRemoteAddress(PoolingNHttpClientConnectionManager.java:666)
! at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$InternalAddressResolver.resolveRemoteAddress(PoolingNHttpClientConnectionManager.java:637)
! at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:474)
! at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:280)
! at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:295)
! at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:381)
! at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:130)
! at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.execute(CloseableHttpAsyncClientBase.java:116)
! at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:138)
! at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
! ... 12 common frames omitted
! Causing: java.io.IOException: elastic:password@elastic-1.company.internal: invalid IPv6 address literal
! at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:939)
! at org.elasticsearch.client.RestClient.performRequest(RestClient.java:304)
! at org.elasticsearch.client.RestClient.performRequest(RestClient.java:292)
! at co.elastic.clients.transport.rest_client.RestClientHttpClient.performRequest(RestClientHttpClient.java:91)
! at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest(ElasticsearchTransportBase.java:144)
! at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.create(ElasticsearchIndicesClient.java:266)
! at com.scienceminer.glutton.indexing.ElasticSearchIndexer.createIndex(ElasticSearchIndexer.java:123)
! at com.scienceminer.glutton.indexing.ElasticSearchIndexer.setupIndex(ElasticSearchIndexer.java:338)
! at com.scienceminer.glutton.command.LoadCrossrefCommand.run(LoadCrossrefCommand.java:94)
! at com.scienceminer.glutton.command.LoadCrossrefCommand.run(LoadCrossrefCommand.java:50)
! at io.dropwizard.core.cli.ConfiguredCommand.run(ConfiguredCommand.java:98)
! at io.dropwizard.core.cli.Cli.run(Cli.java:78)
! at io.dropwizard.core.Application.run(Application.java:94)
! at com.scienceminer.glutton.web.LookupServiceApplication.main(LookupServiceApplication.java:200)

Proposed solution

Add possibility to add custom headers via the Elastic field in the config or add fields specific for authentication. Would be extra nice if credentials could be loaded from env variables.