DNS resolve failed in hickory_dns

[pirDOL]

Bug report:

Setting ObjectStorage.client.hickory_dns to true leads to download failed error.

Expected behavior:

It should download successfully when setting ObjectStorage.client.hickory_dns to true.

Some diagnose：
set ObjectStorage.client.hickory_dns to false and download task succeeded
execute dig DOMAIN_NAME command and it succeeded which means the resolve.conf is right.

How to reproduce it:

Download any object from s3.

Environment:

• Client version: dfdaemon 1.0.32 (5c6daa7, 2025-10-21)
• OS: CentOS release 4.3 (Final)
• Kernel (e.g. uname -a): 4.14
• Others: NA

[imeoer]

Could you provide the detail download error message after setting ObjectStorage.client.hickory_dns = true?

[pirDOL]

Could you provide the detail download error message after setting ObjectStorage.client.hickory_dns = true?

dfget.log

  2025-11-11T10:20:19.941603119+08:00 ERROR  download task failed: status: Internal, message: "backend error: Unexpected (temporary) at stat, context: { url: http://***, called: http_util::Client::send, service: s3, path: *** } => send http request, source: error sending request for url (http://***)", details: {"message":"Unexpected (temporary) at stat, context: { url: http://***, called: http_util::Client::send, service: s3, path: *** } => send http request, source: error sending request for url (http://***)","header":{},"status_code":null}, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 11 Nov 2025 02:20:19 GMT", "content-length": "0"} }
    at dragonfly-client/src/bin/dfget/main.rs:907

Downloading Failed!
****************************************
Message: Unexpected (temporary) at stat, context: { url: http://***, called: http_util::Client::send, service: s3, path: *** } => send http request, source: error sending request for url (http://***)
****************************************

dfdaemon.log

2025-11-11T10:43:56.061208849+08:00 DEBUG download_task:download_started:head: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-0.12.4/src/connect.rs:497: starting new connection: http://***/ host_id="***-seed" task_id="a7b0694045d06051adfd10a27bd64c3dd22bc4e73bdf474899a99444cb88b7d0" peer_id="***-1cbf04be-3375-4b99-a07a-4fd3af92a212-seed" url="s3://***" remote_ip="***"
2025-11-11T10:43:56.061313319+08:00 ERROR download_task:download_started:head: dragonfly-client-backend/src/object_storage.rs:549: stat request failed a7b0694045d06051adfd10a27bd64c3dd22bc4e73bdf474899a99444cb88b7d0 s3://***: Unexpected (temporary) at stat, context: { url: http://***, called: http_util::Client::send, service: s3, path: *** } => send http request, source: error sending request for url (http://***) host_id="***-seed" task_id="a7b0694045d06051adfd10a27bd64c3dd22bc4e73bdf474899a99444cb88b7d0" peer_id="***-1cbf04be-3375-4b99-a07a-4fd3af92a212-seed" url="s3://***" remote_ip="***"
2025-11-11T10:43:56.061364145+08:00 ERROR download_task: dragonfly-client/src/grpc/dfdaemon_download.rs:305: download started failed by error: backend error: Unexpected (temporary) at stat, context: { url: http://***, called: http_util::Client::send, service: s3, path: *** } => send http request, source: error sending request for url (http://***) host_id="***-seed" task_id="a7b0694045d06051adfd10a27bd64c3dd22bc4e73bdf474899a99444cb88b7d0" peer_id="***-1cbf04be-3375-4b99-a07a-4fd3af92a212-seed" url="s3://***" remote_ip="***"

Some information in the above log ignored for private reason.
I have tested the url and the object in the url can be accessed successfully by aws-go-sdk.
I have used tcpdump to catch packets and it showed that no packet was sent which means that dns does not work properly very likely.
I have tested download from http and dns was resolved successfully.

In short, the problem seems that http + hickory_dns is ok but s3 + hichkory_dns is bad.

[pirDOL]

Could you provide the detail download error message after setting ObjectStorage.client.hickory_dns = true?

I have found the reason of this problem. There are customized directives which can not be deleted in the resolv.conf of production server which leads to the failure of the initialization of HickoryDnsResovler.

Althrough resolv-conf provides loose resolv.conf parse interface[https://github.com/hickory-dns/resolv-conf/pull/18], but because the deep dependency tree(dragonfly-client-backend -> reqwest -> hickory-dns -> resolv-conf) it seems complex to use the new interface.

Would it be possible to add an item in dfdaemon.yaml to select dns resolver, maybe as follows:

server:
  # pluginDir is the directory to store plugins.
  pluginDir: /var/lib/dragonfly/plugins/dfdaemon/
  # cacheDir is the directory to store cache files.
  cacheDir: /var/cache/dragonfly/dfdaemon/
  # dnsResolver is the name of dns resolver default: hickory.
  dnsResolver: hickory
...

The default value for dns.resolver could be hickory-dns for compatibility.

[pirDOL]

Could you provide the detail download error message after setting ObjectStorage.client.hickory_dns = true?

What's more, according to #1293, it seems that hickory is enabled for http backend. But after I added log to reqwest/src/dns/resolve.rs, I found that HickoryDnsResolver was not created when downloading from http. How about the test results for #1293 on http download? Does HickoryDnsResolver really work for http download?