Is there any condition for extract links feature?

[ZzMzaw]

Hello,

I was playing with feroxbuster on a challenge website with a custom directory listing.
The Extract link option is set to true.
Nevertheless, it seems feroxbuster doesn't add links from the HTML page when scanning a folder (with directory listing).

I was wondering if there is something I am not doing appropriately for the feature to be triggered.

Here is the command executed from a Kali Linux with its result (transformed to hide the real URL) and directory listing is possible on /admin/.

feroxbuster --url http://dummy.doesnotexist.org/folder/ --scan-dir-listings --wordlist=/usr/share/wordlists/seclists/Discovery/Web-Content/common.txt -s 200
                                                                                                                                                           
 ___  ___  __   __     __      __         __   ___
|__  |__  |__) |__) | /  `    /  \ \_/ | |  \ |__
|    |___ |  \ |  \ | \__,    \__/ / \ | |__/ |___
by Ben "epi" Risher 🤓                 ver: 2.13.0
───────────────────────────┬──────────────────────
 🎯  Target Url            │ http://dummy.doesnotexist.org/folder
 🚩  In-Scope Url          │ dummy.doesnotexist.org
 🚀  Threads               │ 50
 📖  Wordlist              │ /usr/share/wordlists/seclists/Discovery/Web-Content/common.txt
 👌  Status Codes          │ [200]
 💥  Timeout (secs)        │ 7
 🦡  User-Agent            │ feroxbuster/2.13.0
 💉  Config File           │ /etc/feroxbuster/ferox-config.toml
 🔎  Extract Links         │ true
 📂  Scan Dir Listings     │ true
 🏁  HTTP methods          │ [GET]
 🔃  Recursion Depth       │ 4
───────────────────────────┴──────────────────────
 🏁  Press [ENTER] to use the Scan Management Menu™
──────────────────────────────────────────────────
[####################] - 29s    14249/14249   0s      found:0       errors:0      
[####################] - 21s     4747/4747    223/s   http://dummy.doesnotexist.org/folder/ 
[####################] - 21s     4747/4747    225/s   http://dummy.doesnotexist.org/folder/.git/logs/ 
[####################] - 19s     4747/4747    253/s   http://dummy.doesnotexist.org/folder/admin/

Thanks in advance for your help and support.

[epi052]

howdy! if this is a HTB machine, can you lmk which one? It would make diagnosing the problem much simpler. One possibility just from what's shown: are the links you expect to see extracted within the same domain/subdomain as the target? If not, we may be finding them and then scope settings reject them.

[ZzMzaw]

It is not HTB, it is root-me.org (a French challenge platform) with free registration: https://www.root-me.org/en/Challenges/Web-Server/HTTP-Directory-indexing.

Links are not absolute so I would expect them not to be rejected unless I miss something to properly define the scope.

Just in case it can help, I attached the HTML sent back when requesting: http://dummy.doesnotexist.org/folder/admin/:
response.html

I'll remove it as soon as it is not useful anymore.

EDIT: I though it was an issue on my side, discovering (and really appreciating) feroxbuster but please let me know if you prefer me to file an issue instead of a discussion.

[epi052]

morning!

I'm not sure where the issue is yet.

Looking at the response provided, we're not going to detect it as directory listing outright. All we do for dir listing detection is examine the <title> tag for known standard dir listing strings e.g. Index of /, Directory Listing for /. The heuristic for dir listing detection could be improved to be more general.

the response you gave has the following title

<title>
Challenge01 [Root Me : plateforme d'apprentissage dédiée au Hacking et à la Sécurité de l'Information...]
</title>

However, regardless of the title, IF admin came from the wordlist and had a response code of 200 (based on your -s 200), it would have hit the 'normal' link extraction path. My current guess is that your -s 200 is limiting how /admin/ is processed. Try removing -s 200 and let ferox handle the auto-filtering of 404-like replies. Let me know how that works out

[ZzMzaw]

Hello,

Thanks for your feedback.
Dropping the -s 200 didn't help so much. After trying once, it just got rid of 503 (-C 503) to make
It seems that using --force-recursion allows better detection (at least for the main folder on which there is directory listing).

Nevertheless, I still cannot detect all the path from the directory listing.

I'll continue to play with feroxbuster and create an issue if I figure out that something is really going wrong.

Thanks for your help.

[epi052]

morning!

I added some new dir listing detection heuristics. your target page should trigger those checks, and then dirlisting link extraction should follow.

you can use a debug build to give it a try if you like: https://github.com/epi052/feroxbuster/actions/runs/19783486169

if you do, lmk how it goes.