Skip to content

For "homoglyph" fuzzer consider Unicode glyphs from confusables (by Unicode consortium) #241

Description

@sateeshkumarb

When dnstwist is invoked with fuzzer option homoglyph it uses the map glyphs_unicode (https://github.com/elceef/dnstwist/blob/master/dnstwist.py#L615) as a basis for finding visually equivalent Unicode characters for corresponding ascii characters.

The Unicode consortium publishes https://unicode.org/Public/security/latest/confusables.txt
which lists codes which are visually confusable. The mapping used by dnstwist can be enhanced
to make use of the visually confsuable characters from the above document.

Additionally, some of the Unicode characters that are visually similar to ascii codes (which are mentioned in the above document)
aren't IDNA complaint, but given the evolving threat landscape it might be good to include those Unicode characters
too in the fuzzed output generated (when the fuzzer is homoglyph).

Below script compares the visually matching glyphs (to ascii characters) derived from "confusables.txt"
(https://unicode.org/Public/security/latest/confusables.txt) (Unicode version 17.0.0) with Unicode glyph mapping
used by "dnstwist" for some of the well known domains.

Code from dnstwist version 20250130 was used for below script.

The resulting output shows a very low percentage of overlap between glyphs from confusable and glyphs dnstwist uses.

import string
from pathlib import Path
from pprint import pprint

import dnstwist

def glyphs_from_confusables():
    # confusables.txt downloaed from:
    # https://unicode.org/Public/security/latest/confusables.txt
    confusables_path = Path("confusables.txt")
    confusable_glyphs = {}

    for c in string.ascii_lowercase + string.digits:
        confusable_glyphs[c] = []

    with open(confusables_path, mode="r", encoding="UTF-8") as fr:
        for line in fr.readlines():
            line = line.strip()
            if not line or line.startswith("#"):
                continue

            src, target, _ = line.split(";", 2)
            for t in [x for x in target.split(" ") if x]:
                try:
                    t_character = chr(int(t, 16))
                except ValueError:
                    continue  # skip malformed entries

                # char_to_index now has entries for all ascii chars, so we are interested
                # in adding unicode entries that can be confused with these.
                if t_character in confusable_glyphs:
                    s_character = chr(int(src.strip(), 16))
                    # s_character is the unicode char that could be confused with ascii char (t_character)
                    confusable_glyphs[t_character].append(s_character)
    # pprint(confusable_glyphs, compact=True)
    return confusable_glyphs


def validate():
    dnstwist_glyphs = dnstwist.Fuzzer.glyphs_unicode
    confusable_glyphs = glyphs_from_confusables()

    results = {}
    test_domains = [
        "go.com",
        "fb.com",
        "paypal.com",
        "stripe.com",
        "revolut.com",
        "apple.com",
        "google.com",
        "microsoft.com",
        "amazon.com",
        "instagram.com",
        "twitter.com",
        "linkedin.com",
        "github.com",
    ]

    for domain in test_domains:
        for char in set(domain.replace(".", "")):
            dt_set = set(dnstwist_glyphs.get(char, []))
            cf_set = set(confusable_glyphs.get(char, []))

            overlap = dt_set & cf_set
            dt_only = dt_set - cf_set
            cf_only = cf_set - dt_set

            results[char] = {
                # uncomment to see the overlapping and non-overlapping charsets
                # 'dnstwist_chars': dt_set,
                # 'confusables_chars': cf_set,
                # 'overlap_chars': overlap,
                # 'dnstwist_only_chars': dt_set - cf_set,
                # 'confusables_only_chars': cf_set - dt_set,
                # 'dnstwist_count': len(dt_set),
                # 'confusables_count': len(cf_set),
                # 'overlap_count': len(overlap),
                "dnstwist_only_count": len(dt_only),
                "confusables_only_count": len(cf_only),
                "overlap_pct": (
                    len(overlap) / len(dt_set | cf_set) if dt_set | cf_set else 0
                ),
            }
    pprint(results,indent=2)


if __name__ == "__main__":
    validate()

Output from above script


{ 'a': { 'confusables_only_count': 34,
         'dnstwist_only_count': 13,
         'overlap_pct': 0.020833333333333332},
  'b': { 'confusables_only_count': 40,
         'dnstwist_only_count': 5,
         'overlap_pct': 0.021739130434782608},
  'c': { 'confusables_only_count': 29,
         'dnstwist_only_count': 5,
         'overlap_pct': 0.05555555555555555},
  'd': { 'confusables_only_count': 31,
         'dnstwist_only_count': 6,
         'overlap_pct': 0.075},
  'e': { 'confusables_only_count': 26,
         'dnstwist_only_count': 12,
         'overlap_pct': 0.02564102564102564},
  'f': { 'confusables_only_count': 27,
         'dnstwist_only_count': 1,
         'overlap_pct': 0.034482758620689655},
  'g': { 'confusables_only_count': 19,
         'dnstwist_only_count': 7,
         'overlap_pct': 0.07142857142857142},
  'h': { 'confusables_only_count': 22,
         'dnstwist_only_count': 9,
         'overlap_pct': 0.06060606060606061},
  'i': { 'confusables_only_count': 54,
         'dnstwist_only_count': 12,
         'overlap_pct': 0.05714285714285714},
  'k': { 'confusables_only_count': 15,
         'dnstwist_only_count': 7,
         'overlap_pct': 0.0},
  'l': { 'confusables_only_count': 194,
         'dnstwist_only_count': 3,
         'overlap_pct': 0.010050251256281407},
  'm': { 'confusables_only_count': 0,
         'dnstwist_only_count': 5,
         'overlap_pct': 0.0},
  'n': { 'confusables_only_count': 50,
         'dnstwist_only_count': 9,
         'overlap_pct': 0.016666666666666666},
  'o': { 'confusables_only_count': 135,
         'dnstwist_only_count': 11,
         'overlap_pct': 0.020134228187919462},
  'p': { 'confusables_only_count': 35,
         'dnstwist_only_count': 2,
         'overlap_pct': 0.05128205128205128},
  'r': { 'confusables_only_count': 44,
         'dnstwist_only_count': 10,
         'overlap_pct': 0.05263157894736842},
  's': { 'confusables_only_count': 28,
         'dnstwist_only_count': 7,
         'overlap_pct': 0.05405405405405406},
  't': { 'confusables_only_count': 23,
         'dnstwist_only_count': 6,
         'overlap_pct': 0.03333333333333333},
  'u': { 'confusables_only_count': 34,
         'dnstwist_only_count': 16,
         'overlap_pct': 0.0196078431372549},
  'v': { 'confusables_only_count': 35,
         'dnstwist_only_count': 5,
         'overlap_pct': 0.024390243902439025},
  'w': { 'confusables_only_count': 27,
         'dnstwist_only_count': 8,
         'overlap_pct': 0.027777777777777776},
  'y': { 'confusables_only_count': 32,
         'dnstwist_only_count': 6,
         'overlap_pct': 0.09523809523809523},
  'z': { 'confusables_only_count': 22,
         'dnstwist_only_count': 6,
         'overlap_pct': 0.0967741935483871}}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions