Skip to content

configuration.md

Latest commit

 

History

History
211 lines (157 loc) · 5.21 KB

configuration.md

File metadata and controls

211 lines (157 loc) · 5.21 KB
Raw

Configuration

Options

Pass *Options to any parse function to control extraction behavior. All fields are optional -- nil uses sensible defaults.

result, err := defuddle.ParseFromURL(ctx, url, &defuddle.Options{
    Markdown: true,
    Debug:    true,
})

Output Options

Markdown

// Include markdown alongside HTML
opts := &defuddle.Options{Markdown: true}
result, _ := defuddle.ParseFromURL(ctx, url, opts)
fmt.Println(*result.ContentMarkdown)

SeparateMarkdown

// Convert original HTML to markdown independently from cleaned content
opts := &defuddle.Options{SeparateMarkdown: true}

URL

Set the base URL for resolving relative links when parsing HTML strings:

d, _ := defuddle.NewDefuddle(html, &defuddle.Options{
    URL: "https://example.com/article",
})

Clutter Removal

Each removal stage runs independently. Disable specific stages to keep more content:

opts := &defuddle.Options{
    RemoveExactSelectors:   defuddle.PtrBool(false), // keep ads, social buttons
    RemovePartialSelectors: defuddle.PtrBool(false), // keep class/id pattern matches
    RemoveHiddenElements:   defuddle.PtrBool(false), // keep display:none elements
    RemoveLowScoring:       defuddle.PtrBool(false), // keep low-scoring blocks
    RemoveContentPatterns:  defuddle.PtrBool(false), // keep boilerplate text
}

All removal flags default to true when nil. Use defuddle.PtrBool(v) to set explicit values.

Stages

Stage Default What It Removes
ExactSelectors on Ads, social widgets, share buttons via exact CSS selectors
PartialSelectors on Elements matching ad/clutter patterns in class and id attributes
HiddenElements on Elements with display:none, visibility:hidden, or Tailwind hidden classes
LowScoring on Blocks that score below the content threshold (sidebars, footers, related articles)
ContentPatterns on Boilerplate text (read time, breadcrumbs, article cards)

Remove All Clutter Removal

// Equivalent to --no-clutter-removal in the CLI
opts := &defuddle.Options{
    RemoveExactSelectors:   defuddle.PtrBool(false),
    RemovePartialSelectors: defuddle.PtrBool(false),
    RemoveHiddenElements:   defuddle.PtrBool(false),
    RemoveLowScoring:       defuddle.PtrBool(false),
    RemoveContentPatterns:  defuddle.PtrBool(false),
}

Content Selection

ContentSelector

Force a specific element as the content root, bypassing auto-detection:

opts := &defuddle.Options{
    ContentSelector: "article.post-body",
}

RemoveImages

Strip all images from the extracted content:

opts := &defuddle.Options{RemoveImages: true}

HTTP Options

Custom Client

import (
    "net/http"
    "time"
)

client := &http.Client{Timeout: 60 * time.Second}
opts := &defuddle.Options{
    Client: client,
    Headers: http.Header{"User-Agent": []string{"MyBot/1.0"}},
}

MaxConcurrency

Controls parallel URL fetching in ParseFromURLs:

opts := &defuddle.Options{MaxConcurrency: 10} // default: 5

Element Processing

Enable specialized processors for specific content types via the public boolean flags. The per-processor option structs (CodeOptions, ImageOptions, MathOptions, FootnoteOptions, HeadingOptions, RoleOptions) reference types in an internal package and are not part of the public external API — external modules cannot import them. The boolean flags below enable each processor with sensible defaults and are the supported external surface.

Code Blocks

opts := &defuddle.Options{
    ProcessCode: true, // detect language and normalize code block whitespace
}

Images

opts := &defuddle.Options{
    ProcessImages: true, // normalize images and drop very small images
}

Math

opts := &defuddle.Options{
    ProcessMath: true, // extract MathML/LaTeX and clean up math scripts
}

Footnotes

opts := &defuddle.Options{
    ProcessFootnotes: true, // detect, link, number, and group footnotes into a section
}

Headings

opts := &defuddle.Options{
    ProcessHeadings: true,
}

ARIA Roles

opts := &defuddle.Options{
    ProcessRoles: true, // convert role="paragraph"/"list"/"button"/"link" to native elements
}

Debug Mode

opts := &defuddle.Options{Debug: true}
result, _ := defuddle.ParseFromURL(ctx, url, opts)

info := result.DebugInfo
fmt.Printf("Removed %d elements\n", info.Statistics.RemovedElementCount)
for _, step := range info.ProcessingSteps {
    fmt.Printf("  %s\n", step)
}

Debug output includes:

  • RemovedElements -- each element removed, with reason and selector
  • ProcessingSteps -- ordered list of pipeline steps executed
  • Timings -- nanosecond-precision timing for each stage
  • Statistics -- element counts, word count, image count, link count
  • ExtractorUsed -- name of the site extractor, if any

Defaults Summary

Option Default
Markdown false
SeparateMarkdown false
Debug false
RemoveImages false
All removal flags true (when nil)
All process flags false
MaxConcurrency 5
HTTP timeout 30s (library and CLI)
Max response size 5 MB