Skip to content

@TeamHG-Memex TeamHG

Change the repository type filter

All

    Repositories list

    74 repositories

    • sitehound

      Public
      This is the facade for installation and access to the individual components
      Shell
      •
      Apache License 2.0
      •6•15•3•0•Updated Jan 22, 2026Jan 22, 2026

    • scrapy-kafka-export

      Public
      Scrapy extension which writes crawled items to Kafka
      Python
      •
      MIT License
      •8•30•5•0•Updated Jan 22, 2026Jan 22, 2026

    • privoxy

      Public
      Privoxy HTTP Proxy based on jess/privoxy
      Dockerfile
      •
      Other
      •2•6•3•0•Updated Jan 22, 2026Jan 22, 2026

    • url-summary

      Public
      Show summary of a large number of URLs in a Jupyter Notebook
      Python
      •
      MIT License
      •7•17•3•0•Updated Jan 22, 2026Jan 22, 2026

    • proxy-middleware

      Public
      Scrapy middleware that reads proxy config from settings
      Python
      •
      MIT License
      •4•4•3•0•Updated Jan 22, 2026Jan 22, 2026

    • tor-proxy

      Public
      a tor socks proxy docker image
      Dockerfile
      •
      Other
      •4•11•3•0•Updated Jan 22, 2026Jan 22, 2026

    • sitehound-backend

      Public
      Sitehound's backend
      HTML
      •
      Apache License 2.0
      •4•6•3•0•Updated Jan 22, 2026Jan 22, 2026

    • sitehound-frontend

      Public
      Site Hound (previously THH) is a Domain Discovery Tool
      topicsdomain-discovery
      HTML
      •
      Apache License 2.0
      •9•23•5•1•Updated Jan 22, 2026Jan 22, 2026

    • program-index

      Public
      A list of memex-related tools and their repository URLs
      MIT License
      •57•6•0•0•Updated Jan 22, 2026Jan 22, 2026

    • scrapy-crawl-once

      Public
      Scrapy middleware which allows to crawl only new content
      scrapy
      Python
      •
      MIT License
      •21•79•7•2•Updated Jan 22, 2026Jan 22, 2026

    • extract-html-diff

      Public
      extract difference between two html pages
      htmldiff
      HTML
      •
      MIT License
      •4•32•5•0•Updated Jan 22, 2026Jan 22, 2026

    • fuzzyset

      Public
      A simple fuzzy matching set for python strings
      Python
      •48•1•0•0•Updated Jan 22, 2026Jan 22, 2026

    • scrapy-rotating-proxies

      Public
      use multiple proxies with Scrapy
      proxyscrapy
      Python
      •
      MIT License
      •160•771•48•5•Updated Jan 22, 2026Jan 22, 2026

    • linkdepth

      Public
      [UNMAINTAINED] scrapy spider to check link depth over time
      Python
      •1•4•3•0•Updated Jan 22, 2026Jan 22, 2026

    • memex-cdr

      Public
      This repository hosts code and schema information related to the Memex Crawl Data Repository (CDR)
      Python
      •8•1•0•0•Updated Jan 22, 2026Jan 22, 2026

    • tensorboard_logger

      Public
      Log TensorBoard events without touching TensorFlow
      Python
      •
      MIT License
      •49•628•12•1•Updated Jan 22, 2026Jan 22, 2026

    • hh-deep-deep

      Public
      THH ↔ deep-deep integration
      public-wip
      Python
      •
      MIT License
      •2•3•2•0•Updated Jan 22, 2026Jan 22, 2026

    • hh-page-classifier

      Public
      Headless Horseman Page Classifier service
      Python
      •
      MIT License
      •4•6•3•0•Updated Jan 22, 2026Jan 22, 2026

    • html-text

      Public
      Extract text from HTML
      HTML
      •
      MIT License
      •22•134•15•0•Updated Jan 22, 2026Jan 22, 2026

    • eli5

      Public
      A library for debugging/inspecting machine learning classifiers and explaining their predictions
      pythonnlpdata-sciencemachine-learningscikit-learnxgboostlightgbminspectionexplanationcrfsuite
      Jupyter Notebook
      •
      MIT License
      •328•2.8k•148•14•Updated Jan 22, 2026Jan 22, 2026

    • soft404

      Public
      A classifier for detecting soft 404 pages
      Jupyter Notebook
      •13•58•6•1•Updated Jan 22, 2026Jan 22, 2026

    • scrapy-statsd

      Public
      Publish Scrapy stats to statsd daemon
      Python
      •
      MIT License
      •5•1•0•0•Updated Jan 22, 2026Jan 22, 2026

    • linkrot

      Public
      [UNMAINTAINED] A script (Scrapy spider) to check a list of URLs.
      Jupyter Notebook
      •3•4•3•0•Updated Jan 22, 2026Jan 22, 2026

    • web-page-annotator

      Public
      Annotate parts of web pages in the browser
      Python
      •
      MIT License
      •3•11•4•0•Updated Jan 22, 2026Jan 22, 2026

    • json-lines

      Public
      Read JSON lines (jl) files, including gzipped and broken
      Python
      •
      MIT License
      •8•35•5•0•Updated Jan 22, 2026Jan 22, 2026

    • scrapy-cdr

      Public
      Item definition and utils for storing items in CDR format for scrapy
      Python
      •
      MIT License
      •5•7•3•0•Updated Jan 22, 2026Jan 22, 2026

    • domain-discovery-crawler

      Public
      Broad crawler for domain discovery
      Python
      •
      MIT License
      •8•19•5•0•Updated Jan 22, 2026Jan 22, 2026

    • scrapy-inline-requests

      Public
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •27•1•0•0•Updated Jan 22, 2026Jan 22, 2026

    • sshadduser

      Public
      A simple tool to add a new user with OpenSSH keys.
      Python
      •
      MIT License
      •0•3•3•0•Updated Jan 22, 2026Jan 22, 2026

    • captcha-broad-crawl

      Public
      Broad crawl of onion sites in search for captchas
      Python
      •
      MIT License
      •2•3•2•0•Updated Jan 22, 2026Jan 22, 2026