Skip to content

🚧 Implement Hashtag Scraping Functionality #11

Description

@itsKayWat

Goal: Enable the scraper to collect video URLs from TikTok hashtag pages.
Tasks:

  • Research TikTok's hashtag page structure and identify video elements.
  • Develop a method to paginate through hashtag pages to capture a large number of videos.
  • Implement code to extract video URLs from hashtag pages.
  • Add a command-line option or configuration setting to switch between channel and hashtag scraping modes.
  • Enhance URL Extraction

Goal: Provide more options for organizing downloaded video files by hashtag.
Tasks:

  • Add options to organize videos by date, uploader, or hashtag.
  • Implement a configurable naming scheme for downloaded files.
  • Update Documentation

Goal: Ensure the project is easy to use and understand.
Tasks:

  • Update the README file with instructions on how to use the new hashtag scraping feature.
  • Provide examples of how to use the scraper with both channel and hashtag URLs.
  • Document any new configuration options.
  • Testing and Bug Fixing

Goal: Improve user experience by providing flexibility in configuration
Tasks:

  • Allow users to specify the number of videos to scrape.
  • Implement options for setting download directory, number of threads, and other parameters via command line.
  • Consider a configuration file for persistent settings.
  • Implement Rate Limiting Handling

[ anon request ]
"For example, I want to crawl all video links from the hashtag #sontungmtp. Currently, this hashtag has 592.2K posts, but when scrolling manually on the website, I can only get around 300 videos.

If your application can actually retrieve information from all 592.2K posts, I would be very happy

I have a few questions:
1.) If I input the hashtag sontungmtp, is it really possible to get data from all 592.2K posts?
2.) Is it free or paid?"

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    Status
    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions