Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added Major Changes #3

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

Added Major Changes #3

wants to merge 10 commits into from

Conversation

aneesh08192
Copy link

1. Multithreaded on

i. Crawling
ii. Downloading

2. Improved stability and speed of time

3. Added more info to README

Major Discovery:

  • The whole website can be crawled by img in the search bar.

End of Download view:

image

On crawl txt will be filled with grabbed URLS
To download media in 16 threads max
Default 100 threads can min or max based on your comforts
At point of 3, Sample download view of detail provided.
removal of unused modules on main.py
@aneesh08192
Copy link
Author

Is this change in review to merge in your code?

@aneesh08192
Copy link
Author

Can you check this and merge into your code? It is optimised.

@anburocky3
Copy link
Owner

Can you check this and merge into your code? It is optimised.

Hi, please give me a week to review and merge. Little busy on other works.

@aneesh08192
Copy link
Author

Any update, Please?

@anburocky3
Copy link
Owner

Why we need to have all images URL in actress.txt file?

@anburocky3
Copy link
Owner

@aneesh08192 Waiting for your input, to merge this into this repo.

@aneesh08192
Copy link
Author

aneesh08192 commented Dec 9, 2022

@anburocky3 Initially, I have been crawling all URLs and started to make a multi-thread on downloading parallel.
Example: 100 images at a time. (based on internet speed, I'm using 8MBps internet speed. Threads of 30 are fine for me, each set 30 images at a time)

@anburocky3
Copy link
Owner

Can we do it without saving URL in unnecessary txt file?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants