About Github/Virustotal Sources

[Bundy01]

Hi,

@dogancanbakir, For the Virustotal source, would it be possible to limit 40 results per request?
If so, that would solve the quota problem.

The Github source is quite slow. I noticed that the number of requests was very high despite sometimes low results (e.g., 15 results for 1,010 requests).

If this can be adjusted in the files for these sources, that would be great.

Regards.

[Bundy01]

For Virustotal ? See comments.

*** a/virustotal.go
--- b/virustotal.go
@@
 	go func() {
 		defer func(startTime time.Time) {
 			s.timeTaken = time.Since(startTime)
 			close(results)
 		}(time.Now())
-
-		randomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())
+		// MODIFICATION: enforce a global client-side cap of 40 results.
+		const maxResults = 40
+
+		randomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())
 		if randomApiKey == "" {
 			return
 		}
 		var cursor = ""
 		for {
 			select {
 			case <-ctx.Done():
 				return
 			default:
 			}

			var url = fmt.Sprintf("https://www.virustotal.com/api/v3/domains/%s/subdomains?limit=40", domain)
 			if cursor != "" {
 				url = fmt.Sprintf("%s&cursor=%s", url, cursor)
 			}
 			s.requests++
 			resp, err := session.Get(ctx, url, "", map[string]string{"x-apikey": randomApiKey})
 			if err != nil {
 				results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
 				s.errors++
 				session.DiscardHTTPResponse(resp)
 				return
 			}
 			defer func() {
 				if err := resp.Body.Close(); err != nil {
 					results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
 					s.errors++
 				}
 			}()
 
 			var data response
 			err = jsoniter.NewDecoder(resp.Body).Decode(&data)
 			if err != nil {
 				results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
 				s.errors++
 				return
 			}
 
 			for _, subdomain := range data.Data {
 				select {
 				case <-ctx.Done():
 					return
-				case results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain.Id}:
-					s.results++
+				case results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain.Id}:
+					s.results++
+				}
+				// MODIFICATION: stop the goroutine once the global cap is reached.
+				if s.results >= maxResults {
+					return
 				}
 			}
 			cursor = data.Meta.Cursor
 			if cursor == "" {
 				break
 			}
 		}
 	}()
 
 	return results
 }
// Name returns the name of the source
func (s *Source) Name() string {
	return "virustotal"
}

func (s *Source) IsDefault() bool {
	return true
}

func (s *Source) HasRecursiveSupport() bool {
	return true
}

func (s *Source) KeyRequirement() subscraping.KeyRequirement {
	return subscraping.RequiredKey
}

func (s *Source) NeedsKey() bool {
	return s.KeyRequirement() == subscraping.RequiredKey
}

func (s *Source) AddApiKeys(keys []string) {
	s.apiKeys = keys
}

func (s *Source) Statistics() subscraping.Statistics {
	return subscraping.Statistics{
		Errors:    s.errors,
		Results:   s.results,
		Requests:  s.requests,
		TimeTaken: s.timeTaken,
		Skipped:   s.skipped,
	}
}

I tested the code and it works. If you don't mind, can I make a PR?

[dogancanbakir]

This will always produce 40 results each time on VirusTotal, which means some other results might be missed. Regarding GitHub, did you look at the rate limit options? You can configure each one with the " -rls value maximum number of http requests to send per second for providers in key=value format (-rls "hackertarget=10/s,shodan=15/s")
"

[Bundy01]

@dogancanbakir, Yes, that would limit it to 40 results. This would allow testing up to 500 hosts (the limit of the free API).
So that premium users are not disadvantaged, I thought about adding a source called ‘virustotal-free’ if the name is not a problem.

However, the best thing would be to create an option in the config file and from the terminal to choose the number of results (in multiples of 40).

If this option is commented or not present in the command, the current behavior is preferred for premium users or when checking one or two hosts.

For example, I use a file containing about 110 hosts. With a limit of 80 results, I have noticed 170 requests. So in my case, I can attempt 120 or even 160 results before the limit of 500 is reached.

But I don't know how to add options to Subfinder. Let me know what you think, but the option would be the best solution to not use all the API credit on 7-8 hosts that have many results.

[Bundy01]

For Github,

rate-limits: ["chaos=60/m", "github=83/m", "hackertarget=2/s", "leakix=1/s", "netlas=1/s", "securitytrails=2/s", "virustotal=4/m", "waybackarchive=15/m"]

I don't know if this is how the API behaves, but there are often 1010 requests for a single tested host.

subfinder -s github -d example.com
…
[INF] Found 164 subdomains for example.com in 1 minute 50 seconds
[INF] Printing source statistics for example.com

 Source               Duration      Results   Requests     Errors
────────────────────────────────────────────────────────────────────
 github               1m50.982s         164       1010          0

[dogancanbakir]

The problem is there is no official doc pointing that you can configure the limit! Let me know if we're missing something!

[Bundy01]

Are you not enthusiastic about the idea of an option for the Virustotal source?

Personally, I prefer to test my entire hosts list rather than testing it partially. The other sources would compensate for the limitation of missing subdomains.

[dogancanbakir]

Well, I'm always open to ideas. Why don't you raise a PR and we discuss it then?

[Bundy01]

Hi, I've made good progress, but I don't know how to modify the config.yaml file automatically when Subfinder starts up. It would be to add an option directly. Do you have any suggestions? Thx