Ambiguous data in IP address lists

[vel21ripn]

The main issue is with the AMAZON_AWS and AWS_EC2 addresses.
2284 addresses are simultaneously in different lists.

grep 3.4.16.0/20 *
ndpi_amazon_aws_ec2_match.c.inc: { 0x03041000 /* 3.4.16.0/20 */, 20, NDPI_PROTOCOL_AWS_EC2 },
ndpi_amazon_aws_match.c.inc: { 0x03041000 /* 3.4.16.0/20 */, 20, NDPI_PROTOCOL_AMAZON_AWS },

It's unclear how to resolve this.
The second issue is TOR and WHATSAPP_CALL. 1009 addresses are simultaneously in different lists.
IMHO, addresses from the TOR list should not be included in other lists.
The full list is attached.

ip_dup_list.txt

[IvanNardi]

@vel21ripn, sorry for the delay

The main issue is with the AMAZON_AWS and AWS_EC2 addresses. 2284 addresses are simultaneously in different lists.

We had a similar issue in the past with Microsoft addresses; I will try to come up with a similar solution

It's unclear how to resolve this. The second issue is TOR and WHATSAPP_CALL. 1009 addresses are simultaneously in different lists. IMHO, addresses from the TOR list should not be included in other lists. The full list is attached.

We don't have a list of WHATSAPP_CALL addresses!!
I think the confusion is caused by the fact that we have two types of lists:

• the list of ips for protocol classification (list of ip/id pairs)
• the list of ips for setting flow risks (list of ip/flow_risk pairs)

I think you tried to load the second type of lists as if they were the first type....

[vel21ripn]

I found a bug in the address list parsing scripts.
There was an error in the script that caused all address lists to be merged.
So far, the issue only persists with the AMAZON_AWS and AWS_EC2 addresses.

[IvanNardi]

There was an error in the script that caused all address lists to be merged.

If the issue is still present upstrem, could you push a PR with the fix, please?

[vel21ripn]

This script is not part of nDPI.
If you're interested, check out my fork: https://github.com/vel21ripn/nDPI/tree/flow_info-4/utils/ndpi_network_list_compile*

To compile, run ndpi_network_list_compile.sh.
Running ./ndpi_network_list_compile -v 2>&1 >/dev/zero | grep -e '!=' shows address conflicts in different lists.