Stage Zero is completed, you can parse all http urls from a pcap file and filter them by their source or destination

Author: abzcoding

CHANGELOG.md

@@ -1,11 +1,15 @@
 # APTDetector Changelog
 
-Since February 07, 2016 there have been 1 releases and 13 commits for
+Since February 07, 2016 there have been 1 releases and 14 commits for
 
 an average of one 13-commit release every 1 weeks.
+## 0.1.4
+* [Stage Zero][stages] is now completed
+* [BaseSniffer][basesniffer] connections can now filter to show specifiec source or destination
+* fixed a problem in readthedocs
 
 ## 0.1.3
-
+*(February 13, 2016)*
 * [BaseSniffer][basesniffer] is now finished and capable of parsing a [Pcap][Pcap] file
 * fixed doctests to pass travis ci
 * released the new newversion to pypy
@@ -44,3 +48,4 @@ Project Started.
 [pcap-parser]: https://github.com/caoqianli/pcap-parser
 [URLSniffer]: https://github.com/abzcoding/aptdetector/blob/master/aptdetector/network/sniffer.py
 [Pcap]: https://en.wikipedia.org/wiki/Pcap
+[stages]: https://github.com/abzcoding/aptdetector/blob/master/STAGES.md

README.md

@@ -31,10 +31,7 @@ aptdetector can be added to a project in a few ways. There's the obvious one:
 Then, [thanks to PyPI][aptdetector_pypi], aptdetector is just an import away:
 
 ``` python
-    from aptdetector.network.sniffer import URLSniffer
-    sniffer = URLSniffer
-    sniffer.pcap_file = 'sample.pcap'
-    sniffer.connections(source='10.66.133.90',simplify=True,show_port=True)
+    import aptdetector
 ```
 
 However, due to the nature of utilities, application developers might
@@ -43,6 +40,22 @@ dependencies. See the [Integration][integration] section of the docs
 [aptdetector_pypi]: https://pypi.python.org/pypi/aptdetector
 [integration]: https://aptdetector.readthedocs.org/en/latest/architecture.html#integration
 
+## Status
+
+**Stage Zero** is now completed. you can use [v0.1.4][104] of software to test it:
+``` python
+    from aptdetector.network.sniffer import URLSniffer
+    from aptdetector.network.sniffer import BaseSniffer
+    sni = BaseSniffer()
+    sni.pcap_file='examples/test.pcap'
+    sni.parse()
+    sni.connections(source='173.244.195.17',show_port=True,simplify=True)
+    sni.connections(destination='192.168.204.136',show_port=False,simplify=True)
+```
+
+you can check out the [Stages][stages] if you want to get a sense of project roadmap.
+[104]: https://pypi.python.org/pypi/aptdetector/0.1.4
+[stages]: https://github.com/abzcoding/aptdetector/blob/master/STAGES.md
 ## Disclaimer
 
 Please do not use this program in production!!

STAGES.md

@@ -0,0 +1,109 @@
+# Project Stages
+
+In this project we've tried to identify threats by using network analysis. to overcome the difficulties of the real world we've tried to break the project into multiple stages , so that if we couldn't finish the project , somebody else might get intersted and carry on the job.
+
+
+
+## Stage Zero
+
+In the very fist steps we must be capable of watching the network for any malware moving around, before any system gets infected.
+
+* parse a [Pcap][pcap] file that was sniffed from network to get all passed URLs
+* it must be capable of filtering the result by source or destination
+
+
+
+## Stage One
+
+We want to generalize the malware detection part eventually , but right now i think the [Cuckoo Sandbox][cuckoo] would be sufficient.
+
+* create a workflow to analyse urls and files using [Cuckoo Sandbox][cuckoo]
+* it must be capable of passing the urls that were from [Stage Zero][stagezero] to [Cuckoo Sandbox][cuckoo]
+
+
+
+## Stage Two
+
+Some of the current malwares in the wild are just a mutation of older ones, but due to lack of signature they cannot be detected , but maybe adding blacklisted hosts and community signature would help us to overcome that problem.
+
+* use [Blacklists][blacklist] and [Signatures][signature] to increase the malware detection rate
+
+
+
+## Stage Three
+
+Some malwares would use known ports with their own protocol so they can evade detection , for example if some host is talking on port 443 but not using the https protocol, it is a little bit suspicious! don't you agree with me?
+
+* use protocol analysis to detect unusual activity on known ports
+
+
+
+## Stage Four
+
+Many of the infected hosts will talk to [botnet C&C servers][botcnc] using [API][api] Call
+
+* analyse http headers for any unsual http api call
+
+
+
+## Stage Five
+
+Many of the infected hosts contact their [botnet C&C servers][botcnc] periodically and/or with similar Packets , so in this stage we will introduce ways to detect those patterns and mark them as suspecius traffic.
+
+* ***Time Based*** : infected host asks for specific (non whitelisted) dns name priodically.
+* ***Dns Answer Based*** : in case many Dns name requests ends up with the same IP address (many APTs would try to hide by using different dns names for their C&C servers).
+* ***TTL Value Based*** : packets that are transfered between infected hosts and C&C server have a very low TTL to be effective in running commands.
+* ***Domain Name Based*** : another possible method is to check the percentage of meaningfull name in dns name.
+
+
+
+## Stage Six
+
+Do all the previous steps in *Realtime* (not from a saved [Pcap][pcap] file)
+
+* another plus in this stage would be to check for any [IRC][irc] traffic to mark them as suspicious.
+
+
+
+## Stage Seven
+
+Use [WhiteList][whitelist] and [Machine Learning][maclearn] Algorithms to Lower down the [False Positives][falsepositive] .
+
+## Stage Eight
+
+If we're 100% sure that a network is clean ; for example in an Industrial Network when it's completely off the grid, and we've not connected any device to it; we can Train our program to consider all traffic in that stage clean , and the when we've connected our network to outside world we can use [Anomaly Detection][anomalydet] to increase our [Zero Day][zeroday] detection rate.
+
+
+
+## Stage Nine
+
+use [Traffic Classification][trafclass] to Manually analyse the suspecouis categories.
+
+
+
+## Stage Ten
+
+use [Dynamic Analyses][dynanal] and [Sandboxing][sandbox] to increase malware detection rate.
+
+
+
+## ETC
+
+and many other ideas that will be added gradually...
+
+[cuckoo]: https://downloads.cuckoosandbox.org/docs/
+[pcap]: https://en.wikipedia.org/wiki/Pcap
+[blacklist]: https://zeltser.com/malicious-ip-blocklists/
+[signature]: http://sanesecurity.com/
+[botcnc]: https://en.wikipedia.org/wiki/Botnet#Organization
+[api]: https://en.wikipedia.org/wiki/Application_programming_interface
+[irc]: https://en.wikipedia.org/wiki/Internet_Relay_Chat
+[falsepositive]: https://en.wikipedia.org/wiki/False_positives_and_false_negatives#False_positive_error
+[whitelist]: https://en.wikipedia.org/wiki/Whitelist
+[maclearn]: https://en.wikipedia.org/wiki/Machine_learning
+[anomalydet]: https://en.wikipedia.org/wiki/Anomaly_detection
+[zeroday]: https://en.wikipedia.org/wiki/Zero-day_(computing)
+[trafclass]: https://en.wikipedia.org/wiki/Traffic_classification
+[dynanal]: http://opensecuritytraining.info/MalwareDynamicAnalysis.html
+[sandbox]: https://blog.avast.com/2012/11/16/what-is-the-avast-autosandbox-and-how-does-it-work/
+

TODO.md

@@ -1,15 +1,15 @@
 TODO
 ====
 
-- extracting urls from network traffic
+~~extracting http urls from network traffic~~
 - create a workflow for automated malware detection
 
 network
 ----------
 
 ~~implement network sniffer~~
-- try to extend [BaseSniffer][URLSniffer] so that we can parse every single url
-that was passed around in a [Pcap][Pcap] file
+~~try to extend [BaseSniffer][URLSniffer] so that we can parse every single url
+that was passed around in a [Pcap][Pcap] file~~
 
 malware
 ----------

aptdetector/network/packet.py

@@ -17,15 +17,43 @@ def __init__(self):
         self.__url = None
 
     def __str__(self):
-        """string representation of :class:TcpPacket object"""
+        """string representation of :class:`TcpPacket` object"""
         res = str(self.sourceHost + ":" + str(self.sourcePort) + " ---> " +
                   self.destinationHost + ":" + str(
                       self.destinationPort) + "\n" + self.request)
         return res
 
+    @params(self=object, target_id=int, show_port=bool, reverse=bool)
+    def create_packet(self,
+                      target_id,
+                      show_port={bool: False},
+                      reverse={bool: False}):
+        """create an address based on target_id"""
+        if (target_id == 1 and reverse is False) or (target_id == 2 and
+                                                     reverse is True):
+            if show_port:
+                return self.sourceHost + ":" + str(self.sourcePort)
+            else:
+                return self.sourceHost
+        elif (target_id == 2 and reverse is False) or (target_id == 1 and
+                                                       reverse is True):
+            if show_port:
+                return self.destinationHost + ":" + str(self.destinationPort)
+            else:
+                return self.destinationHost
+        else:
+            return None
+
+    @classmethod
     @returns(bool)
-    def valid_ip(self, addr):
-        """check for valid ip"""
+    def valid_ip(cls, addr):
+        """check for valid ip
+
+        Args:
+            addr    (str):  an string that need to be checked
+        Returns:
+            True if addr is a valid ip address , False otherwise
+        """
         try:
             socket.inet_aton(addr)
             return True
@@ -35,61 +63,61 @@ def valid_ip(self, addr):
     @property
     @returns(str)
     def sourceHost(self):
-        """sample"""
+        """get source host's ip"""
         return self.__sourceHost
 
     @sourceHost.setter
     @params(self=object, value=str)
     def sourceHost(self, value):
-        """sample"""
-        if self.valid_ip(value):
+        """set source host's port"""
+        if TcpPacket.valid_ip(value):
             self.__sourceHost = value
 
     @property
     @returns(int)
     def sourcePort(self):
-        """sample"""
+        """get source host's port"""
         return self.__sourcePort
 
     @sourcePort.setter
     @params(self=object, value=int)
     def sourcePort(self, value):
-        """sample"""
+        """set source host's port"""
         self.__sourcePort = value
 
     @property
     @returns(str)
     def destinationHost(self):
-        """sample"""
+        """get destination host's ip"""
         return self.__destinationHost
 
     @destinationHost.setter
     @params(self=object, value=str)
     def destinationHost(self, value):
-        """sample"""
-        if self.valid_ip(value):
+        """set destination host's ip"""
+        if TcpPacket.valid_ip(value):
             self.__destinationHost = value
 
     @property
     @returns(int)
     def destinationPort(self):
-        """sample"""
+        """get destination host's port"""
         return self.__destinationPort
 
     @destinationPort.setter
     @params(self=object, value=int)
     def destinationPort(self, value):
-        """sample"""
+        """set destination host's port"""
         self.__destinationPort = value
 
     @property
     @returns(str)
     def request(self):
-        """sample"""
+        """get requested url address"""
         return self.__url
 
     @request.setter
     @params(self=object, value=str)
     def request(self, value):
-        """sample"""
+        """set requested url address"""
         self.__url = value