Download and looking at traffic flow data
- Bigger than a GB ?
- Bigger than a TB ?
- Bigger than the memory on your PC?
- Bigger than the size of your hard drive?
Lets try to get some data that is larger than the size of my PC memory - motorway flow data
http://tris.highwaysengland.co.uk
- Need to download and clean up your data - how to do it when you have lots of data?
- Discover your data ... work out how to get it since it probably wont be straight forward
- Websites are generally made to look nice, rather than easy to download data, so have to do some quick hacking about
- Write some code in Python to do it!
- Then you can load it up in memory etc using Pandas and so how it works
- As you get bigger you will need to look at databases of some sort ..
- requests - http://docs.python-requests.org/en/master/
- pandas - http://pandas.pydata.org/
- beautiful soup - https://www.crummy.com/software/BeautifulSoup/bs4/doc/
- jupyter notebook - http://jupyter.org/