NAME: Guttapati Jayasurya Reddy
This project is about taking important information from police reports in Norman, Oklahoma, which are stored in PDF files. These reports tell us things like the time something happened, the incident number, where it happened, what type of incident it was (like a traffic stop or a welfare check), and a special code called ORI that tells us which police department handled it.
First, we download the PDF from the internet using a link (URL). Then, we read the PDF and find the important parts inside it. We need to pick out the date, the number, the location, and the type of incident. Sometimes, the location takes more than one line, so we have to put those lines together.
After we get all the information, we save it in a database called SQLite. This helps us organize the data better so we can search for things and see the details later. The database has columns for the date and time, incident number, location, type of incident, and ORI.
At the end, we count how many times each type of incident happened, like how many traffic stops or suspicious activities were in the report.
pipenv install
pipenv shell
YOU CAN MAKE THE PROJECT WORK BY RUNNING THIS COMMAND:
pipenv run python project0/main.py --incidents "url"
https://drive.google.com/file/d/1tIbaOTNW8QV92uS91_DXpdqbZXb3nEir/view?usp=sharing
download_pdf(url) The download_pdf(url) function takes a link (URL) and gets the PDF from that link. First, it uses the requests.get(url) line to ask the website for the PDF file. Then, it checks if the download was successful using response.raise_for_status(). If there’s no problem, the function saves the PDF to a file on your computer. It uses the folder "resources" and saves the file with the name "daily_incident_summary.pdf". The file is saved using this line: with open(pdf_file, "wb") as f: f.write(response.content). If something goes wrong, it prints an error message and stops.
extract_incident_data(data) The extract_incident_data(data) function reads the PDF file and finds the important parts of each incident. First, it opens the PDF using PdfReader(data) and gets all the pages. For each page, it reads the text using page.extract_text(). The text is stored in a long string variable called text. Then, the function splits this text into lines using the line for each in text.split('\n'). Each line has different parts like the date, incident number, location, and type of incident. The function looks at certain parts of each line. For example, date_time = each[:16].strip() grabs the date and time from the first part of the line, and incident_number = each[28:48].strip() grabs the incident number from a later part of the line. It keeps checking for each part of the incident (like location and nature). If the line doesn’t have a date or incident number, the function adds more text to the last incident (for locations that take up more than one line). Finally, it makes a list of all the incidents and returns that list.
add_incidents_to_db(incidents) The add_incidents_to_db(incidents) function puts the list of incidents into a database. First, it connects to the database using sqlite3.connect(db_file) and makes sure the database is in the "resources" folder. It then drops (removes) the old "incidents" table, if it exists, using the line cursor.execute("DROP TABLE IF EXISTS incidents"). After that, it creates a new table using cursor.execute(""" CREATE TABLE IF NOT EXISTS incidents..."""). This table has five columns: date_time, incident_number, location, nature, and incident_ori. Once the table is ready, the function puts each incident into the table using cursor.executemany(...), where it adds the incidents one by one. The function then saves the changes to the database and returns the connection.
print_incident_counts(conn) The print_incident_counts(conn) function checks how many times each incident type (nature) happened. First, it runs a query using cursor.execute(...) to group the incidents by their type (nature). It counts how many incidents are in each group using COUNT(*) as count in the query. Then, it fetches all the results with cursor.fetchall() and prints each result. For example, it prints how many traffic stops or welfare checks happened, and shows the number for each type of incident.
To run the tests, use this command:
pipenv run python -m pytest
test_download_pdf_valid_url: This test checks if the PDF is downloaded correctly when we use a good (valid) link. It makes sure the PDF file is saved in the right place.
test_download_pdf_invalid_url: This test checks what happens when a bad (invalid) link is used. It makes sure no file is downloaded and the function returns nothing (None).
test_extract_incident_data: This test looks at how the program takes information from the PDF. It checks if the function can pull out important details about the incidents from the PDF file.
test_add_incidents_to_db: This test checks wheather the incidents extracted from the PDF are saved correctly in the database or not. It checks if the number of records saved matches the number of incidents we found.
test_print_incident_counts: This test checks if the program counts how many times each type of incident happens and prints it. It makes sure the output shows the right types of incidents and their counts.
I thought the PDF format will stay the same. If the PDF changes, the program might not work right.
If any information is missing, it can cause wrong entries in the database.
If the location has more than one line, the code will add it to the last entry when the date and incident number are empty.