Fetch Periscope data

[carlospreising]

Funny enough, I had already been working on a selenium script to grab periscope data, started like two weeks ago.

Here I also change the way lists are grabbed, so that we support unzipped CSV's in the filenames that periscope outputs. I also added (a bit clumsily) a button to fetch new periscope data.

New .env requirements for periscope data: PERISCOPE_URL and PERISCOPE_PASS

You can test the selenium script only by running uv run src/fetch_list.py

[github-advanced-security]

ruff found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

[carlospreising]

going to make sure these pass tests before undrafting!

[bmos]

Check out the contributing guide, ruff can autofix a lot

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
src
app.py					1-16, 51-91
src/components
sidebar.py					1-17
src/utils
fetch_list.py					1-36, 38-63, 67-75, 78-79
scan_lists.py					156, 161-170, 179-183, 195-199
Project Total

This report was generated by python-coverage-comment-action

[bmos]

Can you look at doing this with requests instead?
I don't love having to use a webdriver to accomplish this, although it might be necessary.
Just open your browser inspector's network tab before logging in to periscope and you'll see what the traffic looks like during login/data load.

[carlospreising]

@bmos I started out trying to do it via request/bs4, but I couldn't figure out how to work out how to send the password, since it looks like they hash it in a specific way (md5 + doing other stuff to it, I think?). I'm not very well versed in web dev/auth so I could be wrong here, or looking at the wrong thing.

Once authenticated, the hover and other dynamic elements made me turn to selenium, since there isn't just like a, "here's a csv link!". I can look into this more though, I think I have a few people I can ask about it.

[carlospreising]

oh wait we might be able to just POST it, I think this is possible. Will update.

[bmos]

This is looking so great! I will probably refactor it to move things around a bit and maybe some UI polish, but expect this to be merged sometime soon 🥰

[carlospreising]

I am definitely not a UI person, so I'm glad to hear that!