Add Github workflows to autogenerate main readmes

[Majestic9169]

Should be simple python (i hope), as long as the writeup templates are well defined

[Majestic9169]

okay so CTFd provides an api for all the challenges in a ctf, this is useful for the ctf specific readmes. however

• it requires auth of some kind. I tried using both an access and the session cookie and failed, so for now just download challenges.json and run the ctf_readme.py script

[Majestic9169]

the main readme is more challenging though. CTF time does not provide an api for the events participated in by a team, so those will have to come manually from the repo itself. can add a simple file to keep track of the event ids of these ctfs, however next challenge is that ctftime does not provide an api for the rank of a team in a ctf as far I saw. This is another challenge. how tf are the maintaining these in the team profiles.

we can still work around these by creating a dictionary type file as mentioned above. now the qn what format should that file be. json does not seem human friendly to write, so i'm considering yml unless others have suggestions?

once we have that file we can create github workflows to automate this generation of both readmes

thoughts @harshkhandeparkar @chirag-ghosh @shikharish ?

[harshkhandeparkar]

we can still work around these by creating a dictionary type file as mentioned above. now the qn what format should that file be. json does not seem human friendly to write, so i'm considering yml unless others have suggestions?

Where does the data come from?

[Majestic9169]

I was going to add it manually, but then @shikharish introduced me to web scraping

[harshkhandeparkar]

Scrape CTFtime?

[chirag-ghosh]

I see 2 options

• Scrape the profile page
• Use the following APIs to get data (as stated above we will need to store the event ids in the repo)
  • https://ctftime.org/api/v1/teams/304605/ to get our team information
  • https://ctftime.org/api/v1/results/ to get the results of events (seems like only recent data is available through this API)
  • https://ctftime.org/api/v1/events/2607/ to get specific event information

There is no api for writeups. Ideally we would want writeups to be displayed in ctftime as well as this repo. If we are to automate this process, we have to do web automation. Can be a good exercise if anyone is up for it

[Majestic9169]

the things needed in the main readme are mainly

• list of ctfs
• their date
• our position out of the total teams
• link to official writeups (if available)
• an internal link to the ctf dir in the repo

list of ctfs

This is coming by scraping our ctftime profile page. The main_readme.py script in the ./scripts dir is achieving this currently. There is one issue though that this lists ctfs which we may not have seriously participated in, or only registered for as well. Not really an issue, just different from what the current README.md looks like.

• We are also getting our position in each CTF from this page, but we don't get the total teams here.

Their date

This is coming from the events api currently, also implemented in the script. This gives us more control over the date, however to get the total teams that participated in the ctf we would either need to scrape the event page anyway, or use the results api (which i would prefer not to use). And if we have to scrape the event page anyway, we can get the date from their too. Plus scraping works better than api calls on the campus net for some reason.

The official writeups would probably have to be added manually, I don't see a way to get those in an automatic fashion. Of course we could exclude those too. Implementing a standard naming convention for the CTF dirs would making added the internal link easy too.

❯ python main_readme.py 
{'team': 304605}
304605
[[], ['2607', '26', 'BITSCTF 2025'], ['2584', '333', 'TUCTF 2024'], ['2467', '361', 'x3CTF 2025 (feat. mvm)'], ['2503', '203', 'IrisCTF 2025'], [], ['2540', '96', 'BackdoorCTF 2024'], ['2461', '132', 'niteCTF 2024'], ['2572', '77', 'World Wide CTF 2024'], ['2353', '307', 'DeadSec CTF 2024'], ['2377', '173', 'Wani CTF 2024'], ['2257', '161', ' GPN CTF 2024'], ['2364', '1244', 'NahamCon CTF 2024']]
event id = 2607
07/02/2025
event id = 2584
24/01/2025
event id = 2467
24/01/2025
event id = 2503
04/01/2025
event id = 2540
22/12/2024
event id = 2461
13/12/2024
event id = 2572
30/11/2024
event id = 2353
26/07/2024
event id = 2377
21/06/2024
event id = 2257
31/05/2024
event id = 2364
23/05/2024

Issues

• I didn't think about automating writeup publishing at all, that's something to think about. I was thinking a writeups section could be added to the koss website. I have seen some really cool writeup/blog websites
• What I want to know is

1. Do we want to show our position out of the total number of teams? I don't see a way to get the total number of teams without fetching large quantities of unnecessary data
2. Are we fine with showing the CTFs we haven't participated in seriously or don't have any writeups for?

sorry about the block of text, think of this as a progress report asking for design choices.

[Majestic9169]

I haven't pushed main_readme.py myb