Ingesting Executive orders

[saihaj]

We start use web scraper to ingest

https://www.federalregister.gov/presidential-documents/executive-orders

[kishorliv]

Hi,
I stumbled upon this project and found it interesting. Upon investigating a bit, i found that its possible to scrape executive orders without scraping the web page. If you look at 'All Executive Orders Since 1937' and click the 'JSON' button, it actually responds with a json with via a rest endpoint:

https://www.federalregister.gov/api/v1/documents.json?conditions%5Bcorrection%5D=0&conditions%5Bpresidential_document_type%5D=executive_order&conditions%5Btype%5D%5B%5D=PRESDOCU&fields%5B%5D=citation&fields%5B%5D=document_number&fields%5B%5D=end_page&fields%5B%5D=html_url&fields%5B%5D=pdf_url&fields%5B%5D=type&fields%5B%5D=subtype&fields%5B%5D=publication_date&fields%5B%5D=signing_date&fields%5B%5D=start_page&fields%5B%5D=title&fields%5B%5D=disposition_notes&fields%5B%5D=executive_order_number&fields%5B%5D=not_received_for_publication&fields%5B%5D=full_text_xml_url&fields%5B%5D=body_html_url&fields%5B%5D=json_url&include_pre_1994_docs=true&maximum_per_page=10000&order=executive_order&per_page=10000

and the nice part is this response contains the next_page_url too so very easy to scrape the rest of the executive orders... i am not sure why per_page=10000 is not returning all the executive orders, it just seems to return the default number of documents(20) which is specified in their docs here: https://www.federalregister.gov/api/v1/documentation

Edit: found the issue with per_page=10000, the max is 1000 so probably a typo from their end.. per_page=1000 returns 1000 documents at once...