Update MPs/Senators photos

[mackaymackay]

We have had an email from the office of an MP pointing out that many of the photos we are using on They Vote For You are now very old and should be changed to match the new ones available on the official parliamentary website for members and senators.

[jamezpolley]

theyvoteforyou/app/lib/data_loader/people.rb

Lines 12 to 17 in 0272550

	def self.load_missing_images!
	Person.where(small_image_url: nil).find_each do |person|
	Rails.logger.info "Checking small photo for person #{person.id}..."
	url = "https://www.openaustralia.org.au/images/mps/#{person.id}.jpg"
	person.update(small_image_url: url) if CheckResourceExists.call(url)
	end

As expected, the code won't ever look for an updated photo.

A long-term fix for this would be to update the code so that it does something more - perhaps an additional task that looks for refreshed images that is run every few months.

Short term - it's possible that simply clearing the cache might work. The concern I have is that although this should pull in new images, it may be that there are some images no longer available. So I'll do this on staging first, keeping a copy of the existing image cache, and seeing what gets missed.

[jamezpolley]

But of course there's an additional wrinkle. That code isn't scraping the APS site; it's scraping the cache that's been built elsewhere...

https://github.com/openaustralia/openaustralia-parser/blob/adac530d5d1c2aa440d3a6ce8a944776a2ff314f/lib/people_image_downloader.rb#L47-L53

Yes, yes, that is indeed a TODO

[jamezpolley]

Have made a first attempt to run the code. Actually, the first few. It keeps failing the first time it hits an error:

deploy@ip-172-31-43-242:/srv/www/staging/current/openaustralia-parser$  bundle exec member-images.rb
Loading config from: /srv/www/staging/releases/20251113021312/openaustralia-parser/lib/../configuration.yml
Downloading person images...
Found biography page for John Douglas Anthony
WARNING: Skipping photo for The Rt. Hon. John Douglas Ac Ch Anthony because they don't exist in the list of people
Found biography page for John Lindsay Armitage
Found biography page for Marshall Baillieu
Found biography page for Reginald John Birney
Found biography page for Reginald Bishop
Found biography page for Neville Thomas Bonner
Found biography page for John William Bourchier
Found biography page for James Mark Bradfield
Found biography page for Joseph Tilley Brown
Found biography page for Robert James Brown
WARNING: Skipping photo for Robert James Brown, AM because they don't exist in the list of people
Found biography page for Gordon Munro Bryant
Found biography page for Melville Harold Bungey
Found biography page for William George Burns
Found biography page for Stephen Edward Calder
Found biography page for Clyde Robert Cameron
Found biography page for John Leslie Carrick
Found biography page for Moses Henry Cass
Found biography page for James Howard Catts
bundler: failed to load command: member-images.rb (member-images.rb)
Mechanize::ResponseCodeError: 403 => Net::HTTPForbidden for https://parlinfo.aph.gov.au/parlInfo/search/display/display.w3p;query=Dataset:allmps%20James%20Luke%20Cavanagh -- unhandled response
  /srv/www/staging/shared/bundle/ruby/2.7.0/gems/mechanize-2.8.5/lib/mechanize/http/agent.rb:333:in `fetch'
  /srv/www/staging/shared/bundle/ruby/2.7.0/gems/mechanize-2.8.5/lib/mechanize/http/agent.rb:1012:in `response_redirect'
  /srv/www/staging/shared/bundle/ruby/2.7.0/gems/mechanize-2.8.5/lib/mechanize/http/agent.rb:325:in `fetch'
  /srv/www/staging/shared/bundle/ruby/2.7.0/gems/mechanize-2.8.5/lib/mechanize.rb:465:in `get'
  /srv/www/staging/releases/20251113021312/openaustralia-parser/lib/people_image_downloader.rb:98:in `biography_page_for_person_with_name'
  /srv/www/staging/releases/20251113021312/openaustralia-parser/lib/people_image_downloader.rb:121:in `block in person_bio_page'
  /srv/www/staging/releases/20251113021312/openaustralia-parser/lib/people_image_downloader.rb:121:in `each'
  /srv/www/staging/releases/20251113021312/openaustralia-parser/lib/people_image_downloader.rb:121:in `find'
  /srv/www/staging/releases/20251113021312/openaustralia-parser/lib/people_image_downloader.rb:121:in `person_bio_page'
  /srv/www/staging/releases/20251113021312/openaustralia-parser/lib/people_image_downloader.rb:59:in `block in download'
  /srv/www/staging/releases/20251113021312/openaustralia-parser/lib/people_image_downloader.rb:42:in `each'
  /srv/www/staging/releases/20251113021312/openaustralia-parser/lib/people_image_downloader.rb:42:in `download'
  /srv/www/staging/releases/20251113021312/openaustralia-parser/lib/people.rb:180:in `download_images'
  member-images.rb:13:in `<top (required)>'

That particular URL seems particularly problematic, but I've seen it be successful there and go on, only to eventually hit a 403 somewhere else and abort.

So it looks like this might not be as simple as "flush the cache, run the script". I'm thinking we might need to:

• Add some error handling - rather than giving up on the first error
• Maybe rate-limit requests, in case we're looking like we're overloading the server.

[jamezpolley]

I've just remembered openaustralia/openaustralia#648 - there's a newer API that we could use to get information rather than scraping pictures from the website