Skip to content

Reindexing queueing for touched DOIs #82

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
codycooperross merged 4 commits into from
Mar 10, 2026
Merged

Reindexing queueing for touched DOIs #82

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b9259d3
Test implementation of reindexing queueing for touched DOIs
codycooperross Feb 27, 2026
0c8913a
Linting
codycooperross Feb 27, 2026
8097820
Change threads to 10
codycooperross Feb 27, 2026
94a5543
Compatibility with lupo job
codycooperross Mar 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions app/constants/relation_types.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,4 +68,8 @@ module RelationTypes
"datacite-crossref",
"crossref",
].freeze

# relation_type_ids that affect counts in lupo.
SOURCE_RELATION_TYPES = ["references", "parts", "versions"].freeze
TARGET_RELATION_TYPES = ["views", "downloads", "citations", "part_of", "version_of"].freeze
end
28 changes: 28 additions & 0 deletions app/models/event.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,4 +139,32 @@ def obj_hash
indexes :citation_year, type: :integer
indexes :cache_key, type: :keyword
end

class << self
def reindex_touched_dois(start_date:, end_date:, threads: 10)
total = 0

start_date.to_date.upto(end_date.to_date) do |date|
dois = Set.new

where(updated_at: date.all_day)
.where(source_relation_type_id: RelationTypes::SOURCE_RELATION_TYPES)
.distinct.pluck(:source_doi)
.each { |doi| dois << doi }
where(updated_at: date.all_day)
.where(target_relation_type_id: RelationTypes::TARGET_RELATION_TYPES)
.distinct.pluck(:target_doi)
.each { |doi| dois << doi }

# Test performance before enabling SQS queues.
# Parallel.each(dois.to_a, in_threads: threads) do |doi|
# SqsUtilities.send_events_doi_index_message(doi)
# end

total += dois.size
end

total
end
end
end
4 changes: 4 additions & 0 deletions app/utilities/sqs_utilities.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ def send_events_other_doi_job_message(data)
send_message(data, shoryuken_class: "OtherDoiJob", queue_name: "events_other_doi_job")
end

def send_events_doi_index_message(data)
send_message(data, shoryuken_class: "ReindexByDoiJob", queue_name: "lupo_background")
end
Comment on lines +9 to +11
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== Shoryuken queue groups =="
rg -n '^\s*groups:|^\s*[a-zA-Z_]+:|^\s*queues:|^\s*-\s*[a-zA-Z0-9_]+$' config/shoryuken.yml

echo
echo "== References to lupo_background =="
rg -nP '\blupo_background\b|_lupo_background\b' app config lib spec || true

echo
echo "== ReindexByDoiJob definitions/usages =="
rg -nP '\bclass\s+ReindexByDoiJob\b|\bReindexByDoiJob\b' app lib spec || true

Repository: datacite/events

Length of output: 638

Define lupo_background queue and ReindexByDoiJob worker, or document external handling.

Line 10 enqueues to lupo_background with ReindexByDoiJob, but neither the queue nor the worker class exist in the repository. Messages will fail silently unless this is explicitly handled by an external system. Either add the queue to config/shoryuken.yml, implement the ReindexByDoiJob class, or document that this route is managed externally.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@app/utilities/sqs_utilities.rb` around lines 9 - 11,
send_events_doi_index_message enqueues to a non-existent queue/class which will
cause silent failures; either add the "lupo_background" queue to your Shoryuken
config (config/shoryuken.yml) and register it, or implement the worker class
ReindexByDoiJob (with a perform method) to process messages, or add clear
documentation in the repo stating this queue/job are handled externally; update
send_events_doi_index_message tests if needed to reference the implemented
ReindexByDoiJob and ensure the queue name "lupo_background" is present in
shoryuken config.


private

def send_message(body, options = {})
Expand Down
18 changes: 18 additions & 0 deletions lib/tasks/event.rake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,4 +34,22 @@ namespace :event do

puts("Rake task has completed!")
end

desc "Re-queue SQS re-index messages for unique DOIs in events updated within a date range"
# Dates are inclusive. END_DATE defaults to START_DATE for single-day runs.
task reindex_touched_dois: :environment do
raise "START_DATE is required" if ENV["START_DATE"].blank?

start_date = Date.parse(ENV["START_DATE"])
end_date = Date.parse(ENV["END_DATE"].presence || ENV["START_DATE"])

raise "END_DATE must be on or after START_DATE" if end_date < start_date

count = Event.reindex_touched_dois(
start_date: start_date,
end_date: end_date,
)

puts "Sent #{count} unique DOIs for re-indexing."
end
end