ActiveJob::SerializationError: Unsupported argument type: ActiveRecord::Relation

[sentry-io]

Sentry Issue: PLOTS2-1AN

ActiveJob::SerializationError: Unsupported argument type: ActiveRecord::Relation
  app/models/user.rb:368:in `send_digest_email'
    SubscriptionMailer.send_digest(id, @nodes, @frequency).deliver_later
  app/jobs/digest_mail_job.rb:14:in `perform'
    users.each(&:send_digest_email)
...
(59 additional frame(s) were not displayed)

Unsupported argument type: ActiveRecord::Relation

[jywarren]

smartinez87/exception_notification#319 (comment)

The real problem is the way the email is created and how ActiveJob serializes the data to pass it to sidekiq.

This is odd. I guess it could happen with the new v6 sidekiq as of #8385 maybe???

[jywarren]

Also rails/rails#25993. Not sure why this started happening though.

OK, so the deliver_later was added recently. Maybe it was just deployed first time, although that seems unlikely.

Raised when an unsupported argument type is set as a job argument. We currently support NilClass, Fixnum, Float, String, TrueClass, FalseClass, Bignum, BigDecimal, and objects that can be represented as GlobalIDs (ex: Active Record). Raised if you set the key for a Hash something else than a string or a symbol. Also raised when trying to serialize an object which can't be identified with a Global ID - such as an unpersisted Active Record model.

(from http://edgeapi.rubyonrails.org/classes/ActiveJob/SerializationError.html)

[jywarren]

Meaning we can pass an ActiveRecord object but not an ActiveRecord relation, i think.

plots2/app/models/user.rb

Lines 255 to 260 in 87b776f

	def content_followed_in_period(start_time, end_time, order_by = 'node_revisions.timestamp DESC', node_type = 'note', include_revisions = false)
	tagnames = TagSelection.where(following: true, user_id: uid)
	node_ids = []
	tagnames.each do |tagname|
	node_ids += NodeTag.where(tid: tagname.tid).collect(&:nid)
	end

[jywarren]

We can try fixing this but #9575 is the fallback.

[jywarren]

Aha, so we should be able to send full ActiveRecord objects... why isn't this working?

https://www.xspdf.com/help/51642031.html

[jywarren]

OK, my best guess is that Sidekiq's new version isn't getting configured correctly at all, and so even though we should be able to send it ActiveRecord objects, it's not booting properly and we need to revert to sidekiq v5 in #9575.

See this breaking change for notes on changes to REDIS_PROVIDER and REDIS_URL variables:

BREAKING CHANGE Validate proper usage of the REDIS_PROVIDER variable. This variable is meant to hold the name of the environment variable which contains your Redis URL, so that you can switch Redis providers quickly and easily with a single variable change. It is not meant to hold the actual Redis URL itself. If you want to manually set the Redis URL (not recommended as it implies you have no failover), then you may set REDIS_URL directly. [#3969]

or maybe:

BREAKING CHANGE Remove the daemonization, logfile and pidfile arguments to Sidekiq. Use a proper process supervisor (e.g. systemd or foreman) to manage Sidekiq. See the Deployment wiki page for links to more resources.

https://github.com/mperham/sidekiq/blob/master/Changes.md#60

I reverted the v6 PR, but leaving this open to track what happens and possibly develop a better solution.

[jywarren]

I'm wondering if Redis failed to start properly for related reasons? (see https://sentry.io/share/issue/4cefe4966e0b4d1395761ec1bf97a1b3/) and this might also be related to the Sidekiq changes? Not sure but this error appeared around when we deployed the Sidekiq v6 upgrade and then we stopped seeing mail/SMTP errors which had been around for a few days, which is worrisome.

MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. Commands that may modify the data set are disabled, because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option). Please check the Redis logs for details about the RDB error.

[jywarren]

Noting logs also showed this, which may mean we need to flush old jobs with too many email recipients in BCC:

2021-04-27T22:46:47.144Z 1 TID-owyhhnyj1 WARN: Net::SMTPServerBusy: 452-4.5.3 Your message has too many recipients. For more information regarding

[jywarren]

Noting also this Serialization error seems not new. Hundreds previously at #9420

So, the revert here may be more for the MISCONF Redis error above! Let's investigate more!