Unexpected behaviour in ActiveJob callbacks

[shahidkhaliq]

Hello, thanks for all the hard work on this gem! I've been trying to use this gem to prevent ActiveJob jobs from enqueueing within transactions but have run into some unexpected behavior. If my job looks like the following:

class TestJob < ActiveJob::Base
  before_enqueue do |job|
    puts "1"
  end

  around_enqueue do |job, block|
    puts "2"
    block.call
  end

  include AfterCommitEverywhere

  around_enqueue do |job, block|
    puts "3"
    after_commit { block.call }
  end

  around_enqueue do |job, block|
    puts "4"
    block.call
  end

  def perform; end
end

And I try to enqueue it from within a transaction:

ActiveRecord::Base.transaction do
  TestJob.perform_later
end

It outputs 123 1234 instead of 1234 that I'm expecting. It seems like all the callbacks preceding the callback that has after_commit are run twice instead of once.

This is on Rails 6.0.6.1 with after_commit_everywhere (1.1.0).

Am I doing something wrong here or is this a bug in the gem?

[Envek]

To be honest I'm not sure why this code runs like that.

However, it doesn't seem that your intention to postpone enqueue from callbacks would work at all, as ActiveJob callbacks are built with plain ActiveSupport callbacks and you need to halt callback chain by throwing :abort, see: https://api.rubyonrails.org/classes/ActiveSupport/Callbacks.html#method-i-run_callbacks

Anyway, I would recommend implementing this in some other way, e.g. by redefining the whole enqueue method of ActiveJob job.

Also see https://github.com/Envek/after_commit_everywhere/issues/27#issuecomment-1692865436

[AndreiMotinga]

@Envek maybe you could help me. I don't quite understand the problem yet, only guessing that it's related.

I have code that runs in transaction that enques background job to sync some related records to main record.

      MyModel.transaction do
        do_things
        ...
        ...
        MyService.call(record:, input:)
      end

      MyService
        ...
        call
          do_things
          ...
          ...
          AfterCommitEverywhere.after_commit { SyncMyModelsJob.perform_async(record.uuid) }
        end
      end

I expect to

1. record to be updated
2. bg job enqueud
3. by the time bg job runs, it pulls original (now committed and changed) record from db and syncs its related records.

problem is i'm facing some inconsistent behavior. it works most of the time, but every 4th-5th time (if I trigger continuously ) by the time bg job fires original record isn't updated yet.

so lets say record has name: one, I update to name: two, works as expected, three, four.. when I update record to name: five bg fires but original record is still four and so I lose the update on synced records.

[Envek]

but every 4th-5th time (if I trigger continuously ) by the time bg job fires original record isn't updated yet.

Do you have database replication? It is possible if commit was succeeded on master and then background job tried to read from a replica which hasn't replayed these changes yet.

[AndreiMotinga]

Yes, we do have it. Interesting. what to do in that case?

[Envek]

Well, you can enable synchronous transaction commit in your database (when master waits for some of replicas before completing COMMIT command, though that can kill overall performance). You can calculate replication lag and use perform_in. Or you can point workers to the master database.

[AndreiMotinga]

just to document it

@Envek looks like in my case it didnt really have anything to do with after_commit_everywhere. issue was still there when not using the gem.

I could not solve the problem using

ActiveRecord::Base.connected_to(role: :writing) do
  # write operations
end

but using ActiveRecord::Base.connection.stick_to_master! as in

class SyncMyModelsJob
  include Sidekiq::Worker
  
  def perform(uuid)
     ActiveRecord::Base.connection.stick_to_master!
    .....
end

appears to be fixing it.

Thanks for the help.