Fails when reading broken index file

[daveharris]

Hi,

We use RefineryCMS-blog which is a simple engine within RefineryCMS. acts_as_indexed is used for indexing posts, comments and categories.

For example, a Post has the declaration (code snippet):

acts_as_indexed :fields => [:title, :body]

We are running acts_as_indexed-0.7.8, refinerycms-2.0.9, refinerycms-blog-2-0-stable. Refinery 2.0.9 locks us these specific version . We have a lot of Refinery customisations we can't easily upgrade RefineryCMS.

This morning we had the below sequence of errors:

1. EOFError: end of file reached

/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/storage.rb:87 in "load"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/storage.rb:87 in "block (3 levels) in operate"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/storage.rb:86 in "open"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/storage.rb:86 in "open"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/storage.rb:86 in "block (2 levels) in operate"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/storage.rb:180 in "block in lock_file"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/storage.rb:177 in "open"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/storage.rb:177 in "open"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/storage.rb:177 in "lock_file"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/storage.rb:83 in "block in operate"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/storage.rb:80 in "each"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/storage.rb:80 in "operate"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/storage.rb:30 in "remove"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/search_index.rb:50 in "remove_record"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/search_index.rb:54 in "update_record"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/class_methods.rb:87 in "index_update"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/instance_methods.rb:28 in "update_index"

2. Errno::ENOENT: No such file or directory - [rails_root]/tmp/index/production/refinery/blog/post/99_111_109.ind.tmp

/usr/local/rbenv/versions/1.9.3-p385/lib/ruby/1.9.1/fileutils.rb:1515 in "stat"
/usr/local/rbenv/versions/1.9.3-p385/lib/ruby/1.9.1/fileutils.rb:1515 in "block in fu_each_src_dest"
/usr/local/rbenv/versions/1.9.3-p385/lib/ruby/1.9.1/fileutils.rb:1531 in "fu_each_src_dest0"
/usr/local/rbenv/versions/1.9.3-p385/lib/ruby/1.9.1/fileutils.rb:1513 in "fu_each_src_dest"
/usr/local/rbenv/versions/1.9.3-p385/lib/ruby/1.9.1/fileutils.rb:508 in "mv"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/storage.rb:169 in "write_file"

3. ArgumentError: dump format error(0x2)

/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/storage.rb:87 in "load"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/storage.rb:87 in "block (3 levels) in operate"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/storage.rb:86 in "open"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/storage.rb:86 in "open"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/storage.rb:86 in "block (2 levels) in operate"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/storage.rb:180 in "block in lock_file"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/storage.rb:177 in "open"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/storage.rb:177 in "open"
/vendor/bundle/ruby/1.9.1/gems/acts_as_indexed-0.7.8/lib/acts_as_indexed/storage.rb:177 in "lock_file"

Given the order of errors, I presume that for some reason the index file couldn't be written correctly, which then caused read and write errors.

I have checked the server and we have plenty of hard drive space, there were no deployments at the time and there were no other outages anywhere. I was able to resolve it by deleting the tmp/index directory, but in the heat of the moment I forgot to keep a backup of the broken file to investigate it further.

We have had the same problem 3 or 4 times in the last 6 months or so it's starting to become a bit of a problem with no known cause or obvious fix.

I realise that you have now moved the codebase onto v0.8.3 but perhaps you could give me some guidance here? Is there anything we can do? It doesn't seem to be a problem with Refinery's integration of acts_as_indexed into their app. Looking at the code there doesn't seem to be any handling of error cases, is it done somewhere else? What should acts_as_indexed do if the index file is somehow corrupted?

Thanks for your help - much appreciated. If you need any other information please let me know.

Dave

[Irostovsky]

Time-to-time I have the same issue: "No such file or directory" and then "dump format error(0x1)".
Version of acts_as_indexed: 0.7.8 with Refinerycms 2.0.5.

[daveharris]

Hi @Irostovsky, it seems that there is no active development on acts_as_indexed anymore :(
I wonder what Refienry will do...

[dougal]

Pull requests are always welcome, folks.

Lots of open-source project maintainers have day jobs too. Recently I have been unable to devote time to AAI.

The issue here is that the locking of the index files is not as safe across multiple processes as it could be. This is handled in storage.rb.

[daveharris]

Hi @dougal,
Well said, good point about day jobs. I didn't mean to insult, I just didn't know what the status of the project was.
Dave

[dougal]

No insult taken!

I do plan to look at this again, the difficulty is in replicating these exceptions as they are race conditions across multiple processes. I wrote an experimental version a few months back using log structured storage, I might push this out as a branch for people to test and report results. Although testing in a production environment is not something I would like to encourage.