Skip to content

Crawler detect bot checker #33

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@
*.gem
Gemfile.lock
/spec/internal/log/*
/spec/internal/db/combustion_test.sqlite
2 changes: 1 addition & 1 deletion .ruby-version
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2.6.5
2.7.6
4 changes: 2 additions & 2 deletions README.markdown
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Features
* Simple trending based on most hits in the last day, week, month, etc.
* Rake task to group old hit records for better performance
* [ActsAsTaggableOn](https://github.com/mbleigh/acts-as-taggable-on) integration for trending tags/topics support
* [Voight-Kampff](https://github.com/biola/Voight-Kampff) integration for bot checking
* [crawler_detect](https://github.com/loadkpi/crawler_detect) integration for bot checking

Requirements
============
Expand Down Expand Up @@ -87,5 +87,5 @@ Notes
* The `punching_bag:combine` rake tasks is not run automatically. You'll have to run it manually or add it as a cron job.
* The `punching_bag:combine` rake task can take a while depending on how many records need to be combined.
* Passing the `request` object to the `punch` method is optional but without it requests from bots, crawlers and spiders will be tracked.
* See the [Voight-Kampff](https://github.com/biola/Voight-Kampff) documentation if you'd like to customize the list of user-agents considered bots.
* See the [crawler_detect](hhttps://github.com/loadkpi/crawler_detect) documentation here.
* The tag related features will only work if you have [ActsAsTaggableOn](https://github.com/mbleigh/acts-as-taggable-on) installed and enabled on the same models as Punching Bag.
4 changes: 2 additions & 2 deletions lib/punching_bag.rb
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
module PunchingBag
require 'punching_bag/engine' if defined?(Rails)
require 'punching_bag/acts_as_punchable'
require 'voight_kampff'
require 'punching_bag/railtie'

def self.punch(punchable, request = nil, count = 1)
if request.try(:bot?)
if request.try(:is_crawler?)
false
else
p = Punch.new
Expand Down
1 change: 1 addition & 0 deletions lib/punching_bag/engine.rb
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
require 'punching_bag'
require 'rails'
require 'active_record'
require 'crawler_detect'

module PunchingBag
class Engine < Rails::Engine
Expand Down
8 changes: 8 additions & 0 deletions lib/punching_bag/railtie.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
require 'rack/crawler_detect'
module PunchingBag
class Railtie < Rails::Railtie
initializer "punching_bag.insert_middleware" do |app|
app.config.middleware.use Rack::CrawlerDetect
end
end
end
2 changes: 1 addition & 1 deletion punching_bag.gemspec
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ spec = Gem::Specification.new do |s|
s.homepage = "https://github.com/biola/punching_bag"
s.license = 'MIT'
s.add_dependency 'railties', '>= 3.2'
s.add_dependency 'voight_kampff', '>= 1.0'
s.add_dependency 'crawler_detect', '>= 1.0'
s.add_development_dependency 'activerecord', '~> 5.2.3'
s.add_development_dependency 'combustion', '~> 1.1'
s.add_development_dependency 'rspec-its', '~> 1.3'
Expand Down
Binary file removed spec/internal/db/combustion_test.sqlite
Binary file not shown.
2 changes: 1 addition & 1 deletion spec/lib/punching_bag/acts_as_punchable_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

describe PunchingBag::ActiveRecord::ClassMethods do

let(:request) { instance_double(ActionDispatch::Request, bot?: false) }
let(:request) { instance_double(ActionDispatch::Request, is_crawler?: false) }

let(:article1) { Article.create title: 'Article 1', content: 'Ding, ding ding... ding. Ding. DING. DING! ' }
let(:article2) { Article.create title: 'Article 2', content: 'Ding, ding ding... ding. Ding. DING. DING! ' }
Expand Down
4 changes: 2 additions & 2 deletions spec/lib/punching_bag_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@
let(:request) { nil }

context 'when request is from a bot' do
let(:request) { instance_double(ActionDispatch::Request, bot?: true) }
let(:request) { instance_double(ActionDispatch::Request, is_crawler?: true) }

it 'does nothing' do
expect(PunchingBag.punch(article, request)).to be false
end
end

context 'when the request is valid' do
let(:request) { instance_double(ActionDispatch::Request, bot?: false) }
let(:request) { instance_double(ActionDispatch::Request, is_crawler?: false) }

it 'creates a new punch' do
expect { PunchingBag.punch(article, request) }.to change { Punch.count }.by 1
Expand Down