Skip to content

Fixes #38275 - Capsules index page may takes long time to load #10473

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from

Conversation

hao-yu
Copy link
Member

@hao-yu hao-yu commented Mar 7, 2025

Capsules index page may takes up to 1 minute to load when there are many organizations and thousands of locations assigned to every capsules.

@hao-yu
Copy link
Member Author

hao-yu commented Mar 7, 2025

Based on my local test,, "SmartProxy.preload" appears to be the better choice.

SmartProxy.where.not(id: 1).each {|capsule| total=0; capsule.content_counts["content_view_versions"].each{|id,data| tot
al += data["repositories"].keys.size}; p "#{capsule.name}: #{total} repositories, #{capsule.organizations.size} orgs, #{capsule.location
s.size} locations"}
"haocap611-rhel8.usersys.redhat.com: 3029 repositories, 2 orgs, 23 locations"
"hao-capsule-1.apac.sysmgmt.redhat.com: 3029 repositories, 2 orgs, 23 locations"

Eager load performance:

irb(main):052:0> p Benchmark.measure{ capsules = SmartProxy.eager_load(:organizations, :locations).search_for("").includes(:features).al
l.to_a; p "#{capsules.size} records" }
"3 records"
#<Benchmark::Tms:0x0000557aac251878 @label="", @real=4.398844921961427, @cstime=0.0, @cutime=0.0, @stime=0.0998739999999998, @utime=1.378627999999992, @total=1.4785019999999918>
irb(main):053:0> p Benchmark.measure{ capsules = SmartProxy.eager_load(:organizations, :locations).search_for("", order: "name desc").in
cludes(:features).all.to_a; p "#{capsules.size} records" }
"3 records"
#<Benchmark::Tms:0x0000557aae44d5a8 @label="", @real=5.026257554069161, @cstime=0.0, @cutime=0.0, @stime=0.4797729999999998, @utime=1.5370279999999923, @total=2.016800999999992>
irb(main):054:0> p Benchmark.measure{ capsules = SmartProxy.eager_load(:organizations, :locations).search_for("name =  hao-capsule-1.apa
c.sysmgmt.redhat.com").includes(:features).all.to_a; p "#{capsules.size} records" }
"1 records"
#<Benchmark::Tms:0x0000557abbe00cc8 @label="", @real=2.174584382912144, @cstime=0.0, @cutime=0.0, @stime=0.04729799999999962, @utime=0.6104930000000053, @total=0.6577910000000049>
irb(main):055:0> p Benchmark.measure{ capsules = SmartProxy.eager_load(:organizations, :locations).search_for("organization = ACME").inc
ludes(:features).all.to_a; p "#{capsules.size} records" }
"3 records"
#<Benchmark::Tms:0x0000557abbd7dff8 @label="", @real=4.617417722009122, @cstime=0.0, @cutime=0.0, @stime=0.0861370000000008, @utime=1.8405069999999881, @total=1.926643999999989>
irb(main):056:0> p Benchmark.measure{ capsules = SmartProxy.eager_load(:organizations, :locations).search_for('organization = "Test Org"
', order: "name desc").includes(:features).all.to_a; p "#{capsules.size} records" }
"3 records"
#<Benchmark::Tms:0x0000557abbd54400 @label="", @real=4.469098828965798, @cstime=0.0, @cutime=0.0, @stime=0.09903500000000065, @utime=1.4342220000000054, @total=1.533257000000006>
irb(main):057:0> p Benchmark.measure{ capsules = SmartProxy.eager_load(:organizations, :locations).search_for("location = Brisbane").inc
ludes(:features).all.to_a; p "#{capsules.size} records" }
"3 records"
#<Benchmark::Tms:0x0000557abbda55a8 @label="", @real=5.1073503559455276, @cstime=0.0, @cutime=0.0, @stime=0.11913799999999952, @utime=1.4598600000000204, @total=1.5789980000000199>

VS preload performance

irb(main):064:0> p Benchmark.measure{ capsules = SmartProxy.preload(:organizations, :locations, :features).search_for("").all.to_a; p "#
{capsules.size} records" }
"3 records"
#<Benchmark::Tms:0x0000557aba251518 @label="", @real=0.048533953027799726, @cstime=0.0, @cutime=0.0, @stime=0.0019260000000009825, @utime=0.026086999999989757, @total=0.02801299999999074>
irb(main):065:0> p Benchmark.measure{ capsules = SmartProxy.preload(:organizations, :locations, :features).search_for("", order: "name d
esc").all.to_a; p "#{capsules.size} records" }
"3 records"
#<Benchmark::Tms:0x0000557abb4a6d58 @label="", @real=0.04920649412088096, @cstime=0.0, @cutime=0.0, @stime=0.0008879999999997779, @utime=0.02768300000002455, @total=0.028571000000024327>
irb(main):066:0> p Benchmark.measure{ capsules = SmartProxy.preload(:organizations, :locations, :features).search_for("name =  hao-capsu
le-1.apac.sysmgmt.redhat.com").all.to_a; p "#{capsules.size} records" }
"1 records"
#<Benchmark::Tms:0x0000557aae2491f8 @label="", @real=0.03261191095225513, @cstime=0.0, @cutime=0.0, @stime=0.0018139999999995382, @utime=0.01732699999999454, @total=0.01914099999999408>
irb(main):067:0> p Benchmark.measure{ capsules = SmartProxy.preload(:organizations, :locations, :features).search_for("organization = AC
ME").all.to_a; p "#{capsules.size} records" }
"3 records"
#<Benchmark::Tms:0x0000557abbdcc180 @label="", @real=0.4945501140318811, @cstime=0.0, @cutime=0.0, @stime=0.008556000000000452, @utime=0.10416200000000231, @total=0.11271800000000276>
irb(main):068:0> p Benchmark.measure{ capsules = SmartProxy.preload(:organizations, :locations, :features).search_for('organization = "T
est Org"', order: "name desc").all.to_a; p "#{capsules.size} records" }
"3 records"
#<Benchmark::Tms:0x0000557abbdb5598 @label="", @real=0.6735348519869149, @cstime=0.0, @cutime=0.0, @stime=0.02122000000000135, @utime=0.12184700000000248, @total=0.14306700000000383>
irb(main):069:0> p Benchmark.measure{ capsules = SmartProxy.preload(:organizations, :locations, :features).search_for("location = Brisba
ne").all.to_a; p "#{capsules.size} records" }
"3 records"
#<Benchmark::Tms:0x0000557abbde0658 @label="", @real=0.8661013420205563, @cstime=0.0, @cutime=0.0, @stime=0.029835000000000278, @utime=0.10537800000000175, @total=0.13521300000000203 

@hao-yu
Copy link
Member Author

hao-yu commented Mar 7, 2025

Test results of preload + select

# First run:
2025-03-06T01:07:47 [I|app|9bccc458] Processing by SmartProxiesController#index as HTML
2025-03-06T01:07:50 [I|app|9bccc458]   Rendered smart_proxies/index.html.erb within layouts/application (Duration: 3567.6ms | Allocations: 50861)
2025-03-06T01:07:51 [I|app|9bccc458]   Rendered layouts/base.html.erb (Duration: 295.2ms | Allocations: 42066)
2025-03-06T01:07:51 [I|app|9bccc458]   Rendered layout layouts/application.html.erb (Duration: 3940.3ms | Allocations: 104893)
2025-03-06T01:07:51 [I|app|9bccc458] Completed 200 OK in 4075ms (Views: 3852.0ms | ActiveRecord: 121.0ms | Allocations: 116039)

# Second run:
2025-03-06T01:08:26 [I|app|0d31f34a] Processing by SmartProxiesController#index as HTML
2025-03-06T01:08:29 [I|app|0d31f34a]   Rendered smart_proxies/index.html.erb within layouts/application (Duration: 3409.7ms | Allocations: 42126)
2025-03-06T01:08:29 [I|app|0d31f34a]   Rendered layouts/base.html.erb (Duration: 249.3ms | Allocations: 43976)
2025-03-06T01:08:29 [I|app|0d31f34a]   Rendered layout layouts/application.html.erb (Duration: 3689.3ms | Allocations: 99452)
2025-03-06T01:08:29 [I|app|0d31f34a] Completed 200 OK in 3782ms (Views: 3617.9ms | ActiveRecord: 83.2ms | Allocations: 107398)

Test results of eager_load + select

# First run:
2025-03-06T01:18:56 [I|app|84b64b5a] Processing by SmartProxiesController#index as HTML
2025-03-06T01:19:01 [I|app|84b64b5a]   Rendered smart_proxies/index.html.erb within layouts/application (Duration: 5272.1ms | Allocations: 69006)
2025-03-06T01:19:02 [I|app|84b64b5a]   Rendered layouts/base.html.erb (Duration: 653.5ms | Allocations: 41958)
2025-03-06T01:19:02 [I|app|84b64b5a]   Rendered layout layouts/application.html.erb (Duration: 6095.9ms | Allocations: 122931)
2025-03-06T01:19:02 [I|app|84b64b5a] Completed 200 OK in 6278ms (Views: 6005.6ms | ActiveRecord: 130.5ms | Allocations: 134625)

# Second run:
2025-03-06T01:20:43 [I|app|4bcc56f4] Processing by SmartProxiesController#index as HTML
2025-03-06T01:20:47 [I|app|4bcc56f4]   Rendered smart_proxies/index.html.erb within layouts/application (Duration: 3641.8ms | Allocations: 65847)
2025-03-06T01:20:48 [I|app|4bcc56f4]   Rendered layouts/base.html.erb (Duration: 508.8ms | Allocations: 41937)
2025-03-06T01:20:48 [I|app|4bcc56f4]   Rendered layout layouts/application.html.erb (Duration: 4221.2ms | Allocations: 119750)
2025-03-06T01:20:48 [I|app|4bcc56f4] Completed 200 OK in 4378ms (Views: 4148.6ms | ActiveRecord: 118.3ms | Allocations: 130896)

@hao-yu hao-yu force-pushed the fixes_38275_slow_capsules_page branch from 2bdff40 to 04b834d Compare March 7, 2025 07:49
Capsules index page may takes up to 1 minute to load when there
are many organizations and thousands of locations assigned to
every capsules.
@hao-yu hao-yu force-pushed the fixes_38275_slow_capsules_page branch from 04b834d to 17788d9 Compare March 7, 2025 08:08
@@ -6,7 +6,10 @@ class SmartProxiesController < ApplicationController
before_action :find_status, :only => [:ping, :tftp_server]

def index
@smart_proxies = resource_base_search_and_page.includes(:features)
# Large content_counts data may cause slow page load, especially with eager load.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the api controller suffer from the same slowness?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no. I think api controller doesn't do eager load.

@smart_proxies = resource_base_search_and_page.includes(:features)
# Large content_counts data may cause slow page load, especially with eager load.
# So exclude this field since it is not used in index page.
needed_columns = SmartProxy.attribute_names.reject { |name| name == "content_counts" }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Iirc content_counts column on the smart_proxies table is something that comes from katello. It feels weird to accomodate for it from here. Could the exclusion be handled in katello instead?

Copy link
Member Author

@hao-yu hao-yu Apr 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think just make the change from "eager_load" to "preload" was enough to improve the performance significantly. Excluding the "content_count" is just to make it much better.

I can remove the "content_count" part in this patch. What do you think?

Copy link
Member Author

@hao-yu hao-yu Apr 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is the test data

Loading production environment (Rails [6.1.7.8](http://6.1.7.8/))
Switch to inspect mode.
Benchmark.measure{ capsules = SmartProxy.preload(:organizations, :locations, :features).search_for("").all.to_a; p capsules.size }
8
#<Benchmark::Tms:0x000055d0f35e8868 @label="", @real=0.21791561204008758, @cstime=0.0, @cutime=0.0, @stime=0.008923000000000236, @utime=0.14369000000000298, @total=0.15261300000000322>

Benchmark.measure{ capsules = SmartProxy.preload(:organizations, :locations, :features).search_for("", order: "name desc").all.to_a; p capsules.size }
8
#<Benchmark::Tms:0x000055d0f45e6968 @label="", @real=0.10522056301124394, @cstime=0.0, @cutime=0.0, @stime=0.0041479999999998185, @utime=0.05878600000000134, @total=0.06293400000000116>

Benchmark.measure{ capsules = SmartProxy.preload(:organizations, :locations, :features).search_for("name = capsule01.example.com").all.to_a; p capsules.size }
1
#<Benchmark::Tms:0x000055d0f48948b8 @label="", @real=0.0292418641038239, @cstime=0.0, @cutime=0.0, @stime=0.0008479999999995158, @utime=0.019142000000002213, @total=0.01999000000000173>

Benchmark.measure{ capsules = SmartProxy.preload(:organizations, :locations, :features).search_for("organization = TestOrg1").all.to_a; p capsules.size }
8
#<Benchmark::Tms:0x000055d0f6a888c0 @label="", @real=1.073761180974543, @cstime=0.0, @cutime=0.0, @stime=0.10191399999999984, @utime=0.30263300000000015, @total=0.404547>

Benchmark.measure{ capsules = SmartProxy.preload(:organizations, :locations, :features).search_for("organization = TestOrg2", order: "name desc").all.to_a; p capsules.size }
8
#<Benchmark::Tms:0x000055d0f6d04298 @label="", @real=1.4741708659566939, @cstime=0.0, @cutime=0.0, @stime=0.033474000000000004, @utime=0.7390229999999995, @total=0.7724969999999995>

Benchmark.measure{ capsules = SmartProxy.preload(:organizations, :locations, :features).search_for("location = TestLocation1").all.to_a; p capsules.size }
8
#<Benchmark::Tms:0x000055d0ea625758 @label="", @real=1.1359932541381568, @cstime=0.0, @cutime=0.0, @stime=0.027599000000000373, @utime=0.41822099999999907, @total=0.44581999999999944>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants