Skip to content

Fixed the memory leak problem #3070

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 10, 2025

Conversation

qhyou11
Copy link
Contributor

@qhyou11 qhyou11 commented May 30, 2025

I have a task template when starting a job with it , it would call other tasks and query the status of these tasks. After a few hours running ,the RSS of semaphore process can raise to nearly 6GB. So I dig into the pprof :
╰─➤ go tool pprof ./semaphore mem-1748411057.prof
File: semaphore
Build ID: bc74ae534f682663ad480d52d875d15dff56ceea
Type: inuse_space
Time: 2025-05-28 13:44:17
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 3440.74MB, 81.64% of 4214.56MB total
Dropped 58 nodes (cum <= 21.07MB)
Showing top 10 nodes out of 74
flat flat% sum% cum cum%
741.16MB 17.59% 17.59% 757.16MB 17.97% net/textproto.readMIMEHeader
473.54MB 11.24% 28.82% 582.55MB 13.82% encoding/json.(*decodeState).literalStore
456.68MB 10.84% 39.66% 456.68MB 10.84% github.com/gorilla/context.Set
398.12MB 9.45% 49.10% 398.12MB 9.45% net/http.(*Request).WithContext
366.10MB 8.69% 57.79% 366.10MB 8.69% github.com/gorilla/mux.extractVars
297.57MB 7.06% 64.85% 769.61MB 18.26% github.com/semaphoreui/semaphore/api/projects.GetTaskMiddleware.func1
206.02MB 4.89% 69.74% 343.53MB 8.15% context.withCancel
181.02MB 4.30% 74.03% 181.02MB 4.30% net/url.parse
171.51MB 4.07% 78.10% 171.51MB 4.07% context.WithValue
149.02MB 3.54% 81.64% 262.02MB 6.22% github.com/semaphoreui/semaphore/api.authenticationHandler
(pprof) list github.com/semaphoreui/semaphore/api/projects.GetTaskMiddleware.fun
c1
Total: 4.12GB
ROUTINE ======================== github.com/semaphoreui/semaphore/api/projects.GetTaskMiddleware.func1 in /go/src/semaphore/api/projects/tasks.go
297.57MB 769.61MB (flat, cum) 18.26% of Total
. . 105:func GetTaskMiddleware(next http.Handler) http.Handler {
. . 106: return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
. . 107: project := context.Get(r, "project").(db.Project)
. . 108: taskID, err := helpers.GetIntParam("task_id", w, r)
. . 109:
. . 110: if err != nil {
. . 111: util.LogErrorF(err, log.Fields{"error": "Bad request. Cannot get task_id from request"})
. . 112: w.WriteHeader(http.StatusBadRequest)
. . 113: return
. . 114: }
. 472.04MB 115:
. . 116: task, err := helpers.Store(r).GetTask(project.ID, taskID)
. . 117: if err != nil {
. . 118: util.LogErrorF(err, log.Fields{"error": "Bad request. Cannot get task from database"})
. . 119: w.WriteHeader(http.StatusBadRequest)
. . 120: return
. . 121: }
297.57MB 297.57MB 122:
. . 123: context.Set(r, "task", task)
. . 124: next.ServeHTTP(w, r)
. . 125: })
. . 126:}
. . 127:

And I do find out that there was memory leak in gorilla/context. for Get function , it just get a value from *http.Request which won't be a problem. But for Set function, it will store an object in a global map object map[*http.Request]map[interface{}]interface{} , so it won't be released after the response is send back to client.
Unfortunately the task object seems bigger than other semaphore object , so it consume more memory . So I clear the context object just after it finished the api call dedicately.For other objects , the memory problem seems no so obviously, if did cause a memory leak, we can call a gc to purge context objects which created 10 minutes ago.

And as suggest by gorilla/context:
You should use the http.Request.Context() feature in Go 1.7.

Fixed the memory leak while using gorilla/context.Set. Clear the global map object with current request after tasks api return or manually release all the expired objects in context using api of gc.

…al map object with current request after tasks api return or manually release all the expired objects in context using api of gc.
@fiftin fiftin merged commit 20631c5 into semaphoreui:develop Jun 10, 2025
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants