Ho to set couchjs heap limit in CouchDB, with a special case for 2.3.1

[raoyitao]

Got JSON.Parse error (out of memory) when querying by view, but cannot enlarge the memory limit for CouchJS

Description

I have a big json document, ~8Mib, and I have a view in design doc, before I upload that big json to database, the view can work, but after that json uploaded, I get the error log:

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
out of memory
SyntaxError: JSON.parse
Stacktrace:
()@./share/server/main.js:1576
()@./share/server/main.js:1603
@./share/server/main.js:1
Failed to execute script.
[error] 2020-06-03T06:23:55.622680Z [email protected] <0.7929.545> -------- OS Process Error <0.9916.545> :: {os_process_error,{exit_status,1}}
[info] 2020-06-03T06:23:55.622835Z [email protected] <0.226.0> -------- couch_proc_manager <0.9916.545> died normal
[error] 2020-06-03T06:23:55.623181Z [email protected] emulator -------- Error in process <0.7929.545> on node '[email protected]' with exit value:
{{nocatch,{os_process_error,{exit_status,1}}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,68}]},{couch_mrview_updater,'-map_docs/2-fun-0-',3,[{file,"src/couch_mrview_updater.erl"},{line,199}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{couch_mrview_updater,map_docs,2,[{file,"src/couch_mrview_updater.erl"},{line,206}]}]}

[error] 2020-06-03T06:23:55.623753Z [email protected] <0.9440.545> bfcf299af8 rexi_server: from: [email protected](<0.9427.545>) mfa: fabric_rpc:map_view/5 throw:{os_process_error,{exit_status,1}} [{couch_mrview_util,get_view_index_state,5,[{file,"src/couch_mrview_util.erl"},{line,137}]},{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,81}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,247}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[error] 2020-06-03T06:23:55.624153Z [email protected] <0.9427.545> bfcf299af8 req_err(4089041121) os_process_error : {exit_status,1}
[<<"couch_mrview_util:get_view_index_state/5 L137">>,<<"couch_mrview_util:get_view/4 L81">>,<<"couch_mrview:query_view/6 L247">>,<<"rexi_server:init_p/3 L140">>]
[notice] 2020-06-03T06:23:55.624458Z [email protected] <0.9427.545> bfcf299af8 127.0.0.1:8430 161.144.139.30 undefined GET /testplan/_design/testdb/_view/by_upload_time?skip=0&limit=21&reduce=false 500 ok 13431
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Seems that the CouchJS reaches its memory limit, as I split that big json into 2 parts, and upload them respectively, then the issue went away. So I think I need to enlarge the memory limit. I found it in document: https://docs.couchdb.org/en/2.3.1/config/query-servers.html

By default, couchjs limits the max runtime allocation to 64MiB. If you run into out of memory issue in your ddoc functions, you can adjust the memory limitation (here, increasing to 512 MiB):

COUCHDB_QUERY_SERVER_JAVASCRIPT="/usr/bin/couchjs -S 536870912 /usr/share/server/main.js"

I tried that, change the value in "local.ini", even I make the limitation up to 2GiB, but it didn't work, still got "out of memory" error. As I observe the memory used by couch process, at most it takes up to ~300MiB

Your Environment

• CouchDB version used: 2.3.1
• Browser name and version: Chrome V81
• Operating system and version: redhat enterprise 7.4

Additional Context

[janl]

In CouchDB 2.3.1 you will have to patch your couchdb script manually for an override: https://github.com/apache/couchdb/blob/2.3.1/rel/files/couchdb.in#L29

This is fixed in 3.0.0: https://github.com/apache/couchdb/blob/3.0.0/rel/files/couchdb.in#L39

[raoyitao]

Thanks, do you mean that I need to download the source, and change that line in "couchdb/rel/files/couchdb.in" from
export COUCHDB_QUERY_SERVER_JAVASCRIPT="bin/couchjs share/server/main.js"
to
export COUCHDB_QUERY_SERVER_JAVASCRIPT="{{prefix}}/bin/couchjs {{prefix}}/share/server/main.js"
and then build the source to get couchdb dist ?

[janl]

Thanks, do you mean that I need to download the source, and change that line in "couchdb/rel/files/couchdb.in" from
export COUCHDB_QUERY_SERVER_JAVASCRIPT="bin/couchjs share/server/main.js"
to
export COUCHDB_QUERY_SERVER_JAVASCRIPT="{{prefix}}/bin/couchjs {{prefix}}/share/server/main.js"
and then build the source to get couchdb dist ?

No, all you need to do is find the existing couchdb script, and edit the line with COUCHDB_QUERY_SERVER_JAVASCRIPT to include your heap limit.