Coverage reports are blocking normal compile jobs and often fail to execute due OOM

[nuclearcat]

It seems coverage report need a lot of resources and we cannot afford running it in main compile cluster:

kci-6911ae842fd2377ea99315fe-coverage-report-x86-4rvl1n52-ddwcg   0/1     OOMKilled   0          88m
kci-6911ae842fd2377ea99315fe-coverage-report-x86-4rvl1n52-fwvqd   0/1     OOMKilled   0          3h25m
kci-6911ae842fd2377ea99315fe-coverage-report-x86-4rvl1n52-ql82p   1/1     Running     0          36m
kci-6911ae842fd2377ea99315fe-coverage-report-x86-4rvl1n52-zmv47   0/1     OOMKilled   0          137m
kci-6911c0062fd2377ea99363d5-coverage-report-x86-2ybj9m4x-pcbrw   0/1     Pending     0          130m
kci-6911c1422fd2377ea993760d-coverage-report-arm64-q6blwj5hq5zc   0/1     Pending     0          125m
kci-6911c14b2fd2377ea993762f-coverage-report-x86-9wus8klz-rzdbv   0/1     Pending     0          124m
kci-6911c1582fd2377ea9937647-coverage-report-arm64-wmosa6zmgrsq   0/1     Pending     0          124m
kci-6911c17b2fd2377ea993776b-coverage-report-x86-3y6afr90-d2q8s   0/1     Pending     0          124m
kci-6911c1c12fd2377ea9937b1a-coverage-report-x86-9547uejz-s98m9   0/1     Pending     0          122m
kci-6911c1f02fd2377ea9937ceb-coverage-report-x86-heb7o2tk-plcp8   0/1     Pending     0          122m
kci-6911c2392fd2377ea9938028-coverage-report-x86-gu92eyz3-56mxz   0/1     Pending     0          120m
kci-6911c25d2fd2377ea9938268-coverage-report-x86-xvkb32ou-bxn9v   1/1     Running     0          120m
kci-6911c42c2fd2377ea993962f-coverage-report-x86-drn52hsc-865gd   0/1     Pending     0          112m
kci-6911c47d2fd2377ea9939c05-coverage-report-x86-bcolh3ay-h4rxx   0/1     Completed   0          111m
kci-6911c4b12fd2377ea9939d89-coverage-report-x86-ivo56ztq-cbnxb   0/1     Pending     0          110m
kci-6911c4cc2fd2377ea9939ecd-coverage-report-x86-qn9m5fgl-kzvw8   0/1     Pending     0          109m
kci-6911c4d72fd2377ea993a04a-coverage-report-x86-0nszygr1-wq8s8   0/1     Pending     0          109m
kci-6911c50b2fd2377ea993a39a-coverage-report-x86-j8eg2zl0-cnbb9   0/1     Pending     0          108m
kci-6911c51c2fd2377ea993a3a3-coverage-report-x86-a2bw9cj6-bp2v4   0/1     Completed   0          108m
kci-6911c5332fd2377ea993a498-coverage-report-x86-yizpx8wo-cqn5l   0/1     Pending     0          108m
kci-6911c5672fd2377ea993a653-coverage-report-x86-ra45vjy3-6n6vs   0/1     Pending     0          107m
kci-6911c5842fd2377ea993a708-coverage-report-x86-j2a4zg8l-c42kh   0/1     Pending     0          106m
kci-6911c5892fd2377ea993a70e-coverage-report-x86-om5bsjtx-prxg8   0/1     Pending     0          106m
kci-6911c5b32fd2377ea993a7cf-coverage-report-x86-nd1mg9kr-tchzq   0/1     Pending     0          106m
kci-6911c6d42fd2377ea993b6be-coverage-report-x86-i8whkxvp-js6ln   0/1     Pending     0          101m
kci-6911c6e02fd2377ea993b6db-coverage-report-x86-w4cugt5q-rzfc5   0/1     Pending     0          101m
kci-6911c7142fd2377ea993ba83-coverage-report-x86-wskiqn31-smdmj   0/1     Pending     0          100m
kci-6911c74a2fd2377ea993be52-coverage-report-x86-k0gty2bn-fkzkx   0/1     Pending     0          99m
kci-6911c76b2fd2377ea993c09d-coverage-report-x86-jn258a70-mjs9r   0/1     Pending     0          98m
kci-6911c78f2fd2377ea993c258-coverage-report-x86-mnbwq4yl-q78md   0/1     Pending     0          98m
kci-6911c7ad2fd2377ea993c33b-coverage-report-x86-malbf8pq-2drnd   1/1     Running     0          97m
kci-6911c7c42fd2377ea993c376-coverage-report-x86-602k9re1-c76h4   0/1     Pending     0          97m
kci-6911c7e72fd2377ea993c4b0-coverage-report-x86-kyqp60ja-4mq9n   0/1     Pending     0          96m
kci-6911c81c2fd2377ea993c717-coverage-report-x86-l2adsihq-dq8lv   0/1     Pending     0          95m
kci-6911c8402fd2377ea993c99c-coverage-report-x86-y4w5z0f1-98fq9   0/1     Completed   0          95m
kci-6911c8632fd2377ea993cc8d-coverage-report-x86-y1ljacu2-rlkwx   0/1     Pending     0          94m
kci-6911c91a2fd2377ea993d5a2-coverage-report-x86-d5ow68k7-tncrm   0/1     Pending     0          91m
kci-6911c9372fd2377ea993d6f6-coverage-report-x86-oxvydjp9-cfznm   1/1     Running     0          91m
kci-6911c9432fd2377ea993d78c-coverage-report-x86-0iopmqkv-bv6qh   0/1     Completed   0          90m
kci-6911c9492fd2377ea993d792-coverage-report-x86-e2ky7i6m-hlbz5   0/1     Pending     0          90m
kci-6911ca272fd2377ea993e415-coverage-report-x86-7dgs54cf-wnn8n   0/1     Pending     0          87m
kci-6911ca402fd2377ea993e4ad-coverage-report-x86-pxafegnk-z9wnt   0/1     Pending     0          86m
kci-6911ca592fd2377ea993e6bf-coverage-report-x86-b37dnexz-nlh6g   0/1     Pending     0          86m
kci-6911ca5f2fd2377ea993e6c8-coverage-report-x86-gdcpihar-zjntm   1/1     Running     0          86m
kci-6911ca922fd2377ea993e8dd-coverage-report-x86-l1vz8ant-jf6x5   0/1     Pending     0          85m
kci-6911ca982fd2377ea993e92f-coverage-report-x86-w1uja9d8-bkxp4   0/1     Pending     0          85m
kci-6911cb142fd2377ea993eef1-coverage-report-x86-ht58i3ws-886k6   1/1     Running     0          83m
kci-6911cb282fd2377ea993efd7-coverage-report-x86-k4im9upe-8g8zb   0/1     Pending     0          82m
kci-6911cb672fd2377ea993f3f7-coverage-report-x86-km78uf1l-w6bjx   1/1     Running     0          81m
kci-6911cb6d2fd2377ea993f40f-coverage-report-x86-316gpx9f-trn9l   0/1     Pending     0          81m
kci-6911cc8c2fd2377ea993fec6-coverage-report-x86-clu0vn3m-s8v8c   1/1     Running     0          76m
kci-6911cc922fd2377ea993fedb-coverage-report-x86-2i04v1dg-z7tw2   0/1     Pending     0          76m
kci-6911ccae2fd2377ea99402e3-coverage-report-x86-al2d4jei-hr6jq   0/1     Pending     0          76m
kci-6911ccb42fd2377ea9940333-coverage-report-x86-7reugms8-vzjxz   0/1     Pending     0          76m
kci-6911ccf02fd2377ea9940668-coverage-report-x86-7mxaqkbh-h9vpm   0/1     Pending     0          75m
kci-6911ce132fd2377ea9940fe0-coverage-report-arm64-x5s7pcnklhjf   0/1     Pending     0          70m
kci-6911ce192fd2377ea994101f-coverage-report-x86-t0oeauji-fz6mm   0/1     Pending     0          70m
kci-6911ce2d2fd2377ea9941068-coverage-report-x86-zgs2bktr-d8gkn   1/1     Running     0          69m
kci-6911cec42fd2377ea99413cb-coverage-report-x86-dx8wirpn-fpwts   1/1     Running     0          67m
kci-6911cecb2fd2377ea994149f-coverage-report-x86-e8np9a7u-k24db   0/1     Completed   0          67m
kci-6911d05d2fd2377ea9941f4e-coverage-report-x86-bh6kso0u-npw9m   0/1     Completed   0          60m
kci-6911d0b32fd2377ea99420b9-coverage-report-x86-rsfvi7go-wgg2s   0/1     Pending     0          59m
kci-6911d0ba2fd2377ea99420c0-coverage-report-x86-7bkzd9ep-v26x9   0/1     Pending     0          59m
kci-6911d1ba2fd2377ea9942a44-kbuild-gcc-12-arm64-buil-ifal9ft45   0/1     Pending     0          54m
kci-6911d1c02fd2377ea9942acc-kbuild-gcc-12-arm64-chro-c7strnnk6   1/1     Running     0          54m
kci-6911d1dc2fd2377ea9942b88-kbuild-gcc-12-arm64-chro-jme5kh7ff   0/1     Pending     0          54m
kci-6911d1f72fd2377ea9942bc7-coverage-report-x86-560grbz8-tt5m8   0/1     Pending     0          53m
kci-6911d20c2fd2377ea9942bd6-coverage-report-x86-n263jcr0-46l78   1/1     Running     0          53m
kci-6911d2122fd2377ea9942bdc-coverage-report-x86-dhbi0sn9-sml2p   0/1     Pending     0          53m
kci-6911d2192fd2377ea9942c7e-coverage-report-x86-rm4geq72-7nbwt   0/1     Pending     0          53m
kci-6911d2272fd2377ea9942caf-coverage-report-x86-qfm40j2w-ttgl9   0/1     Pending     0          52m
kci-6911d2502fd2377ea9942f07-coverage-report-x86-v9bhiufm-8qvtp   0/1     Pending     0          52m
kci-6911d32c2fd2377ea994330a-coverage-report-x86-pvg524wt-j6zpw   1/1     Running     0          48m
kci-6911d3392fd2377ea9943313-coverage-report-x86-upmgjvy4-vbkvz   0/1     Pending     0          48m
kci-6911d3402fd2377ea9943324-coverage-report-x86-t2c9zph8-zkr8n   1/1     Running     0          48m
kci-6911d34d2fd2377ea9943373-coverage-report-x86-lg1oi7uq-vlfn7   0/1     Pending     0          48m
kci-6911d3912fd2377ea99433fc-coverage-report-x86-u7k15fyp-cftmm   0/1     Pending     0          46m
kci-6911d4072fd2377ea99436c8-coverage-report-x86-c7b31hqk-287sn   1/1     Running     0          44m
kci-6911d4672fd2377ea99437f7-coverage-report-x86-c8zxtkfa-hckk7   1/1     Running     0          43m
kci-6911d47b2fd2377ea994381f-coverage-report-x86-ywsi4kz7-wsg2v   1/1     Running     0          42m
kci-6911d4822fd2377ea9943838-coverage-report-x86-03xd8q1e-bfzsx   0/1     Pending     0          42m
kci-6911d49e2fd2377ea99438ad-coverage-report-x86-ua4xodtz-whk7h   0/1     Pending     0          42m
kci-6911d4cf2fd2377ea9943904-coverage-report-x86-nxiduelp-kxzzm   0/1     Completed   0          41m
kci-6911d4d62fd2377ea994390a-coverage-report-x86-pzh4wjfg-9qn7s   1/1     Running     0          41m
kci-6911d4dc2fd2377ea994390d-coverage-report-x86-p7yde9va-2xhbt   0/1     Pending     0          41m
kci-6911d4f92fd2377ea9943a0a-coverage-report-x86-bd3chrg6-cgzcn   0/1     Pending     0          40m
kci-6911d50d2fd2377ea9943a19-coverage-report-x86-2nzsrki1-hs8r2   0/1     Pending     0          40m
kci-6911d5222fd2377ea9943b61-coverage-report-x86-cez620b3-qdpfh   1/1     Running     0          40m
kci-6911d5532fd2377ea9943c43-coverage-report-x86-b0egowuf-kqvzm   1/1     Running     0          39m
kci-6911d5922fd2377ea9943ebb-coverage-report-x86-ah8xryv1-2hm7f   0/1     Pending     0          38m
kci-6911d63b2fd2377ea9943fbd-coverage-report-x86-l301ymki-wmk54   0/1     Pending     0          35m
kci-6911d6412fd2377ea9943fc0-coverage-report-x86-e2i4fy30-jp9nd   0/1     Pending     0          35m
kci-6911d6722fd2377ea99440b6-coverage-report-x86-vtnpwcmq-66jhm   0/1     Pending     0          34m
kci-6911d6882fd2377ea99440c2-coverage-report-x86-qs4wfmxu-vbffd   0/1     Pending     0          34m
kci-6911d6b22fd2377ea9944285-coverage-report-x86-y2h4d8vf-28bdv   0/1     Pending     0          33m
kci-6911d6c72fd2377ea99442c1-coverage-report-x86-gc5pkuy9-9v4jf   0/1     Pending     0          33m
kci-6911d6f82fd2377ea99443cf-coverage-report-x86-x1k8bnhj-g9m9j   0/1     Pending     0          32m
kci-6911d7142fd2377ea9944434-coverage-report-x86-8xeytvuq-f7szk   0/1     Pending     0          31m
kci-6911d7402fd2377ea9944496-coverage-report-x86-ie8z9b7y-7ffqg   0/1     Pending     0          31m
kci-6911d7782fd2377ea99445ef-coverage-report-x86-7e8l4ajc-hnlf8   0/1     Pending     0          30m
kci-6911d7872fd2377ea99445fb-coverage-report-x86-f4i2e5vh-slxrh   0/1     Pending     0          29m
kci-6911d78e2fd2377ea9944601-coverage-report-x86-ew4zfjrg-s4hn9   0/1     Pending     0          29m
kci-6911d7b12fd2377ea99446c7-coverage-report-x86-qh9rxbpl-fkfcc   0/1     Completed   0          29m
kci-6911d7e32fd2377ea9944715-coverage-report-x86-mixz57df-6b9jn   0/1     Pending     0          28m
kci-6911d8002fd2377ea99447b1-coverage-report-x86-faehkwcn-rbmxz   0/1     Completed   0          27m
kci-6911d8322fd2377ea994481c-coverage-report-x86-3wcu2b1s-hmw5p   0/1     Pending     0          27m
kci-6911d8392fd2377ea994481f-coverage-report-x86-lgu08mvz-sgzxh   0/1     Pending     0          27m
kci-6911d8472fd2377ea99448a9-coverage-report-x86-gevjnqd7-4sphr   1/1     Running     0          26m
kci-6911d8632fd2377ea99448fc-coverage-report-x86-6e10fk4q-zrr84   0/1     Completed   0          26m
kci-6911d86a2fd2377ea99448ff-coverage-report-x86-b9oap72j-92js6   0/1     Completed   0          26m
kci-6911d8802fd2377ea9944916-coverage-report-x86-nrha68eb-8cw8m   0/1     Completed   0          25m
kci-6911d8a42fd2377ea99449e5-coverage-report-x86-zbuam10c-qdjpc   0/1     Completed   0          25m
kci-6911d8ab2fd2377ea99449f6-coverage-report-x86-dty9zolc-z94gs   0/1     Completed   0          25m
kci-6911d8c02fd2377ea9944a0d-coverage-report-x86-qnp7g4au-rfm54   0/1     Completed   0          24m
kci-6911d8f42fd2377ea9944a7b-coverage-report-x86-kv85rcgj-wqvsv   1/1     Running     0          23m
kci-6911d9292fd2377ea9944b43-coverage-report-x86-6m327kjh-8fcxm   0/1     Pending     0          23m
kci-6911d92f2fd2377ea9944b46-coverage-report-x86-1vj8fk5a-zcnb9   0/1     Pending     0          22m
kci-6911d9532fd2377ea9944ba5-coverage-report-x86-exs23h65-77h7s   0/1     Pending     0          22m
kci-6911d95a2fd2377ea9944be0-coverage-report-x86-2eupyj3l-gctzk   0/1     Pending     0          22m
kci-6911d97f2fd2377ea9944c9a-coverage-report-x86-aldhcjb0-sct6j   1/1     Running     0          21m
kci-6911d9ab2fd2377ea9944cbf-coverage-report-x86-0erbshau-zrv92   1/1     Running     0          20m
kci-6911d9b92fd2377ea9944ccb-coverage-report-x86-iwjlduak-wg89z   1/1     Running     0          20m
kci-6911d9c02fd2377ea9944cce-coverage-report-x86-2kb5y1ct-59bvq   0/1     Completed   0          20m
kci-6911d9eb2fd2377ea9944dea-coverage-report-x86-qo8u3gfx-j64hz   1/1     Running     0          19m
kci-6911d9f92fd2377ea9944df1-coverage-report-x86-84di7jnc-2pjvk   0/1     Pending     0          19m
kci-6911da002fd2377ea9944e47-coverage-report-x86-zmpvkti2-49l2k   0/1     Pending     0          19m
kci-6911da1d2fd2377ea9944e65-coverage-report-x86-mjznfh7u-tfkrz   0/1     Pending     0          18m
kci-6911da422fd2377ea9944e90-coverage-report-x86-7obqkps4-shgt9   0/1     Pending     0          18m
kci-6911da5f2fd2377ea9944ea7-coverage-report-x86-ljar5p4c-7b7l8   0/1     Pending     0          17m
kci-6911daab2fd2377ea9944f7e-coverage-report-x86-r3ti8yne-dd4ft   0/1     Pending     0          16m
kci-6911dab22fd2377ea9944f84-coverage-report-x86-97cju34a-k89tw   0/1     Pending     0          16m
kci-6911dac92fd2377ea9944f8e-coverage-report-x86-0va6ug5c-swqxb   1/1     Running     0          16m
kci-6911dae02fd2377ea9944fa0-coverage-report-x86-ia3m7x9z-bbvd6   1/1     Running     0          15m
kci-6911daf72fd2377ea994503d-coverage-report-x86-q3j785bz-vcxpl   0/1     Completed   0          15m
kci-6911db352fd2377ea9945330-coverage-report-x86-mxbw53ik-wlpz5   1/1     Running     0          14m
kci-6911db442fd2377ea9945340-coverage-report-x86-skezgxau-f78wd   0/1     Pending     0          14m
kci-6911db4b2fd2377ea994535b-coverage-report-x86-gqdcko0j-22qgd   0/1     Pending     0          13m
kci-6911db692fd2377ea9945385-coverage-report-x86-vgufqldx-c4k7l   0/1     Pending     0          13m
kci-6911db702fd2377ea99453a4-coverage-report-x86-l8vz60of-d8sct   0/1     Pending     0          13m
kci-6911db7e2fd2377ea99453c5-coverage-report-x86-pi5ehg6b-rdkcj   0/1     Pending     0          13m
kci-6911db852fd2377ea99453c9-coverage-report-x86-psb26muz-wmxb6   0/1     Pending     0          12m
kci-6911db9b2fd2377ea99453d8-coverage-report-x86-qdepzsou-j72r4   0/1     Pending     0          12m
kci-6911dbc82fd2377ea99453ee-coverage-report-x86-lo9qpd6k-ksxzs   0/1     Pending     0          11m
kci-6911dbcf2fd2377ea9945400-coverage-report-x86-6n4dgtsh-kfz5q   0/1     Pending     0          11m
kci-6911dbed2fd2377ea9945448-coverage-report-x86-nk74ltya-82dcz   0/1     Pending     0          11m
kci-6911dbf42fd2377ea994544b-coverage-report-x86-syr9e5b7-4k8xf   0/1     Pending     0          11m
kci-6911dc202fd2377ea9945470-coverage-report-x86-b6lrvefg-g9rsn   0/1     Pending     0          10m
kci-6911dc2f2fd2377ea994547d-coverage-report-x86-123nlj0s-gzb6n   0/1     Pending     0          10m
kci-6911dc452fd2377ea994549e-coverage-report-x86-me52qb3y-cn8r2   0/1     Pending     0          9m43s
kci-6911dc5c2fd2377ea9945578-coverage-report-x86-81esj0xk-l8jrz   0/1     Pending     0          9m21s
kci-6911dc6b2fd2377ea9945597-coverage-report-x86-xu36inay-jls58   0/1     Pending     0          9m7s
kci-6911dc802fd2377ea99455b8-coverage-report-x86-4tl9apqg-ss48n   0/1     Pending     0          8m45s
kci-6911dcc42fd2377ea994567b-coverage-report-x86-9fhk7zbr-w9w55   0/1     Pending     0          7m37s
kci-6911dd002fd2377ea99456ad-coverage-report-x86-gr64ziyh-nggqt   0/1     Pending     0          6m38s
kci-6911dd0e2fd2377ea99456b6-coverage-report-x86-f8v35odc-xv464   0/1     Pending     0          6m23s
kci-6911dd3b2fd2377ea99456ce-coverage-report-x86-6wti98sr-88g9c   0/1     Pending     0          5m38s
kci-6911dd4a2fd2377ea99456e2-coverage-report-x86-ap95lfj8-qm9f6   0/1     Pending     0          5m22s
kci-6911dd612fd2377ea9945700-coverage-report-x86-rgh61d4n-86fg7   0/1     Pending     0          5m
kci-6911dd782fd2377ea9945729-coverage-report-x86-92v6j38l-przkk   0/1     Pending     0          4m38s
kci-6911dd952fd2377ea9945763-coverage-report-x86-cjspekaz-qn6w9   0/1     Pending     0          4m8s
kci-6911dd9d2fd2377ea9945766-coverage-report-x86-jbw2xe0m-2t8b5   0/1     Pending     0          4m
kci-6911ddcb2fd2377ea9945805-coverage-report-x86-9ygms3ur-mn5t7   0/1     Pending     0          3m14s
kci-6911ddd22fd2377ea9945808-coverage-report-x86-lt6y9kiu-2zs9x   0/1     Pending     0          3m7s
kci-6911dde92fd2377ea994589d-coverage-report-x86-k9g6so87-xz8z5   0/1     Pending     0          2m44s
kci-6911ddf82fd2377ea99458a6-coverage-report-x86-n6j4rb7o-mm7nn   0/1     Pending     0          2m29s
kci-6911de2e2fd2377ea99458cf-coverage-report-x86-pa0i9qrf-tbbck   1/1     Running     0          95s
kci-6911de352fd2377ea99458d5-coverage-report-x86-v4m2j6gc-2pmvp   0/1     Pending     0          88s
kci-6911de552fd2377ea9945923-coverage-report-x86-x0wv4bed-4hhnh   0/1     Pending     0          56s
kci-6911de6c2fd2377ea994598b-coverage-report-x86-4ecm9i0j-lv78c   0/1     Pending     0          33s
kci-6911de7b2fd2377ea9945994-coverage-report-x86-0qkpcu29-f8mmh   0/1     Pending     0          18s
kci-6911de822fd2377ea994599d-coverage-report-x86-uecl1nda-wrz6t   0/1     Pending     0          11s

Action items:

• [] Disable coverage report and do emergency production update
• [] Move coverage report to separate k8s cluster if we have such available