Too many profiles?

[mkst]

Tailing the logs, profiles seem to be created every time a compile request is made:

16-
17:03:46:25 DEBUG Made new profile: Agile Ant (anon)
18-03:46:25 DEBUG Sandbox Command: WINE=wine WIBO=wibo INPUT=/tmp/code.c OUTPUT=/tmp/object.o COMPILER_DIR=/var/www/decomp.me/repo/backend/compilers/ps1/gcc2.6.3-psx COMPILER_FLAGS='-O2 -G0 -funsigned-char -gcoff -Wa,--expand-div,--expand-li' FUNCTION=SsVabOpen MWCIncludes=/tmp TMPDIR=/tmp /bin/nsjail --mode o --chroot /var/www/decomp.me/repo/sandbox/root --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpuif8f8cn:/tmp --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpuif8f8cn:/run/user/1000 --bindmount_ro /dev --bindmount_ro /bin --bindmount_ro /etc/alternatives --bindmount_ro /etc/fonts --bindmount_ro /etc/passwd --bindmount_ro /lib --bindmount_ro /lib32 --bindmount_ro /lib64 --bindmount_ro /usr --bindmount_ro /proc --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpuif8f8cn:/var/tmp --bindmount_ro /var/www/decomp.me/repo/backend/compilers --bindmount_ro /var/www/decomp.me/repo/backend/libraries --env PATH=/usr/bin:/bin --cwd /tmp --rlimit_fsize soft --rlimit_nofile soft --bindmount_ro /tmp/wine:/wine --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpuif8f8cn/Temp:/wine/drive_c/users/ethteck/Temp --env WINEDEBUG=-all --env WINEPREFIX=/wine --really_quiet --bindmount_ro /var/www/decomp.me/repo/backend/compilers/ps1/gcc2.6.3-psx --env PATH --env WINE --env WIBO --env INPUT --env OUTPUT --env COMPILER_DIR --env COMPILER_FLAGS --env FUNCTION --env MWCIncludes --env TMPDIR -- /bin/bash -euo pipefail -c 'cpp -E -lang-c -nostdinc "${INPUT}" -o "${INPUT}".i && printf "%s" "${COMPILER_FLAGS}" | xargs -- ${COMPILER_DIR}/gcc -c -pipe -B${COMPILER_DIR}/ -o "${OUTPUT}" "${INPUT}.i"'
--
24-03:46:25 DEBUG Sandbox Command: COMPILER_BASE_PATH=/var/www/decomp.me/repo/backend/compilers /bin/nsjail --mode o --chroot /var/www/decomp.me/repo/sandbox/root --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmp4cqn08si:/tmp --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmp4cqn08si:/run/user/1000 --bindmount_ro /dev --bindmount_ro /bin --bindmount_ro /etc/alternatives --bindmount_ro /etc/fonts --bindmount_ro /etc/passwd --bindmount_ro /lib --bindmount_ro /lib32 --bindmount_ro /lib64 --bindmount_ro /usr --bindmount_ro /proc --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmp4cqn08si:/var/tmp --bindmount_ro /var/www/decomp.me/repo/backend/compilers --bindmount_ro /var/www/decomp.me/repo/backend/libraries --env PATH=/usr/bin:/bin --cwd /tmp --rlimit_fsize soft --rlimit_nofile soft --bindmount_ro /tmp/wine:/wine --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmp4cqn08si/Temp:/wine/drive_c/users/ethteck/Temp --env WINEDEBUG=-all --env WINEPREFIX=/wine --really_quiet --env PATH --env COMPILER_BASE_PATH -- /bin/bash -euo pipefail -c 'mips-linux-gnu-objdump --disassemble-zeroes --line-numbers --reloc --disassemble --start-address=0 -m mips:3000 /tmp/out.s'
25:03:46:26 DEBUG Made new profile: Smelly Clam (anon)
26-03:46:26 DEBUG Sandbox Command: WINE=wine WIBO=wibo INPUT=/tmp/code.c OUTPUT=/tmp/object.o COMPILER_DIR=/var/www/decomp.me/repo/backend/compilers/n64/ido5.3 COMPILER_FLAGS='-O2 -mips2' FUNCTION=func_800BB028 MWCIncludes=/tmp TMPDIR=/tmp /bin/nsjail --mode o --chroot /var/www/decomp.me/repo/sandbox/root --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpbfcz9p_o:/tmp --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpbfcz9p_o:/run/user/1000 --bindmount_ro /dev --bindmount_ro /bin --bindmount_ro /etc/alternatives --bindmount_ro /etc/fonts --bindmount_ro /etc/passwd --bindmount_ro /lib --bindmount_ro /lib32 --bindmount_ro /lib64 --bindmount_ro /usr --bindmount_ro /proc --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpbfcz9p_o:/var/tmp --bindmount_ro /var/www/decomp.me/repo/backend/compilers --bindmount_ro /var/www/decomp.me/repo/backend/libraries --env PATH=/usr/bin:/bin --cwd /tmp --rlimit_fsize soft --rlimit_nofile soft --bindmount_ro /tmp/wine:/wine --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpbfcz9p_o/Temp:/wine/drive_c/users/ethteck/Temp --env WINEDEBUG=-all --env WINEPREFIX=/wine --really_quiet --bindmount_ro /var/www/decomp.me/repo/backend/compilers/n64/ido5.3 --env PATH --env WINE --env WIBO --env INPUT --env OUTPUT --env COMPILER_DIR --env COMPILER_FLAGS --env FUNCTION --env MWCIncludes --env TMPDIR -- /bin/bash -euo pipefail -c 'USR_LIB="${COMPILER_DIR}" "${COMPILER_DIR}/cc" -c -Xcpluscomm -G0 -non_shared -Wab,-r4300_mul -woff 649,838,712 -32 ${COMPILER_FLAGS} -o "${OUTPUT}" "${INPUT}"'
--
41-
42:03:46:26 DEBUG Made new profile: Slow Fish (anon)
43-03:46:26 DEBUG Sandbox Command: WINE=wine WIBO=wibo INPUT=/tmp/code.c OUTPUT=/tmp/object.o COMPILER_DIR=/var/www/decomp.me/repo/backend/compilers/n64/ido5.3 COMPILER_FLAGS='-G 0 -nostdinc -DVERSION_US=1 -fullwarn -verbose -woff 624,649,838,712,516,513,596,564,594,709 -mips2 -EB -D_MIPS_SZLONG=32 -DNDEBUG -D_FINALROM -DF3DEX_GBI -DLANGUAGE_C -D_LANGUAGE_C -O2 -g3' FUNCTION=func_i4_80193EF0 MWCIncludes=/tmp TMPDIR=/tmp /bin/nsjail --mode o --chroot /var/www/decomp.me/repo/sandbox/root --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpf5d3zpbz:/tmp --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpf5d3zpbz:/run/user/1000 --bindmount_ro /dev --bindmount_ro /bin --bindmount_ro /etc/alternatives --bindmount_ro /etc/fonts --bindmount_ro /etc/passwd --bindmount_ro /lib --bindmount_ro /lib32 --bindmount_ro /lib64 --bindmount_ro /usr --bindmount_ro /proc --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpf5d3zpbz:/var/tmp --bindmount_ro /var/www/decomp.me/repo/backend/compilers --bindmount_ro /var/www/decomp.me/repo/backend/libraries --env PATH=/usr/bin:/bin --cwd /tmp --rlimit_fsize soft --rlimit_nofile soft --bindmount_ro /tmp/wine:/wine --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpf5d3zpbz/Temp:/wine/drive_c/users/ethteck/Temp --env WINEDEBUG=-all --env WINEPREFIX=/wine --really_quiet --bindmount_ro /var/www/decomp.me/repo/backend/compilers/n64/ido5.3 --env PATH --env WINE --env WIBO --env INPUT --env OUTPUT --env COMPILER_DIR --env COMPILER_FLAGS --env FUNCTION --env MWCIncludes --env TMPDIR -- /bin/bash -euo pipefail -c 'USR_LIB="${COMPILER_DIR}" "${COMPILER_DIR}/cc" -c -Xcpluscomm -G0 -non_shared -Wab,-r4300_mul -woff 649,838,712 -32 ${COMPILER_FLAGS} -o "${OUTPUT}" "${INPUT}"'
--
61-03:46:26 DEBUG https://o4504592968581120.ingest.sentry.io:443 "POST /api/4504592976904192/envelope/ HTTP/11" 200 0
62:03:46:27 DEBUG Made new profile: Herbivorous Bat (anon)
63-03:46:27 DEBUG Sandbox Command: WINE=wine WIBO=wibo INPUT=/tmp/code.c OUTPUT=/tmp/object.o COMPILER_DIR=/var/www/decomp.me/repo/backend/compilers/n64/ido5.3 COMPILER_FLAGS='-O2 -mips2' FUNCTION=func_8000C3D8 MWCIncludes=/tmp TMPDIR=/tmp /bin/nsjail --mode o --chroot /var/www/decomp.me/repo/sandbox/root --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpa7ez9ywx:/tmp --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpa7ez9ywx:/run/user/1000 --bindmount_ro /dev --bindmount_ro /bin --bindmount_ro /etc/alternatives --bindmount_ro /etc/fonts --bindmount_ro /etc/passwd --bindmount_ro /lib --bindmount_ro /lib32 --bindmount_ro /lib64 --bindmount_ro /usr --bindmount_ro /proc --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpa7ez9ywx:/var/tmp --bindmount_ro /var/www/decomp.me/repo/backend/compilers --bindmount_ro /var/www/decomp.me/repo/backend/libraries --env PATH=/usr/bin:/bin --cwd /tmp --rlimit_fsize soft --rlimit_nofile soft --bindmount_ro /tmp/wine:/wine --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpa7ez9ywx/Temp:/wine/drive_c/users/ethteck/Temp --env WINEDEBUG=-all --env WINEPREFIX=/wine --really_quiet --bindmount_ro /var/www/decomp.me/repo/backend/compilers/n64/ido5.3 --env PATH --env WINE --env WIBO --env INPUT --env OUTPUT --env COMPILER_DIR --env COMPILER_FLAGS --env FUNCTION --env MWCIncludes --env TMPDIR -- /bin/bash -euo pipefail -c 'USR_LIB="${COMPILER_DIR}" "${COMPILER_DIR}/cc" -c -Xcpluscomm -G0 -non_shared -Wab,-r4300_mul -woff 649,838,712 -32 ${COMPILER_FLAGS} -o "${OUTPUT}" "${INPUT}"'
--
163-03:46:28 DEBUG Sandbox Command: COMPILER_BASE_PATH=/var/www/decomp.me/repo/backend/compilers /bin/nsjail --mode o --chroot /var/www/decomp.me/repo/sandbox/root --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmplm1l7gyi:/tmp --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmplm1l7gyi:/run/user/1000 --bindmount_ro /dev --bindmount_ro /bin --bindmount_ro /etc/alternatives --bindmount_ro /etc/fonts --bindmount_ro /etc/passwd --bindmount_ro /lib --bindmount_ro /lib32 --bindmount_ro /lib64 --bindmount_ro /usr --bindmount_ro /proc --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmplm1l7gyi:/var/tmp --bindmount_ro /var/www/decomp.me/repo/backend/compilers --bindmount_ro /var/www/decomp.me/repo/backend/libraries --env PATH=/usr/bin:/bin --cwd /tmp --rlimit_fsize soft --rlimit_nofile soft --bindmount_ro /tmp/wine:/wine --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmplm1l7gyi/Temp:/wine/drive_c/users/ethteck/Temp --env WINEDEBUG=-all --env WINEPREFIX=/wine --really_quiet --env PATH --env COMPILER_BASE_PATH -- /bin/bash -euo pipefail -c 'mips-ps2-decompals-objdump --disassemble-zeroes --line-numbers --reloc --disassemble --start-address=0 -m mips:5900 /tmp/out.s'
164:03:46:29 DEBUG Made new profile: Shy Meerkat (anon)
165-03:46:29 DEBUG Sandbox Command: WINE=wine WIBO=wibo INPUT=/tmp/code.c OUTPUT=/tmp/object.o COMPILER_DIR=/var/www/decomp.me/repo/backend/compilers/n64/ido5.3 COMPILER_FLAGS='-O2 -mips2' FUNCTION=func_8000C3D8 MWCIncludes=/tmp TMPDIR=/tmp /bin/nsjail --mode o --chroot /var/www/decomp.me/repo/sandbox/root --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmp_jnyix2g:/tmp --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmp_jnyix2g:/run/user/1000 --bindmount_ro /dev --bindmount_ro /bin --bindmount_ro /etc/alternatives --bindmount_ro /etc/fonts --bindmount_ro /etc/passwd --bindmount_ro /lib --bindmount_ro /lib32 --bindmount_ro /lib64 --bindmount_ro /usr --bindmount_ro /proc --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmp_jnyix2g:/var/tmp --bindmount_ro /var/www/decomp.me/repo/backend/compilers --bindmount_ro /var/www/decomp.me/repo/backend/libraries --env PATH=/usr/bin:/bin --cwd /tmp --rlimit_fsize soft --rlimit_nofile soft --bindmount_ro /tmp/wine:/wine --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmp_jnyix2g/Temp:/wine/drive_c/users/ethteck/Temp --env WINEDEBUG=-all --env WINEPREFIX=/wine --really_quiet --bindmount_ro /var/www/decomp.me/repo/backend/compilers/n64/ido5.3 --env PATH --env WINE --env WIBO --env INPUT --env OUTPUT --env COMPILER_DIR --env COMPILER_FLAGS --env FUNCTION --env MWCIncludes --env TMPDIR -- /bin/bash -euo pipefail -c 'USR_LIB="${COMPILER_DIR}" "${COMPILER_DIR}/cc" -c -Xcpluscomm -G0 -non_shared -Wab,-r4300_mul -woff 649,838,712 -32 ${COMPILER_FLAGS} -o "${OUTPUT}" "${INPUT}"'
--
171-03:46:30 DEBUG https://o4504592968581120.ingest.sentry.io:443 "POST /api/4504592976904192/envelope/ HTTP/11" 200 0
172:03:46:30 DEBUG Made new profile: Tame Llama (anon)
173:03:46:32 DEBUG Made new profile: Cuddly Reindeer (anon)
174-03:46:32 DEBUG Sandbox Command: WINE=wine WIBO=wibo INPUT=/tmp/code.c OUTPUT=/tmp/object.o COMPILER_DIR=/var/www/decomp.me/repo/backend/compilers/n64/ido5.3 COMPILER_FLAGS='-O2 -mips2' FUNCTION=func_800B0F28 MWCIncludes=/tmp TMPDIR=/tmp /bin/nsjail --mode o --chroot /var/www/decomp.me/repo/sandbox/root --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpjepb5n3e:/tmp --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpjepb5n3e:/run/user/1000 --bindmount_ro /dev --bindmount_ro /bin --bindmount_ro /etc/alternatives --bindmount_ro /etc/fonts --bindmount_ro /etc/passwd --bindmount_ro /lib --bindmount_ro /lib32 --bindmount_ro /lib64 --bindmount_ro /usr --bindmount_ro /proc --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpjepb5n3e:/var/tmp --bindmount_ro /var/www/decomp.me/repo/backend/compilers --bindmount_ro /var/www/decomp.me/repo/backend/libraries --env PATH=/usr/bin:/bin --cwd /tmp --rlimit_fsize soft --rlimit_nofile soft --bindmount_ro /tmp/wine:/wine --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpjepb5n3e/Temp:/wine/drive_c/users/ethteck/Temp --env WINEDEBUG=-all --env WINEPREFIX=/wine --really_quiet --bindmount_ro /var/www/decomp.me/repo/backend/compilers/n64/ido5.3 --env PATH --env WINE --env WIBO --env INPUT --env OUTPUT --env COMPILER_DIR --env COMPILER_FLAGS --env FUNCTION --env MWCIncludes --env TMPDIR -- /bin/bash -euo pipefail -c 'USR_LIB="${COMPILER_DIR}" "${COMPILER_DIR}/cc" -c -Xcpluscomm -G0 -non_shared -Wab,-r4300_mul -woff 649,838,712 -32 ${COMPILER_FLAGS} -o "${OUTPUT}" "${INPUT}"'
--
210-03:46:32 DEBUG Sandbox Command: COMPILER_BASE_PATH=/var/www/decomp.me/repo/backend/compilers /bin/nsjail --mode o --chroot /var/www/decomp.me/repo/sandbox/root --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpiq_ucgj2:/tmp --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpiq_ucgj2:/run/user/1000 --bindmount_ro /dev --bindmount_ro /bin --bindmount_ro /etc/alternatives --bindmount_ro /etc/fonts --bindmount_ro /etc/passwd --bindmount_ro /lib --bindmount_ro /lib32 --bindmount_ro /lib64 --bindmount_ro /usr --bindmount_ro /proc --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpiq_ucgj2:/var/tmp --bindmount_ro /var/www/decomp.me/repo/backend/compilers --bindmount_ro /var/www/decomp.me/repo/backend/libraries --env PATH=/usr/bin:/bin --cwd /tmp --rlimit_fsize soft --rlimit_nofile soft --bindmount_ro /tmp/wine:/wine --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpiq_ucgj2/Temp:/wine/drive_c/users/ethteck/Temp --env WINEDEBUG=-all --env WINEPREFIX=/wine --really_quiet --env PATH --env COMPILER_BASE_PATH -- /bin/bash -euo pipefail -c 'mips-linux-gnu-objdump --disassemble-zeroes --line-numbers --reloc --disassemble --start-address=0 -m mips:4300 /tmp/out.s'
211:03:46:33 DEBUG Made new profile: Cold-blooded Pony (anon)
212-03:46:33 DEBUG Sandbox Command: WINE=wine WIBO=wibo INPUT=/tmp/code.c OUTPUT=/tmp/object.o COMPILER_DIR=/var/www/decomp.me/repo/backend/compilers/gc_wii/mwcc_233_163n COMPILER_FLAGS='-O4,p -nodefaults -proc gekko -fp hard -Cpp_exceptions off -enum int -fp_contract on -inline auto -DM2CTX -DMUST_MATCH -DWIP' FUNCTION=it_802842A4 MWCIncludes=/tmp TMPDIR=/tmp /bin/nsjail --mode o --chroot /var/www/decomp.me/repo/sandbox/root --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpr1w__f6d:/tmp --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpr1w__f6d:/run/user/1000 --bindmount_ro /dev --bindmount_ro /bin --bindmount_ro /etc/alternatives --bindmount_ro /etc/fonts --bindmount_ro /etc/passwd --bindmount_ro /lib --bindmount_ro /lib32 --bindmount_ro /lib64 --bindmount_ro /usr --bindmount_ro /proc --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpr1w__f6d:/var/tmp --bindmount_ro /var/www/decomp.me/repo/backend/compilers --bindmount_ro /var/www/decomp.me/repo/backend/libraries --env PATH=/usr/bin:/bin --cwd /tmp --rlimit_fsize soft --rlimit_nofile soft --bindmount_ro /tmp/wine:/wine --bindmount /var/www/decomp.me/repo/sandbox/tmp/tmpr1w__f6d/Temp:/wine/drive_c/users/ethteck/Temp --env WINEDEBUG=-all --env WINEPREFIX=/wine --really_quiet --bindmount_ro /var/www/decomp.me/repo/backend/compilers/gc_wii/mwcc_233_163n --env PATH --env WINE --env WIBO --env INPUT --env OUTPUT --env COMPILER_DIR --env COMPILER_FLAGS --env FUNCTION --env MWCIncludes --env TMPDIR -- /bin/bash -euo pipefail -c 'printf "%s" "${COMPILER_FLAGS}" | xargs -x -- ${WIBO} "${COMPILER_DIR}/mwcceppc.exe" -pragma "msg_show_realref off" -c -proc gekko -nostdinc -stderr -o "${OUTPUT}" "${INPUT}"'

I don't think we need to be creating so many profiles (we have 5m i think)

[mkst]

So the code here isn't doing what we want.

decomp.me/backend/coreapp/middleware.py

Line 77 in eaca358

if not profile:

[mkst]

This Yandex bot is causing profiles to be created

95.108.213.242 - - [08/Feb/2025:09:56:12 +0100] "GET /api/scratch/U9B5m/family HTTP/1.1" 200 1255 "https://decomp.me/scratch/U9B5m" "Mozilla/5.0 (compatible; YandexRenderResourcesBot/1.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0"
95.108.213.240 - - [08/Feb/2025:09:56:12 +0100] "POST /api/scratch/U9B5m/compile HTTP/1.1" 200 9731 "https://decomp.me/scratch/U9B5m" "Mozilla/5.0 (compatible; YandexRenderResourcesBot/1.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0"
95.108.213.167 - - [08/Feb/2025:09:56:12 +0100] "GET /api/scratch/U9B5m HTTP/1.1" 200 33041 "https://decomp.me/scratch/U9B5m" "Mozilla/5.0 (compatible; YandexRenderResourcesBot/1.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0"
213.180.203.227 - - [08/Feb/2025:09:56:47 +0100] "GET /api/scratch/ZsYzK/family HTTP/1.1" 200 423 "https://decomp.me/scratch/ZsYzK" "Mozilla/5.0 (compatible; YandexRenderResourcesBot/1.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0"
95.108.213.127 - - [08/Feb/2025:09:56:47 +0100] "GET /api/scratch/ZsYzK HTTP/1.1" 200 32082 "https://decomp.me/scratch/ZsYzK" "Mozilla/5.0 (compatible; YandexRenderResourcesBot/1.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0"
213.180.203.62 - - [08/Feb/2025:09:56:47 +0100] "POST /api/scratch/ZsYzK/compile HTTP/1.1" 200 25944 "https://decomp.me/scratch/ZsYzK" "Mozilla/5.0 (compatible; YandexRenderResourcesBot/1.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0"
95.108.213.207 - - [08/Feb/2025:09:56:55 +0100] "GET /api/scratch/apaBR HTTP/1.1" 200 32706 "https://decomp.me/scratch/apaBR" "Mozilla/5.0 (compatible; YandexRenderResourcesBot/1.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0"
95.108.213.168 - - [08/Feb/2025:09:56:55 +0100] "GET /scratch/1Mt2L HTTP/1.1" 200 10608 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0"
95.108.213.176 - - [08/Feb/2025:09:56:58 +0100] "POST /api/scratch/AKyUS/compile HTTP/1.1" 200 15886 "https://decomp.me/scratch/AKyUS" "Mozilla/5.0 (compatible; YandexRenderResourcesBot/1.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0"
95.108.213.153 - - [08/Feb/2025:09:57:08 +0100] "GET /scratch/1Mt2L HTTP/1.1" 200 10608 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0"
95.108.213.155 - - [08/Feb/2025:09:57:21 +0100] "GET /api/scratch/CTNJr HTTP/1.1" 200 32650 "https://decomp.me/scratch/CTNJr" "Mozilla/5.0 (compatible; YandexRenderResourcesBot/1.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0"
5.255.231.113 - - [08/Feb/2025:09:57:40 +0100] "GET /api/scratch/i5AS7 HTTP/1.1" 200 32504 "https://decomp.me/scratch/i5AS7" "Mozilla/5.0 (compatible; YandexRenderResourcesBot/1.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0"
95.108.213.125 - - [08/Feb/2025:09:57:40 +0100] "GET /api/scratch/i5AS7/family HTTP/1.1" 200 414 "https://decomp.me/scratch/i5AS7" "Mozilla/5.0 (compatible; YandexRenderResourcesBot/1.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0"
213.180.203.222 - - [08/Feb/2025:09:57:41 +0100] "POST /api/scratch/i5AS7/compile HTTP/1.1" 200 2547 "https://decomp.me/scratch/i5AS7" "Mozilla/5.0 (compatible; YandexRenderResourcesBot/1.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0"
95.108.213.240 - - [08/Feb/2025:09:58:05 +0100] "GET /api/scratch/FCirn/family HTTP/1.1" 200 1220 "https://decomp.me/scratch/FCirn" "Mozilla/5.0 (compatible; YandexRenderResourcesBot/1.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0"
95.108.213.205 - - [08/Feb/2025:09:58:05 +0100] "GET /api/scratch/vEs3d HTTP/1.1" 200 32850 "https://decomp.me/scratch/FCirn" "Mozilla/5.0 (compatible; YandexRenderResourcesBot/1.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0"
95.108.213.167 - - [08/Feb/2025:09:58:05 +0100] "POST /api/scratch/FCirn/compile HTTP/1.1" 200 6584 "https://decomp.me/scratch/FCirn" "Mozilla/5.0 (compatible; YandexRenderResourcesBot/1.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0"

vs

17:56:12 DEBUG Made new profile: Scaly Horse (anon)
17:56:12 DEBUG Made new profile: Nosy Reindeer (anon)
17:56:12 DEBUG Made new profile: Dominant Viper (anon)
17:56:47 DEBUG Made new profile: Dominant Cattle (anon)
17:56:47 DEBUG Made new profile: Scaly Cattle (anon)
17:56:47 DEBUG Made new profile: Stubborn Cheetah (anon)
17:56:55 DEBUG Made new profile: Tall Parrot (anon)
17:56:58 DEBUG Made new profile: Fast Leopard (anon)
17:57:21 DEBUG Made new profile: Jealous Mantis (anon)
17:57:40 DEBUG Made new profile: Stubborn Baboon (anon)
17:58:05 DEBUG Made new profile: Deadly Manatee (anon)

I'm not sure why the directive to ignore /api is being, uh, ignored. Perhaps because the call to /api is made from the /scratch page.

Do we want to disable bots from indexing the /scratch endpoint? Or just the Yandex one? I'm not sure - I don't think i'm particularly for or against.

Ah via here, this YandexRenderResourcesBot:

Loads resources for page rendering with JavaScript.
Ignores instructions in robots.txt if the HTML page where these resources are hosted can be accessed by the Yandex robot.
The robot doesn't access the resources if the HTML pages where these resources are used are restricted in robots.txt.

[ethteck]

Do we want to disable bots from indexing the /scratch endpoint? Or just the Yandex one? I'm not sure - I don't think i'm particularly for or against.

I think I am fine with bots indexing /scratch, but we should definitely filter profile creation on all bots we're aware of

[mkst]

PR'd :)

[mkst]

If we want to clear up the 5+ mio profiles, what do we need to do?

remove all profiles where:

  WHERE 1=1
  AND profile has no owner
  AND profile has no scratches
  AND creation_date < '2025-01-01' -- keep stuff from this year?
  AND id < 5_000_000 -- maybe this is a faster check than the date?