Crash - ValueError: state value is not 0|1 ?

[miggland]

My Inventree server crashed today, while not in active use.

I have tried restarting the containers, but no luck yet.

This is the log from when the first crash occurred:

[2022-08-29 09:20:01 +0200] [12] [CRITICAL] WORKER TIMEOUT (pid:1425)
[2022-08-29 09:20:03 +0200] [12] [WARNING] Worker with pid 1425 was terminated due to signal 9
[2022-08-29 09:20:10 +0200] [1455] [INFO] Booting worker with pid: 1455
[2022-08-29 09:20:39 +0200] [12] [CRITICAL] WORKER TIMEOUT (pid:1455)
[2022-08-29 09:20:39 +0200] [1455] [INFO] Worker exiting (pid: 1455)
[2022-08-29 09:20:40 +0200] [1456] [INFO] Booting worker with pid: 1456
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
[2022-08-29 09:21:10 +0200] [12] [CRITICAL] WORKER TIMEOUT (pid:1456)
[2022-08-29 09:21:10 +0200] [1456] [INFO] Worker exiting (pid: 1456)
[2022-08-29 09:21:10 +0200] [1485] [INFO] Booting worker with pid: 1485
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
[2022-08-29 09:21:15 +0200] [1485] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python3.9/site-packages/gunicorn/util.py", line 359, in import_app
    mod = importlib.import_module(module)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/inventree/InvenTree/InvenTree/wsgi.py", line 15, in <module>
    application = get_wsgi_application()  # pragma: no cover
  File "/root/.local/lib/python3.9/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
    django.setup(set_prefix=False)
  File "/root/.local/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/root/.local/lib/python3.9/site-packages/django/apps/registry.py", line 122, in populate
    app_config.ready()
  File "/home/inventree/InvenTree/plugin/apps.py", line 47, in ready
    registry.load_plugins()
  File "/home/inventree/InvenTree/plugin/registry.py", line 100, in load_plugins
    _maintenance = bool(get_maintenance_mode())
  File "/root/.local/lib/python3.9/site-packages/maintenance_mode/core.py", line 64, in get_maintenance_mode
    return backend.get_value()
  File "/root/.local/lib/python3.9/site-packages/maintenance_mode/backends.py", line 43, in get_value
    return self.from_str_to_bool_value(statefile.read())
  File "/root/.local/lib/python3.9/site-packages/maintenance_mode/backends.py", line 22, in from_str_to_bool_value
    raise ValueError("state value is not 0|1")
ValueError: state value is not 0|1

Now, the server container keeps restarting, with the same error (ValueError) occurring constantly. Any ideas?

[SchrodingersGat]

Can you paste the contents of maintenance_mode_state.txt (in your media directory)?

[miggland]

These are the files:

-rw-r--r--  1 root root               0 Aug 29 09:21 maintenance_mode_state.txt
-rw-r--r--  1 root root               1 Aug 16 17:56 maintenance_mode_state_0QK6cF7.txt
-rw-r--r--  1 root root               1 Aug 17 17:03 maintenance_mode_state_3kWIIwL.txt
-rw-r--r--  1 root root               1 Aug 16 17:56 maintenance_mode_state_3uDPnSN.txt
-rw-r--r--  1 root root               1 Aug 15 17:31 maintenance_mode_state_9A92clD.txt
-rw-r--r--  1 root root               1 Aug 15 17:49 maintenance_mode_state_CGevlv4.txt
-rw-r--r--  1 root root               1 Aug 17 16:52 maintenance_mode_state_DN5BgnI.txt
-rw-r--r--  1 root root               1 Aug 16 21:59 maintenance_mode_state_Ewz5Vuh.txt
-rw-r--r--  1 root root               1 Aug 17 16:52 maintenance_mode_state_Gkz3n14.txt
-rw-r--r--  1 root root               1 Aug 17 16:52 maintenance_mode_state_J9pIC2j.txt
-rw-r--r--  1 root root               1 Aug 17 16:13 maintenance_mode_state_LVxi9jW.txt
-rw-r--r--  1 root root               1 Aug 16 21:59 maintenance_mode_state_LtVzWyC.txt
-rw-r--r--  1 root root               1 Aug 17 16:13 maintenance_mode_state_MSI4N7I.txt
-rw-r--r--  1 root root               1 Aug 16 21:54 maintenance_mode_state_MvrW8j9.txt
-rw-r--r--  1 root root               1 Aug 15 17:49 maintenance_mode_state_NSnf6Fc.txt
-rw-r--r--  1 root root               1 Aug 17 16:13 maintenance_mode_state_NftMfcd.txt
-rw-r--r--  1 root root               1 Aug 17 16:13 maintenance_mode_state_SYadNlB.txt
-rw-r--r--  1 root root               1 Aug 16 21:59 maintenance_mode_state_SbGIwfP.txt
-rw-r--r--  1 root root               1 Aug 16 21:59 maintenance_mode_state_TUIo1me.txt
-rw-r--r--  1 root root               1 Aug 16 21:59 maintenance_mode_state_VMVqBpH.txt
-rw-r--r--  1 root root               1 Aug 17 17:03 maintenance_mode_state_VqaGv3l.txt
-rw-r--r--  1 root root               1 Aug 16 21:59 maintenance_mode_state_VqvrDrj.txt
-rw-r--r--  1 root root               1 Aug 16 21:59 maintenance_mode_state_WEjAttr.txt
-rw-r--r--  1 root root               1 Aug 17 17:36 maintenance_mode_state_WUsCAoI.txt
-rw-r--r--  1 root root               1 Aug 17 17:03 maintenance_mode_state_X4hq5vC.txt
-rw-r--r--  1 root root               1 Aug 16 17:56 maintenance_mode_state_Zf6q5HN.txt
-rw-r--r--  1 root root               1 Aug 15 17:49 maintenance_mode_state_cOcgt4y.txt
-rw-r--r--  1 root root               1 Aug 16 21:54 maintenance_mode_state_cmcVGw4.txt
-rw-r--r--  1 root root               1 Aug 17 17:03 maintenance_mode_state_dV66IEj.txt
-rw-r--r--  1 root root               1 Aug 16 21:59 maintenance_mode_state_dfHdKck.txt
-rw-r--r--  1 root root               1 Aug 15 17:49 maintenance_mode_state_e2NLFU9.txt
-rw-r--r--  1 root root               1 Aug 16 21:59 maintenance_mode_state_eIStats.txt
-rw-r--r--  1 root root               1 Aug 15 17:31 maintenance_mode_state_f9tp2cm.txt
-rw-r--r--  1 root root               1 Aug 16 21:59 maintenance_mode_state_fE0GsVR.txt
-rw-r--r--  1 root root               1 Aug 16 21:54 maintenance_mode_state_hq9DGR3.txt
-rw-r--r--  1 root root               1 Aug 16 21:54 maintenance_mode_state_iAtyYEz.txt
-rw-r--r--  1 root root               1 Aug 15 17:31 maintenance_mode_state_jSPGhif.txt
-rw-r--r--  1 root root               1 Aug 16 21:54 maintenance_mode_state_kBBpttp.txt
-rw-r--r--  1 root root               1 Aug 17 16:13 maintenance_mode_state_lVMcqQP.txt
-rw-r--r--  1 root root               1 Aug 17 17:03 maintenance_mode_state_q0HkjQO.txt
-rw-r--r--  1 root root               1 Aug 16 21:59 maintenance_mode_state_sePYP1F.txt
-rw-r--r--  1 root root               1 Aug 17 16:13 maintenance_mode_state_svuBS9H.txt
-rw-r--r--  1 root root               1 Aug 17 17:03 maintenance_mode_state_uPOC9EV.txt
-rw-r--r--  1 root root               1 Aug 16 21:54 maintenance_mode_state_wASCQmc.txt
-rw-r--r--  1 root root               1 Aug 17 16:13 maintenance_mode_state_za89FCF.txt

Hadn't realised those were there. maintenance_mode_state.txtis empty (see also filesize). The others have 0 or 1:

cat maintenance_mode_state*
10010100000001001101000000001010011101101101

[miggland]

After shutting down, removing all those files, then restarting, the server runs without complaining.

[miggland]

I decided to upgrade to the newest version. After doing that, these files exist:

-rw-r--r--  1 root root                1 Aug 29 17:12 maintenance_mode_state.txt
-rw-r--r--  1 root root                1 Aug 29 16:53 maintenance_mode_state_11HmSfp.txt
-rw-r--r--  1 root root                1 Aug 29 16:53 maintenance_mode_state_MewoH68.txt
-rw-r--r--  1 root root                1 Aug 29 16:53 maintenance_mode_state_Qwtm693.txt
-rw-r--r--  1 root root                1 Aug 29 16:53 maintenance_mode_state_SuLpLCC.txt
-rw-r--r--  1 root root                1 Aug 29 16:53 maintenance_mode_state_dMKgXk5.txt
-rw-r--r--  1 root root                1 Aug 29 16:53 maintenance_mode_state_pDNZrNU.txt
-rw-r--r--  1 root root                1 Aug 29 16:53 maintenance_mode_state_yhEepKo.txt

Content is a mix of 0 and 1. I have not actively enabled or disabled maintenance mode, but run export-records, invoke update, and of course started the new containers.

[SchrodingersGat]

@miggland thanks for reporting - I have started an issue - #3618