Device in a unusable state

[Josverl]

version 1.24.8 does leave an attached CircuitPython device (a QTPy-Knob running CircuitPython v9.2.3) in an unusable state, though. I have to disconnect and reconnect the device if I run mpflash

Originally posted by @scruss in #18 (comment)

[Josverl]

Mpflash does try to stop any running scripts though, so perhaps it should reset a device to return it to a running state.

[scruss]

mpflash 1.24.8 still leaves this device unusable:

                   Connected boards                    
┏━━━━━━━┳━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃Ser.   ┃Fam.┃Board                           ┃Version┃
┡━━━━━━━╇━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
│ttyACM0│cpy │UNKNOWN_BOARD                   │v9.2.3 │
│       │    │Adafruit QT Py M0 with samd21e18│       │

[Josverl]

I got myself in a bit of a tangle with the 1.25.x versions.
In order to better ID boards and variants I've changed to use a SQLite db, and that failed in the initial release so I yanked that from pypi.

I'll publish a pre-release that includes the fix.

[Josverl]

mpflash v1.25.0.rc1 prerelease is on PyPI.
The fix for this issue is included.

Please share your findings.

[scruss]

I use pipx, so I can't see it yet

[Josverl]

Perhaps using pipx install mpflash --pip-args="--preview" ?

[scruss]

pipx install mpflash --pip-args="--pre" seemed to do the job.

But:

$ mpflash
13:54:06 | ℹ️ Migrating JSONL data /home/scruss/Downloads/firmware/firmware.jsonl
to SQLite database.
Traceback (most recent call last):
  File "/home/scruss/.local/bin/mpflash", line 8, in <module>
    sys.exit(mpflash())
             ~~~~~~~^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpflash/cli_main.py", line 17, in mpflash
    migrate_database(boards=True, firmwares=True)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpflash/db/core.py", line 37, in migrate_database
    load_jsonl_to_db(jsonl_file)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpflash/db/loader.py", line 99, in load_jsonl_to_db
    custom=record["custom"],
           ~~~~~~^^^^^^^^^^
KeyError: 'custom'

I thought that something may have been broken in Downloads/firmware so I deleted it. That was a bad idea.

$ rm -rf ~/Downloads/firmware
$ mpflash
Traceback (most recent call last):
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
    self._dbapi_connection = engine.raw_connection()
                             ~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 3297, in raw_connection
    return self.pool.connect()
           ~~~~~~~~~~~~~~~~~^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/pool/base.py", line 449, in connect
    return _ConnectionFairy._checkout(self)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/pool/base.py", line 1264, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/pool/base.py", line 713, in checkout
    rec = pool._do_get()
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/pool/impl.py", line 179, in _do_get
    with util.safe_reraise():
         ~~~~~~~~~~~~~~~~~^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/util/langhelpers.py", line 224, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/pool/impl.py", line 177, in _do_get
    return self._create_connection()
           ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/pool/base.py", line 390, in _create_connection
    return _ConnectionRecord(self)
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/pool/base.py", line 675, in __init__
    self.__connect()
    ~~~~~~~~~~~~~~^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/pool/base.py", line 901, in __connect
    with util.safe_reraise():
         ~~~~~~~~~~~~~~~~~^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/util/langhelpers.py", line 224, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/pool/base.py", line 897, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
                                         ~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/engine/create.py", line 646, in connect
    return dialect.connect(*cargs, **cparams)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/engine/default.py", line 625, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)  # type: ignore[no-any-return]  # NOQA: E501
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
sqlite3.OperationalError: unable to open database file

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/scruss/.local/bin/mpflash", line 8, in <module>
    sys.exit(mpflash())
             ~~~~~~~^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpflash/cli_main.py", line 17, in mpflash
    migrate_database(boards=True, firmwares=True)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpflash/db/core.py", line 29, in migrate_database
    create_database()
    ~~~~~~~~~~~~~~~^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpflash/db/core.py", line 56, in create_database
    Base.metadata.create_all(engine)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/sql/schema.py", line 5924, in create_all
    bind._run_ddl_visitor(
    ~~~~~~~~~~~~~~~~~~~~~^
        ddl.SchemaGenerator, self, checkfirst=checkfirst, tables=tables
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 3247, in _run_ddl_visitor
    with self.begin() as conn:
         ~~~~~~~~~~^^
  File "/usr/lib/python3.13/contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 3237, in begin
    with self.connect() as conn:
         ~~~~~~~~~~~~^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 3273, in connect
    return self._connection_cls(self)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 147, in __init__
    Connection._handle_dbapi_exception_noconnection(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        err, dialect, engine
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 2436, in _handle_dbapi_exception_noconnection
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
    self._dbapi_connection = engine.raw_connection()
                             ~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 3297, in raw_connection
    return self.pool.connect()
           ~~~~~~~~~~~~~~~~~^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/pool/base.py", line 449, in connect
    return _ConnectionFairy._checkout(self)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/pool/base.py", line 1264, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/pool/base.py", line 713, in checkout
    rec = pool._do_get()
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/pool/impl.py", line 179, in _do_get
    with util.safe_reraise():
         ~~~~~~~~~~~~~~~~~^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/util/langhelpers.py", line 224, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/pool/impl.py", line 177, in _do_get
    return self._create_connection()
           ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/pool/base.py", line 390, in _create_connection
    return _ConnectionRecord(self)
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/pool/base.py", line 675, in __init__
    self.__connect()
    ~~~~~~~~~~~~~~^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/pool/base.py", line 901, in __connect
    with util.safe_reraise():
         ~~~~~~~~~~~~~~~~~^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/util/langhelpers.py", line 224, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/pool/base.py", line 897, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
                                         ~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/engine/create.py", line 646, in connect
    return dialect.connect(*cargs, **cparams)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/sqlalchemy/engine/default.py", line 625, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)  # type: ignore[no-any-return]  # NOQA: E501
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
(Background on this error at: https://sqlalche.me/e/20/e3q8)

Seems to be a rather wordy file not found error.

[scruss]

Uninstalled, made an empty ~/Downloads/firmware folder, reinstalled, and … almost successful! It still leaves the CircuitPython board in an unusable state:

$ mpflash list
14:00:23 | ℹ️ Update boards from CSV to SQLite database.
14:00:23 | ℹ️ Reading CSV data...
14:00:24 | ℹ️ Loaded 2515 boards from 
/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpflash/db/m
icropython_boards.zip
Error: Failed to get mcu_info for /dev/ttyUSB0
Error: Failed to get mcu_info for /dev/ttyUSB1

                   Connected boards                    
┏━━━━━━━┳━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃Ser.   ┃Fam.┃Board                           ┃Version┃
┡━━━━━━━╇━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
│ttyACM0│cpy │UNKNOWN_BOARD                   │v9.2.3 │
│       │    │Adafruit QT Py M0 with samd21e18│       │
│ttyUSB0│?   │                                │       │
│ttyUSB1│?   │                                │       │
└───────┴────┴────────────────────────────────┴───────┘
14:00:52 | ℹ️ b''
14:00:52 | ⚠️ Traceback (most recent call last):

14:00:52 | ⚠️   File "<frozen runpy>", line 198, in _run_module_as_main

14:00:52 | ⚠️   File "<frozen runpy>", line 88, in _run_code

14:00:52 | ⚠️   File 
"/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpremote/__
main__.py", line 6, in <module>

14:00:52 | ⚠️     sys.exit(main.main())

14:00:52 | ⚠️              ~~~~~~~~~^^

14:00:52 | ⚠️   File 
"/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpremote/ma
in.py", line 569, in main

14:00:52 | ⚠️     handler_func(state, args)

14:00:52 | ⚠️     ~~~~~~~~~~~~^^^^^^^^^^^^^

14:00:52 | ⚠️   File 
"/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpremote/co
mmands.py", line 448, in do_exec

14:00:52 | ⚠️     _do_execbuffer(state, args.expr[0], args.follow)

14:00:52 | ⚠️     ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

14:00:52 | ⚠️   File 
"/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpremote/co
mmands.py", line 431, in _do_execbuffer

14:00:52 | ⚠️     state.ensure_raw_repl()

14:00:52 | ⚠️     ~~~~~~~~~~~~~~~~~~~~~^^

14:00:52 | ⚠️   File 
"/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpremote/ma
in.py", line 511, in ensure_raw_repl

14:00:52 | ⚠️     self.transport.enter_raw_repl(soft_reset=soft_reset)

14:00:52 | ⚠️     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^

14:00:52 | ⚠️   File 
"/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpremote/tr
ansport_serial.py", line 174, in enter_raw_repl

14:00:52 | ⚠️     raise TransportError("could not enter raw repl")

14:00:52 | ⚠️ mpremote.transport.TransportError: could not enter raw repl

14:01:02 | ℹ️ b''
14:01:02 | ⚠️ Traceback (most recent call last):

14:01:02 | ⚠️   File "<frozen runpy>", line 198, in _run_module_as_main

14:01:02 | ⚠️   File "<frozen runpy>", line 88, in _run_code

14:01:02 | ⚠️   File 
"/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpremote/__
main__.py", line 6, in <module>

14:01:02 | ⚠️     sys.exit(main.main())

14:01:02 | ⚠️              ~~~~~~~~~^^

14:01:02 | ⚠️   File 
"/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpremote/ma
in.py", line 569, in main

14:01:02 | ⚠️     handler_func(state, args)

14:01:02 | ⚠️     ~~~~~~~~~~~~^^^^^^^^^^^^^

14:01:02 | ⚠️   File 
"/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpremote/co
mmands.py", line 448, in do_exec

14:01:02 | ⚠️     _do_execbuffer(state, args.expr[0], args.follow)

14:01:02 | ⚠️     ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

14:01:02 | ⚠️   File 
"/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpremote/co
mmands.py", line 431, in _do_execbuffer

14:01:02 | ⚠️     state.ensure_raw_repl()

14:01:02 | ⚠️     ~~~~~~~~~~~~~~~~~~~~~^^

14:01:02 | ⚠️   File 
"/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpremote/ma
in.py", line 511, in ensure_raw_repl

14:01:02 | ⚠️     self.transport.enter_raw_repl(soft_reset=soft_reset)

14:01:02 | ⚠️     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^

14:01:02 | ⚠️   File 
"/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpremote/tr
ansport_serial.py", line 159, in enter_raw_repl

14:01:02 | ⚠️     raise TransportError("could not enter raw repl")

14:01:02 | ⚠️ mpremote.transport.TransportError: could not enter raw repl

14:01:12 | ℹ️ b''
14:01:12 | ⚠️ Traceback (most recent call last):

14:01:12 | ⚠️   File "<frozen runpy>", line 198, in _run_module_as_main

14:01:12 | ⚠️   File "<frozen runpy>", line 88, in _run_code

14:01:12 | ⚠️   File 
"/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpremote/__
main__.py", line 6, in <module>

14:01:12 | ⚠️     sys.exit(main.main())

14:01:12 | ⚠️              ~~~~~~~~~^^

14:01:12 | ⚠️   File 
"/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpremote/ma
in.py", line 569, in main

14:01:12 | ⚠️     handler_func(state, args)

14:01:12 | ⚠️     ~~~~~~~~~~~~^^^^^^^^^^^^^

14:01:12 | ⚠️   File 
"/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpremote/co
mmands.py", line 448, in do_exec

14:01:12 | ⚠️     _do_execbuffer(state, args.expr[0], args.follow)

14:01:12 | ⚠️     ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

14:01:12 | ⚠️   File 
"/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpremote/co
mmands.py", line 431, in _do_execbuffer

14:01:12 | ⚠️     state.ensure_raw_repl()

14:01:12 | ⚠️     ~~~~~~~~~~~~~~~~~~~~~^^

14:01:12 | ⚠️   File 
"/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpremote/ma
in.py", line 511, in ensure_raw_repl

14:01:12 | ⚠️     self.transport.enter_raw_repl(soft_reset=soft_reset)

14:01:12 | ⚠️     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^

14:01:12 | ⚠️   File 
"/home/scruss/.local/pipx/venvs/mpflash/lib/python3.13/site-packages/mpremote/tr
ansport_serial.py", line 159, in enter_raw_repl

14:01:12 | ⚠️     raise TransportError("could not enter raw repl")

14:01:12 | ⚠️ mpremote.transport.TransportError: could not enter raw repl

mpremote finds the board in raw repl. Exiting with Ctrl-B then Ctrl-D gives me:

Adafruit CircuitPython 9.2.3 on 2025-01-17; Adafruit QT Py M0 with samd21e18
>>> 
>>> 
>>> 
raw REPL; CTRL-B to exit
>

Adafruit CircuitPython 9.2.3 on 2025-01-17; Adafruit QT Py M0 with samd21e18
>>> 
soft reboot

Auto-reload is off.
Running in safe mode! Not running saved code.

You are in safe mode because:
CircuitPython core code crashed hard. Whoops!
Hard fault: memory access or instruction error.
Please file an issue with your program at github.com/adafruit/circuitpython/issues.
Press reset to exit safe mode.

Press any key to enter the REPL. Use CTRL-D to reload.

In order to get the board back to working, I have to enter this at the REPL:

import microcontroller
microcontroller.reset()

[Josverl]

Thanks for sharing the logs

• the crash was during establishing the database, and migration of the existing info on downloaded firmwares.
  • The first keyerror was likely an older record in the .jsonl file that did not have the 'custom' field.
  • the cause 2nd error is unclear, but the errors are handles better now.
• for the reset, I assumed that the board could be reset by mpremote reset, but that requires it to run MicroPython 😬.
  I have added a special case for circuitpython, using the reset sequence you provided.

Note that I have not been able to reproduce the mpremote cannot enter raw repl error.
in my test I used a PICO2W with CP 9.2.8 with a simple blink app in code.py

published: v1.25.0.rc2