Database locked

[forstater]

Problem: When using GUI to analyze a very long data set (time series written out), gui stalled. Error suggests problem writing to database. This error occurred after ~100k events written out. (total size of db was ~500MB). To my knowledge, user was not manually updating the BD histogram when this error occurred.
Commit Version: 53e7a14

Notes Is it possible this problem is due to the processing thread and GUI trying to access the database at the same time? As the number of events gets large the query time for the BD histogram increases.

Console output:

Traceback (most recent call last):
  File "/Users/hnw5/Projects/mosaic/mosaicgui/fiteventsview/fiteventsview.py", line 254, in OnEventIndexLineEditChange
    self.eventIndex=int(self.eventIndexLineEdit.text())
ValueError: invalid literal for int() with base 10: ''

Traceback (most recent call last):
  File "/Users/hnw5/Projects/mosaic/mosaicgui/fiteventsview/fiteventsview.py", line 254, in OnEventIndexLineEditChange
    self.eventIndex=int(self.eventIndexLineEdit.text())
ValueError: invalid literal for int() with base 10: ''
database is locked
database is locked

Process Process-1:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/hnw5/Projects/mosaic/mosaic/apps/SingleChannelAnalysis.py", line 42, in run_eventpartition
    EventPartition.PartitionEvents()
  File "/Users/hnw5/Projects/mosaic/mosaic/partition/metaEventPartition.py", line 198, in PartitionEvents
    self._writeanalysisinfo()
  File "/Users/hnw5/Projects/mosaic/mosaic/partition/metaEventPartition.py", line 296, in _writeanalysisinfo
    self.FsHz
  File "/Users/hnw5/Projects/mosaic/mosaic/mdio/sqlite3MDIO.py", line 170, in writeAnalysisInfo
    self.db.execute( 'INSERT INTO analysisinfo VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)',  (infolist+[mosaic.__version__, mosaic.__build__, None]))
OperationalError: database is locked
^C^CTraceback (most recent call last):
  File "/Users/hnw5/Projects/mosaic/mosaicgui/statisticsview/statisticsview.py", line 251, in OnAppIdle
    def OnAppIdle(self):
KeyboardInterrupt

[forstater]

Update: Reran analysis using gui but turned off BDHistogram window. Error occured after ~95 minutes of processing. Although the GUI can't access the database the processing thread is still continues to write out to the database. The statistics window in the gui also slowly updates.

[abalijepalli]

This condition is not handled among the different GUI windows, so when a query runs long for one of them others may throw an exception. The analysis handles this and therefore will continue to run. There may be a way to gracefully handle this in the UI. However, querying large databases during a running analysis will slow down the analysis as it will lock out the processing thread from writing new events. It is therefore preferable to run long extremely long analysis from the command line.

[forstater]

I understand - thanks for the explanation. One point of clarification – this error occurred in two different ways for the user:

(a) In the console output I’ve shown, the databased stopped being written to.
(b) When the BD histogram panel was turned off in the GUI – the software kept writing to the db, as you mentioned below – the analysis handled it.

Is it possible for the GUI to lock out the processing thread almost permanently?

A second point – I think if we want to have a GUI driven workflow accessible for users, it would be good to have some way of preventing this problem other than directing them to use the command line. I think there will be a significant population of researchers who will give up using mosaic for those data sets rather than swapping to the command line. One solution to this later problem, which we talked about before, is to create a simple gui for setting up (batch) analysis or another way of making the command line use less intimidating.

Would it be possible to enable a feature in the GUI to allow the user to setup the analysis and then hand it off to the processing thread and terminate the gui?

[abalijepalli]

I tried to reproduce this error, but haven't been able to. The test I ran wrote > 200k records to a DB while the BD window simultaneously queried the same database. The GUI actually does handle transaction locks on the database properly. The error is displayed in the BD window as seen below.
It disappears on the next update, as expected. There were no exceptions thrown by either thread. Is it possible the error you are seeing is data set specific? Could you try a different data set to see if it can be replicated?

The simplest way to use the GUI to run an analysis would be to start it but close all the plotting windows. Right now, these windows will still update in the background. However, it is easy to change GUI to prevent background updates. In that case the only thread accessing the DB would be the analysis thread. I'll push changes to implement that behavior.

[forstater]

Let me speak with Haiyan and find out what data sets she’s observed this with.