Closed Figure's state is silently changed in HTML export when plotting asynchronously

[mmatthe]

Describe the issue

I know, this is a seemingly contrived example, but it's a real-world problem for me.

See attached Notebook and according HTML export
issue.zip

I'm running the notebook cell by cell, where I only start the next cell when the previous animation has been finished. The animation stops automatically. The code for animation is:

def runPlot(duration):
    plt.close('all')
    fig = plt.figure(1)
    t = np.linspace(0, 1, 100)
    data_line = plt.plot(t, np.sin(6.28*2*t))[0]
    
    def _impl():
        start = time.time()
        it = 0
        while time.time() - start < duration:
            freq = time.time() - start
            if it == 0:
                data_line.set_ydata(t)
            else:
                data_line.set_ydata(np.sin(6.28*t*freq))
            it = it + 1
                
            fig.canvas.draw()   
            time.sleep(0.03)
        
        
    T = threading.Thread(target=_impl)
    T.start()

Within this code, a thread is started which shows a figure and continuously updates the ydata of the plotted line. The cell immediately returns, but the thread runs for the given amount of seconds. This leads to an animation in the notebook. Note that in the first iteration, a linear line is plotted instead of the sine wave. After running all cells, the notebook looks like this:

Upon pressing save and exporting to HTML, the exported HTML looks like this:

Apparently, for the HTML export, the state of the first plot widget has been replaced by the data that was first set by set_ydata. Adding more calls to runPlot only keeps the last figure in the correct shape for export, whereas the other figures contain the data initially set by set_ydata.

The problem does not happen when I create a new figure without a number all the time, i.e. replace this

    plt.close('all')
    fig = plt.figure(1)

with

fig = plt.figure()

However, as all figures are closed by close('all'), no data should be changed in previously open figures anymore.

Versions

$ python -c "import sys; print('\n',sys.version); import ipympl; print('ipympl version:', ipympl.__version__)" && jupyter --version && jupyter nbextension list && jupyter labextension list

 3.13.1 (main, Dec 19 2024, 10:35:14) [GCC 11.1.0]
ipympl version: 0.9.5
Selected Jupyter core packages...
IPython          : 8.30.0
ipykernel        : 6.29.5
ipywidgets       : 8.1.5
jupyter_client   : 8.6.3
jupyter_core     : 5.7.2
jupyter_server   : 2.14.2
jupyterlab       : 4.3.4
nbclient         : 0.10.2
nbconvert        : 7.16.4
nbformat         : 5.10.4
notebook         : 7.3.1
qtconsole        : not installed
traitlets        : 5.14.3

[mmatthe]

The reason why I can't just discard the close('all') call is, that I run the cells multiple times for testing and hence every time a new figure is created. This eventually leads to matplotlib complaining that too many figures are open.

[ianhi]

Hi @mmatthe thank you for the report, this is definitely a bug! Unfortunately it's a bug in the section of code that's been most difficult to get workign properly over the years: saving a version of the plot into notebook.

The threading is not necessary to reproduce this. You can reproduce with the following:

#cell one
%matplotlib ipympl

import matplotlib.pyplot as plt
import numpy as np

plt.figure()
plt.plot(x,x)

Wait for that cell to finish then run cell 2:

plt.plot(x, x+np.sin(x))

After saving, closing and reopening the notebook the plot will not only display the straight line, not both lines. In fact any modification of the plot that happens after the initial display will not be saved properly.

The issue is that the mimebundle that gets saved into the notebook is only ever updated when the figure is first displayed which is done here:

ipympl/ipympl/backend_nbagg.py

Lines 384 to 385 in fa3d346

	data = self._repr_mimebundle_(**kwargs)
	display(data, raw=True)

I'm not sure if there is an easy way to update the mimebundle once displayed. Though I suspect that @martinRenou might know as the person who got us to this state where anything is saved (used to be that nothing was saved).

The reason why I can't just discard the close('all') call is, that I run the cells multiple times for testing and hence every time a new figure is created. This eventually leads to matplotlib complaining that too many figures are open.

The close is not responsible for this issue (sadly otherwise this is an easy fix). But as a quality of life thing you can change the limit for this warning. Here just set it to never warn:

import matplotlib.pyplot as plt
plt.rcParams.update({'figure.max_open_warning': 0})

from: https://stackoverflow.com/a/50122156/835607

[ianhi]

Ahhh I should have looked harder. This is a known issue. See the discussion (from 2021) here: #359 (comment)

I wonder if #376 might enable a fix for this, though my understanding of this stuff is a bit hazy now after 4 years.

[ianhi]

@mmatthe do you have this option checked?

when I have that option on then then update the plot twice then I get to see the first update, but not the second!

which seems to strongly suggest that an await for the offscreen canvas to have finished updating is necessary here:

ipympl/src/mpl_widget.ts

Line 253 in fa3d346

this.set('_data_url', this.offscreen_canvas.toDataURL());

and then the animations should be saved properly

[ianhi]

Note for potential fix:

(A note for future me unless someone else wants to implement this - im supposed to be thesis writing right now)

I think we either need to wait on the image.onLoad function finishing before we set the _data_url OR move the this.set('_data_url', this.offscreen_canvas.toDataURL()); needs to be moved to the onload function after this line:

ipympl/src/mpl_widget.ts

Line 337 in fa3d346

this.offscreen_context.drawImage(this.image, 0, 0);

apparently setting image.src as done here:

ipympl/src/mpl_widget.ts

Line 251 in fa3d346

this.image.src = image_url;

is async which is why the data_url is being updated too quickly.

---

I think @mmatthe is seeing only the first image because of not having the save widget state option checked as true, in that case there is no obvious way I see to update the mimebundle that was created when the image was first displayed

[mmatthe]

Hi @ianhi,

thanks for the analysis! In fact, I have the option "Save Widget State Automatically" checked to "on". Otherwise, it would only show the straight line of the first call to set_ydata. The things you mention might be related to the issue, however the strange thing is that subsequent calls to set_data are recorded and exported to HTML, but only in figures which have not been closed before. If a figure is closed, subsequent calls to set_ydata seem to be dropped.

Also, I believe that threading is necessary to raise this problem, because it seems to be that the execution of the cell itself needs to be finished but afterwards the ydata needs to be changed.

Anyway, I have found a workaround - I close the figures all the time for development, but for export I just don't close them. Thi switch is done with a flag at the top of the notebook. Then the export is fine and development is not impeded by growing RAM requirements of too many open figures.