Matplotlib error when trying to run get_coherence.py

[ev-anderson]

Hi Anthony,

I'm trying to run NLLOC Coherence for a seismic catalog and am receiving the following error when running get_coherence.py :

ImportError: cannot import name 'register_cmap' from 'matplotlib.cm'

It seems to be stemming from my matplotlib version (3.10.0) and the fact that the register_cmap import is deprecated. I tried downgrading to an older version of matplotlib (3.6.0), but that seemed to break it further, throwing an error and suggesting downgrading now NumPy to version<2.

Before I go down the rabbit hole of downgrading any further dependencies in my coherence environment, I thought I would check to see if you have come across this before. Thanks again for all your help!

Best,
Evan

[alomax]

Hi Evan,

Unfortunately the Python community likes depreciating things and has no problem breaking "old" codes. I have the impression "old" is ≥ 2 years - no problem for Silicon Valley ($$$) and fast moving machine-learning, quite devastating for science which has little $$$ and old is ≥ 50 years...

So I wasted yet again another hour or so in searching, trying, searching, trying, fighting with Python, matplotlib and conda. It did finally work for some reason or good luck. I think you need:

for INSTALL_PYTHON_ENV.txt :

conda create -n nll_ssst_coh_3 python=3.11 numpy scipy matplotlib
conda activate nll_ssst_coh_3
conda install -c conda-forge obspy
conda install -c conda-forge seaborn

with the explicit seaborn install. Then try the attached get_coherence.py which uses from matplotlib import colormaps instead of from matplotlib.cm import register_cmap

get_coherence.py.txt

I am using

> python --version
Python 3.12.7

I hope this works, or at least helps...

Best regards,
Anthony

[ev-anderson]

Hi Anthony,

Reinstalling the environment in the order you suggested, including the explicit seaborn installation and using the updated get_coherence script did the trick!

It seems to be run through the code fine, but now I am confused on the inputs for run_coherence. For LOC_PATH in run_coherence.bash, the comments define this variable as "SSST file path without location file root name (from run_ssst_relocations.log);..." I initially defined it pointing to my last (3/3) SSST iteration output location directory, but was running into an issue with the get_events pickle file not populating.

After looking through get_coherence.py, I saw the naming convention for reading in hyp_files under get_events was: 'nll_hyp_root + '...grid0.loc.hyp', which lead me to believe it needed to read in the individual event hyp files based on this formatting. However, the only hyp output files within my SSST output directories are the summary outputs with formatting: {LOC_ROOT}.sum.grid0.loc.hyp.

Should LOC_PATH be pointing to the initial location directory containing the individual location event files? Or should my SSST be outputting updated individual location event files?

Thanks again for taking the time to help, I really appreciate it!

Best,
Evan

[alomax]

Hello Evan,

Thanks for confirming that the Python env installation now works.

Should LOC_PATH be pointing to the initial location directory containing the individual location event files?

No, in general it should point to the last NLL SSST iteration locations.

Or should my SSST be outputting updated individual location event files?

LOC_PATH must contain individual location event files. These can be from any NLLoc run, but is usually after SSST and usually the last SSST relocations. For your case, this should be iteration 4/3 SSST (loc_ssst_corr4).
The example run_ssst_relocations.bash usually removes event files from the intermeduate SSST relocations to save disk space (NLL-SSST eats disk space!):

	# remove NLL event files for this iteration to free disk space
	#    does not remove NLL event files for iteration ITERATION_MAX+1, needed for NLL-coherence
	#    does not remove NLL event files for initial iteration 0, may be useful for analysis
	# comment out following lines if you need NLL event files for all iterations
	if [ $((${ITERATION} > 0)) = 1 ]; then
		WORK_DIR=$(pwd)
		cd ${OUT_ROOT}
		find . -name "${PROJECT_NAME}.*.*.grid0.loc.*" -print0 | xargs -0 rm
		cd ${WORK_DIR}
	fi

You should comment out this if block if you want to run coherence on locations from an iteration before the last one.

Best regards,
Anthony

[ev-anderson]

Hi Anthony,

It seemed iteration 4/3 was not saving individual event files as I had flagged FILENAME_DEC_SEC for loc files in the initial NLLOC run. Changing the necessary lines in run_ssst_relocations and run_coherence from ${PROJECT_NAME}...grid0.loc." to ${PROJECT_NAME}....grid0.loc.*" seem to solve it and now everything seems to run smoothly...

However, the coherency output is yielding a catalog of ~56k events, while the final iteration ssst catalog contains about ~35k events. Maybe I'm misunderstanding the step from ssst to coherency, but shouldn't the input catalog into NLL-coherency essentially be the "final" ssst catalog?

I used the example values presented in the Parkfield tutorial for this initial run_coherency.bash test.

Best,
Evan

[alomax]

I suppose using FILENAME_DEC_SEC causes problems in the SSST/coherence codes since it adds a decimal point in the filename, while these codes assume a fixed number of decimal separators in decoding the components of the filenames.

Yes, the coherence catalog events should be identical to those in the final SSST catalog. Perhaps you re-ran coherence with the same output directory and already had some of the ~56k events in the output left over from previous run(s)?
Best to rename the output after any configuration changes, or delete all if running nearly the same configuration.

Anthony

[ev-anderson]

Hello Anthony,

Using FILENAME_DEC_SEC also affected run_ssst_relocations which was not properly cleaning up .hyp files after running, which lead to run_coherence outputting the initial location hyp files. NLL coherence now seems to be running smoothly.

With the machinery running properly, I have some questions concerning the coherence output locations. The output catalog from run_coherence is nearly identical to the final SSST catalog. There is some very slight shifts in locations, slight improvements to rms and major ellipsoid axis, but overall it seems largely the same. I looked at the coherence matrix images for a few of the stations in the test folder generated by run_coherence. Here is one such image:

Obviously, the coherency being 0 for these stations/channels is the likely culprit lead for the coherency catalog being largely unchanged from the final SSST locations. Do you have any suggestions or a best approach to diagnosing my incoherent stations/channels?

Thank you again for taking the time to answer my questions, I really appreciate it.

Best,
Evan

[alomax]

Hello Evan,

OK, clearly at some point I should add an FILENAME_DEC_SEC_UNDERSCORE option which uses an underscore instead of decimal point for decimal seconds. And/or (much later, or someone else) move to a more intelligent Python wrapper instead of BASH for NLL SSST and coherence...

Yes, with zero coherence for all events, the NLL-coherence stage should reproduce the input SSST locations, except perhaps for sampling error and numerical noise during the coherence relocation.
Are there any error or warning messages when running get_coherence.py?
Did get_streams.py or whatever procedure you use to retrieve and pre-process waveforms run fine?

Anthony

[ev-anderson]

There are not any errors/warnings thrown that I noticed while running get_coherence.py, but am currently doing a run that is writing screen output to file to make sure.

I did not use get_streams.py as the dataset I am working with is a local ocean-bottom micro-seismic experiment with the waveform data on disk. I made sure to follow the naming convention in the Parkfield example when trimming and isolating event waveforms, obviously adjusting for station names/channels. Would get_coherence.py get very far if it were unable to properly read in the waveforms? I did not notice any read errors when I babysat the code.

I did take a few liberties with the window size of event waveforms, setting WINDOW_START_WAVEFORM=1 and WINDOW_LENGTH=20, and SAMPLING_RATE=100.

Additionally within the coherence calculation block, I left FREQMIN=2, but changed FREQMAX=15. I left the rest of the variables the same as those in the Parkfield example, i.e.:
DISTANCE_MATRIX_SHIFT=4.0, COHERENCEMAX_DIST=5.0, COHERENCE_MIN=0.45, USER_ABS="--use_abs", NO_NORMALIZE_PS=, SNR_ACCEPT_Trace=, etc.

Out of curiosity, is it possible to use all three channels with get_coherence.py?

Best,
Evan

[ev-anderson]

Hi Anthony,

Just a follow-up, I think I figured it out my issue. I missed a few lines in run_coherence.bash and get_coherence.py where I still had not accounted for my use of FILENAME_DEC_SEC with waveform root filename. Seems to be properly reading in the waveform data now.

Thanks again,
Evan

[alomax]

Hi Evan,

Good to hear, lets see what works or not next...

I did take a few liberties with the window size of event waveforms, setting WINDOW_START_WAVEFORM=1 and WINDOW_LENGTH=20, and SAMPLING_RATE=100.

You should check a few waveforms in a waveform viewer or ObsPy to confirm that the window includes at least 5-10 sec before and after the S arrivals.

Out of curiosity, is it possible to use all three channels with get_coherence.py?

I think so, they would be the same as additional vertical channels. The coherence for an event pair is just the maximum coherence between the events for all tested channels. I only use vertical because the P arrival is often small or in the noise on horizontal channels, so there may be less sensitivity to S-P time. But I have not formally investigated using horizontals.

Anthony

[ev-anderson]

Ok, have another thing to troubleshoot! I am now getting an error that says:

./run_coherence.bash: line 245: 2227026 Killed ${BASE_GET_COHERENCE} --target_nslc ${TARGET_NSLCS} --num_cores ${NUM_CORES} --trace_roots "${FDSN_DATASELECT_OUTPATH}/....._" --nll_trans "${NLL_TRANS}"

After which several "BrokenPipeError: [Errno 32] Broken pipe" pop up with python errors seeming to stem from multiprocessing.

Initially I suspected it was the allocation of cores for parallelization within the get coherences block. It looks like NUM_CORES is defined by the number of stations to process within run_coherence.bash, and in my case I'm attempting to process 36 stations. However I am working on a 64 core cluster, so then I thought that shouldn't be a problem?

Next, I tried forcing the run_coherence.py to avoid parallelizing but ran into the same error. I also tried lowering MAX_COUNT_OTHER=100 to see if maybe I was trying to process too many events, but ran into the same error.

[alomax]

This looks like a Python parallel error I used to get until I started using Python 3.8(?) and higher (I am now using Python 3.12).
What version of Python are you using?
My fix for earlier versions of Python is activated by changing the line:
if [ ALL = ALL ]; then # get coherence for all channels in parallel
to something like:
if [ XXX = ALL ]; then # get coherence for all channels in parallel
in order to disable the parallel run.

Does this make sense and perhaps help?

Anthony

[ev-anderson]

Hi Anthony,

I'm using python 3.11. I had an inconsistent naming convention carried over from the initial locations, so I think that was my issue. NLL-coherence seems to be running fine now, and properly reading in the waveform traces. Though I noticed for a lot of my events the following exception was raised:

"Warning: error normalizing trace on P/S, reject trace: zero-size array to reduction operation maximum which has no identity"

I searched through the src code to see where the "zero-size" exception was being raised but could not find it in any of the py scripts. I thought of turning "off" the normalization flag for traces as a resolution to this, but the dataset were recorded in a rather noisy environment so my gut tells me that would not be ideal.

Best,
Evan

[alomax]

Hi Evan,

Good that you solved one problem!

The "zero-size" exception is a Python exception which is thrown in this block in get_coherence.py:

							# normalize trace based on max abs in windows around P and S
							tr_data = tr.data
							# normalization trace is: 1/Pmax up to P, 1/Smax after S, linear continuity from P to S
							half_width = int(0.5 + tr.stats.sampling_rate * (s_time - p_time) / 3.0)
							# peak around P
							ndx_P = int(0.5 + (p_time - start_time) * tr.stats.sampling_rate)
							# print 'DEBUG:', 'start_time', start_time, 'p_time', p_time, 'tr.stats.sampling_rate', tr.stats.sampling_rate, 'ndx_P:', ndx_P, 'half_width', half_width
							tr_max_P_data = np.absolute(
								tr_data[max(0, ndx_P - half_width): min(len(tr_data), ndx_P + half_width)])
							tr_max_P = tr_max_P_data.max()
							# peak around S
							ndx_S = int(0.5 + (s_time - start_time) * tr.stats.sampling_rate)
							# print 'DEBUG:', 'start_time', start_time, 's_time', s_time, 'tr.stats.sampling_rate', tr.stats.sampling_rate, 'half_width', half_width
							tr_max_S_data = np.absolute(
								tr_data[max(0, ndx_S - half_width): min(len(tr_data), ndx_S + half_width)])
							tr_max_S = tr_max_S_data.max()
							# print 'DEBUG:', 'ndx_S:', ndx_S, 'tr_max:', tr_max_P, tr_max_S, streamfile
							norm_trace = np.zeros_like(tr_data)
							norm_trace[:ndx_P] = [1.0 / tr_max_P] * ndx_P
							norm_trace[ndx_P:ndx_S] = np.linspace(1.0 / tr_max_P, 1.0 / tr_max_S, num=(ndx_S - ndx_P))
							norm_trace[ndx_S:] = [1.0 / tr_max_S] * (len(norm_trace) - ndx_S)
							normalized_trace = np.multiply(norm_trace, tr_data)
							# tr.data = normalized_trace.astype(np.float32)
							tr.data = normalized_trace
						except Exception as ex:
							print('Warning: error normalizing trace on P/S, reject trace: ', ex)
							# traceback.print_exc()
							print(' ', streamfile)
							continue

It may be that one of the constructed arrays or slices is zero length, maybe if you uncomment the # print 'DEBUG:' ... lines you can find where there are problematic array indices?

Best regards,
Anthony

[ev-anderson]

Hi Anthony,

Making progress slowly but surely. I think my previous issue was SHIFT_LEN was too large, and printing the DEBUG helped to verify that.

Have a new one for you now! When running NLL locations within run_coherence, I'm getting several NLLOC errors for some events e.g.:

NLLoc: ERROR: opening input oct tree file
NLLoc: ERROR: reading NLLoc search posterior pdf
NLLoc: ERROR: opening Fmamp output file
NLLoc: FATAL ERROR: opening hypocenter summary files

When checking the location directory, for the events that throw this error the oct tree files do exist (as well as .hyp, .scat, etc.) I thought it might be a path issue, but again, other events are not raising this error and are relocating.

Best,
Evan