Excessive FTT warnings

[genevb]

Attn: @jdbrice

In preparing for a physics production with FTT enabled, I see FTT warnings that are generating excessive output of these two specific lines of text:

StFttClusterMaker:WARN - StFttClusterMaker::PassTimeCut - Unknown hit time mode from database: -1717986919
StFttClusterMaker:WARN - Accepting all hits

I expect this to have two impacts:

1. The log files will be bloated. By my measurement, after the first event, 98% of the output is from those two lines. That implies a factor of x50 increase in log files from these messages. The repetition of the same message means it isn't as bad after gzip compression, but it's still a notable x4-x5 increase in gzipped logs.

2. There will be message manager memory consumption due to the messages. I have yet to quantify this (I will follow up), but this will inevitably slow down jobs as they progress and could be particularly problematic for long jobs, which could be terminated for exceeding memory limits on processing nodes, as well as even running past the queue's duration limits. All of which are impactful to our efficient use of the farm.

To reproduce:

cp /star/rcf/test/daq/2022/010/23010027/st_physics_23010027_raw_1000015.daq .
root4star -b -q -l 'bfc.C(5,"pp2022a,StiCA,fst,ftt,ImpBToFt0Mode","st_physics_23010027_raw_1000015.daq")'

I do not think I can start a large production without addressing this.

-Gene

[genevb]

I forgot to mention in my original post that this was using 64-bit optimized SL25b on NFSv4, but this should also be visible in the current DEV as it is, for now, identical to SL25b.

---

From processing 1000 events with and without FTT, these would be the rough conclusions:

1. FTT costs ~5 seconds/event, or perhaps ~15% slower.
2. Without the FTT, processing slows by ~1.5 seconds/event, or perhaps ~5% slow-down over the course of 1000 events.
3. With the FTT, processing slows by ~3 seconds/event, or perhaps ~10% slow-down over the course of 1000 events.

Note of course that I have not actually isolated the impact of the messages from FTT, but rather the impact of FTT as a whole. The observed impact is not insignificant, but the vast majority of files in the Run 22 pp500 st_physics production have a little under 3000 events, implying slow-downs of very roughly ~30% over the course of processing a file (vs. ~15% even without FTT), so this issue isn't a complete show-stopper for production.

[genevb]

A critical point that I hadn't yet brought up here is whether these messages are telling us something important (outside of the impact of the messages themselves) that is relevant for a production using FTT. Are we missing something from the database? I see essentially no content in the offline databases Calibrations_ftt and Geometry_ftt, and there is no relevant "ftt" content under StarDb. So....are we missing something important??
@jdbrice , this is really a question to you!