BMP: 'proc_queue' gets full during power-on commands

[lplana]

Events scheduled on 'proc_queue' are dispatched from a while loop in main. They can be interrupted by messages arriving and commands being executed.

A 'power-on' command takes a long time to execute. During this time, many events can be scheduled. In particular, SysTick schedules 'proc_100hz' every 10 ms. In this situation, 'proc_queue' gets full and no further events can be scheduled. I've tested with a proc_queue length of 256 events and it gets full.

Delaying or missing 'proc_100hz' is usually not a big problem. 'uptime' will drift and A/D, temperature and fan measurements will be delayed, usually without consequences. Delaying or missing other events may have consequences more difficult to deal with.

This is a difficult problem to solve.

[rowleya]

Do we have a number for how long a power command takes which we could then use to say how big the queue would need to be to not overflow (which might be an impossible number to have of course)? That said, are there likely to be multiple power commands which might then mean multiples of this number would be required?

[lplana]

I don't have a measurement of how long a power command takes.

The scenario is a board receiving and executing a power-on command. It's not the board on slot 0 sending power-on commands to other boards, which could be many in quick succession. It's unlikely that several power-on commands to the same board will be executed back-to-back but I suppose it cannot be ruled out completely.

The current size of proc_queue is 16 and I increased it to 256, and the queue did get full anyway. I could try larger sizes to establish the length of a power-on command.

I don't think that increasing proc_queue size is the way to go. Accumulating 100s or 1000s of events in the queue will delay them significantly, which may have consequences.

[rowleya]

Hmm, yes, if the delay is going to cause other issues, that wouldn't help I guess. Without knowing the BMP code, is it possible to split up the receiving and executing of a power command in any way e.g. the reception just schedules it and then the execution is maybe split in to several sequences somehow e.g. sending the power on message, then separately waiting for the power to come on before sending a response (if that actually happens)?

[lplana]

I don't have a good handle on the complete power-on sequence. I suppose that should be the first step.

One of the time-consuming tasks to be completed during power-on is the downloading of the 3 FPGA bit files. These are ~1.5MB long each and are downloaded bit-serially by the bmp.

There are several 'busy-wait' delays (e.g,, to wait for the output of the power modules to stabilize) which could be implemented differently, but I'm not sure that we can do all of them sensibly.

[rowleya]

I guess even splitting up the 3 FPGA writes into 3 separate steps might then help. These could then be further broken down if this still causes an issue and assuming that it is possible to partially write a file.