Attributes writing multiple places

[zhanzhang3]

I am running an Eiger2 CdTe 500k detector and trying to record a list of attributes.
I put a test layout xml file "Eiger2_28idb_layout_test.xml" into "s28eiger-cdte:HDF1:XMLFileName" field.

Eiger2_28idb_Layout_test.xml

And use "attributes_28idb_Eiger2_test.xml" in the "s28eiger-cdte:cam1:NDAttributesFile" field.

attributes_28idb_Eiger2_test.xml

All the attributes in the test file are kind of static information, so I only need to record them once. If I set them in the layout file to be , they will be recorded once but allocated 8k times the space if I have 8k images, like the one attribute info here cut from the output of an "h5ls -rvl h5file":

=====
/entry/instrument/detector/configuration/counting_mode Dataset {1/Inf}
Attribute: NDAttrDescription scalar
Type: 14-byte null-terminated ASCII string
Attribute: NDAttrName scalar
Type: 13-byte null-terminated ASCII string
Attribute: NDAttrSource scalar
Type: 36-byte null-terminated ASCII string
Attribute: NDAttrSourceType scalar
Type: 20-byte null-terminated ASCII string
Location: 1:280476
Links: 1
Chunks: {8000} 2048000 bytes
Storage: 256 logical bytes, 2048000 allocated bytes, 0.01% utilization
Type: 256-byte null-terminated ASCII string

I set them to be hoping they are only to be recorded as attributes of my detector 'group'. They did get recorded that way:

However, they are also recorded as if they are , which are not specified in the layout file.

Is this a correct behavior? That balloons the file quite fast and, I believe, slows the detector down when I add a lot more attributes.

A few screen shots of one test I did with an exposure of 8k-image with 0.0005 seconds each (with actual layout and attribute files I want to use):

The exposure took more like 15 seconds to finish (~500 Hz instead of 2k) and about half of the images are lost with queue of 1000. setting the queue to 6000 caught all but takes a while to finish saving.

During this, I turned off all other plugins, left only the HDF1 running.

Thanks!

[MarkRivers]

The issue with the extra size seems to be that the NDArray attributes get written to datasets with a chunk size set to the number of NDArrays in the file. In my example this is from ADSpinnaker and the number of arrays is 1840. This is correct for attributes that are written for each NDArray, like TimeStamp:

/defaults/NDArrayTimeStamp Dataset {1840/Inf}
    Attribute: NDAttrDescription scalar
        Type:      40-byte null-terminated ASCII string
    Attribute: NDAttrName scalar
        Type:      17-byte null-terminated ASCII string
    Attribute: NDAttrSource scalar
        Type:      7-byte null-terminated ASCII string
    Attribute: NDAttrSourceType scalar
        Type:      19-byte null-terminated ASCII string
    Location:  1:37116
    Links:     1
    Chunks:    {1840} 14720 bytes
    Storage:   14720 logical bytes, 14720 allocated bytes, 100.00% utilization
    Type:      native double

Note that is has 1000% utilization, which is fine.

But for attributes that are only written once on file open or file close, then the chunking is not appropriate and results in a lot of wasted space.

/measurement/instrument/attenuator/setup/user_filter_name Dataset {1/Inf}
    Attribute: NDAttrDescription scalar
        Type:      12-byte null-terminated ASCII string
    Attribute: NDAttrName scalar
        Type:      7-byte null-terminated ASCII string
    Attribute: NDAttrSource scalar
        Type:      24-byte null-terminated ASCII string
    Attribute: NDAttrSourceType scalar
        Type:      20-byte null-terminated ASCII string
    Location:  1:86572
    Links:     1
    Chunks:    {1840} 471040 bytes
    Storage:   256 logical bytes, 471040 allocated bytes, 0.05% utilization
    Type:      256-byte null-terminated ASCII string

[MarkRivers]

The issue with the speed of writing the HDF5 file looks strange. In your HDF5 file plugin the Execution Time shown in the upper left is 3.132 ms. However, the ArrayRate is 0, and the plugin appears to be still writing to the file. The Array Rate should not be 0, it should be about 300.

Does the Array Rate vary up and down?

I see that you are writing to a network file system. Have you measured the performance of this file system with a simple file copy? Have you tried measuring the performance when saving to a local drive, preferably an SSD?

[zhanzhang3]

Ah, I screen shot the image when the writing had already finished. It's still waiting for more images as some are lost. During the actual saving, the rate is around 300, fairly stable. Given the detector image is pretty much all zero now, typically each image is about 10 KB, plus all other overhead of about 30 KB, we're at about 12 MB/s. A couple quick test showed me the writing speed of this network drive is about 100MB/s.

hancock% time rsync -av --progress /local/user28idc/test_1_xpcs_S196.h5 ./testtrans.h5
sending incremental file list
test_1_xpcs_S196.h5
4,211,896,084 100% 481.39MB/s 0:00:08 (xfr#1, to-chk=0/1)

sent 4,212,924,487 bytes received 35 bytes 99,127,635.81 bytes/sec
total size is 4,211,896,084 speedup is 1.00
12.075u 4.007s 0:41.74 38.5% 0+0k 0+8226360io 0pf+0w

If I got rid of attribute writing, I got image saving rate @ 4000, fully keeping up with the image taking speed.

[MarkRivers]

When you use attributes and it is writing a file please run the following command in another shell on the same machine running the Eiger IOC:

top -H

That will show which threads are using the most CPU. I want to see if the HDF5 plugin is CPU bound. Post a screenshot here.

[zhanzhang3]

Here is one snapshot. I assume it fully exhaust one CPU?

[MarkRivers]

This is interesting. You actually have 2 threads that are using 100% of a CPU.

• eigerStreamTask is the task that copies the arrays from the ZMQ stream to ADEiger. It is slowing things down and may be in danger of dropping arrays.
• FileHDF1_Plugin is the HDF5 file writer. It is CPU bound, which is limiting the frame rate to disk.

What CPUs does the machine running ADEiger have? You can get the information from lscpu. This is one of our higher performance Linux machines:

[epics@countach-linux ~]$ lscpu
Architecture:                x86_64
  CPU op-mode(s):            32-bit, 64-bit
  Address sizes:             52 bits physical, 57 bits virtual
  Byte Order:                Little Endian
CPU(s):                      20
  On-line CPU(s) list:       0-19
Vendor ID:                   GenuineIntel
  Model name:                Intel(R) Xeon(R) w5-2445
    CPU family:              6
    Model:                   143
    Thread(s) per core:      2
    Core(s) per socket:      10
    Socket(s):               1
    Stepping:                8
    CPU(s) scaling MHz:      50%
    CPU max MHz:             4600.0000
    CPU min MHz:             800.0000
    BogoMIPS:                6192.00

Since the configuration you show is simply copying the arrays to disk with no other plugins, you could consider using the FileWriter interface instead. It could write only to the RAM drive in the Eiger server, or directly to your network disk. You should try those to compare the performance.

[zhanzhang3]

Yes, I did have dropped arrays in that test of 10k x 0.00025 second exposure. The buffer free dropped to 0 and about 2k images lost. The file writing took some time (with queue size at 6k) but finished without dropping more frames.
Does the Filewriter write the attributes? I assume it won't if I turn off the FileHDF1 plugin. There is no issue if I stop writing attributes with FileHDF1 plugin.

This is what we have for the Eiger2-CdTe computer:

s28eiger-cdte% lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Gold 6444Y
CPU family: 6
Model: 143
Thread(s) per core: 2
Core(s) per socket: 16
Socket(s): 1
Stepping: 8
CPU(s) scaling MHz: 100%
CPU max MHz: 4000.0000
CPU min MHz: 800.0000
BogoMIPS: 7200.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtsc
p lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_
cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_l
m abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vp
id ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt av
x512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect user_shstk av
x_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hfi vnmi avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx
512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr i
bt amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
Virtualization features:
Virtualization: VT-x
Caches (sum of all):
L1d: 768 KiB (16 instances)
L1i: 512 KiB (16 instances)
L2: 32 MiB (16 instances)
L3: 45 MiB (1 instance)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-31
Vulnerabilities:
Gather data sampling: Not affected
Indirect target selection: Not affected
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Reg file data sampling: Not affected
Retbleed: Not affected
Spec rstack overflow: Not affected
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Srbds: Not affected
Tsa: Not affected
Tsx async abort: Not affected
Vmscape: Mitigation; IBPB before exit to userspace
s28eiger-cdte%

[MarkRivers]

It appears your CPU is similar to mine.
There are 2 places where frames can be dropped:

• The ZMQ Stream interface from Eiger server to the IOC
• In the HDF5 plugin. You can prevent this by increasing the QueueSize up to the limit of RAM in your machine.

It appears that each attribute increases the CPU load. You said it works fine when there are no NDAttributes where it can run at the full 4000 frames/s. But it you look at those HDF5 files you should still see a few attributes like timestamps and unique ID. It would be interesting to see if the CPU load and HDF5 array rate scale linearly with the number of attributes.

[zhanzhang3]

Did a quick test, starting from default setup, i.e., no attribute file defined. By the time I reached about 300 attributes, both threads had the CPU overloaded. But the buffer/Queue seems handling it ok, until about 400 attributes need to be recorded.