Automatic tests for drivers compliancy

[hasauino]

First of all, I hope this is the appropriate location to raise such an issue. My apologies if it is not (please feel free to close the issue if that’s the case).

At the organization where I work, we use hundreds of different lab instruments from various vendors for component verification.
One recurring challenge we face is the non-compliance of instrument drivers with the IVI specifications.
Although vendors often claim their drivers are IVI compliant, we frequently encounter subtle deviations from the driver specifications. A particular concern is the lack of compliance with the IVI class specifications.

It would be highly beneficial to have a test pipeline (perhaps a GitHub Action or a similar solution) that automatically checks any driver added to the driver registry for compliance.

[joseph-mueller]

This is exactly the right place to create an issue -- thanks for using this tool.

I will ensure the IVI working group discusses your posting, we should get back to you in a few days.

[joseph-mueller]

Also, it would be helpful if you could give a couple of examples of the sort of issues you encounter (and that would implicitly indicate something about the sort of tests that would be necessary). Could you share a few examples? You do not need to name vendors, but understanding the details of the non-compliance will be helpful for us.

[hasauino]

Examples on non-compliancy:

1. Driver Attribute Types
   According to IVI specs, certain attributes are expected to adhere to specific data types. For instance, in the IVI.NET IviScope specification, the TriggerCoupling attribute must use the predefined enumeration type Ivi.Scope.TriggerCoupling. However, many vendor drivers define their own enumeration types for such attributes. This leads to compatibility issues where an error is raised if an IVI-defined enumeration value is passed, as the vendor, in some cases, does not allow for type casting or conversion. Additionally, the vendor's enumeration type often does not align with the IVI-defined one. For example:

   • In the IVI specification, the Ivi.Scope.TriggerCoupling.AC enumeration has an underlying value of 0.
   • However, the vendor-defined enumeration uses a different underlying value, leading to a mismatch.

2. Missing Attributes
   Vendor drivers may omit attributes that are required by the IVI specification, causing gaps in functionality. Examples on missing attributes:

   • Acquisition.Source
   • Acquisition.TimePerRecord
   • Acquisition.SampleMode

3. Missing Methods:

   Example Acquisition.ConfigureRecord method

Tests:

• Depending on the driver type (IVI-NET, IVI-C, IVI-COM), the driver can be dynamically loaded, and an instrument instance can be constructed using simulation mode. The test can iterate through every interface and assert that all attributes/methods exist. For every instrument class a test can be created, and the same class test shall pass for every registered driver. So it is not a functional test of the instrument but a simple test to assert the driver API only.