Using `pyxcp` as a library

[akermu]

Thank you very much, for bringing xcp to the python world!

We want to use this package as a library and we have currently two issues:

• There's a pretty harsh constraint on the dependencies:

  pyxcp/pyproject.toml

  Line 86 in 36843fd

  traitlets = "<=5.11.2"

• There's currently unconditionally pretty printing installed:

  pyxcp/pyxcp/__init__.py

  Lines 9 to 16 in 36843fd

  	pretty.install()
  	from .master import Master # noqa: F401, E402
  	from .transport import Can, Eth, SxI, Usb # noqa: F401, E402
  	console = Console()
  	tb_install(show_locals=True, max_frames=3) # Install custom exception handler.

• It is very difficult to make a PyXCP object programmatically as it expects to read from a file (can be worked around)

I would provide patches for both of these, if you're interested?

[fross]

We're also interested in using this as a library.

We haven't spend much time on it yet, but the third point you are mentioning is problematic for us too.

We are still using 0.21 since Master can be configured using a simple dictionary.

[SaxElectronics]

@akermu did you manage to use pyxcp as a library? I am having hard times initializing the drivers without the commandline tool.
I would like to have more control over what is going on under the hood to use it really as a library. All the initialization stuff is pretty "hidden". I am interested to the patches, can you paste them here?

@fross Indeed i am experiencing difficulties during the updete to 0.30. I had old versions working just fine. Now almost since a week trying to get 0.30 up and running but I am failing... I don't want to give up as I have spent now almost a week trying to make the new version work. I really like some of the updates @christoph2 has brought in....hope to find a solution soon.

Two main points I see critical currently and missing:

• Flexibility in controling protocol and layer:
  control of protocol (COMMANDS etc..) to be independent control of the transportation layer (e.g. when to initialize or shutdown the bus)
• Flexible way of initialization

[fross]

@SaxElectronics I suspect it's only a matter of updating the configuration of Master from a dictionary to a Config object.

I gave it a shot last week, but did not have the time to complete the configuration (yet). I'll come back to it eventually. The legacy module is a good starting point to dig in this.

e.g.

    from traitlets.config import Config

    [...]

    general_config = Config()

    transport_config = Config()
    transport_config.timeout = 5.0

    can_config = Config()
    can_config.bitrate = 500_000
    can_config.can_id_master = 257
    can_config.can_id_slave = 258
    can_config.channel = "0"
    can_config.daq_identifier = []
    can_config.data_bitrate = 500_000
    can_config.fd = False
    can_config.interface = "Kvaser"
    can_config.max_dlc_required = False
    can_config.use_default_listener = True

    config = Config()
    config.general = general_config
    config.transport = transport_config
    config.transport.can = can_config

    master = Master("CAN", config)

[SaxElectronics]

@SaxElectronics I suspect it's only a matter of updating the configuration of Master from a dictionary to a Config object.

I gave it a shot last week, but did not have the time to complete the configuration (yet). I'll come back to it eventually. The legacy module is a good starting point to dig in this.

e.g.

    from traitlets.config import Config

    [...]

    general_config = Config()

    transport_config = Config()
    transport_config.timeout = 5.0

    can_config = Config()
    can_config.bitrate = 500_000
    can_config.can_id_master = 257
    can_config.can_id_slave = 258
    can_config.channel = "0"
    can_config.daq_identifier = []
    can_config.data_bitrate = 500_000
    can_config.fd = False
    can_config.interface = "Kvaser"
    can_config.max_dlc_required = False
    can_config.use_default_listener = True

    config = Config()
    config.general = general_config
    config.transport = transport_config
    config.transport.can = can_config

    master = Master("CAN", config)

I tried your proposal, but seems not to work out of the box, as pyxcp seems to have some internal checks and failing to initialize:

WARNING  PyXCP:can.py:358 XCPonCAN - 'slcan' has no support for parameter 'fd'.
WARNING  PyXCP:can.py:358 XCPonCAN - 'slcan' has no support for parameter 'data_bitrate'.
WARNING  PyXCP:can.py:358 XCPonCAN - 'slcan' has no support for parameter 'receive_own_messages'.
WARNING  PyXCP:can.py:358 XCPonCAN - 'slcan' has no support for parameter 'timing'.
ERROR    XCPDriver:xcp_driver.py:283 Error during CAN bus initialization: 'LazyConfigValue' object has no attribute 'btr'

even using directly the function convert_config(), seems not like every parameter in a configuration:

WARNING  XCPDriver:xcp_driver.py:239 Starting CAN bus initialization (converted settings)
WARNING  XCPDriver:legacy.py:92 Unknown keyword 'BAUDRATE_PRESET' in config file
WARNING  XCPDriver:legacy.py:92 Unknown keyword 'BTL_CYCLES' in config file
WARNING  XCPDriver:legacy.py:92 Unknown keyword 'SAMPLE_RATE' in config file
WARNING  XCPDriver:legacy.py:92 Unknown keyword 'SAMPLE_POINT' in config file
WARNING  XCPDriver:legacy.py:92 Unknown keyword 'DAQ_IDENTIFIER' in config file

also if you look into the legacy.py, the mappings here seem to be not bullet proof and tested for every driver, I doubt that it will work for every can interface because we see there hard coded mappings to certain hardware vendors, the underlaying hardware can be very different (Vector, SlCan, Kvaser, etc...)

 "TSEG1": "Transport.Can.tseg1_abr",
    "TSEG2": "Transport.Can.tseg2_abr",
    "TTY_BAUDRATE": "Transport.Can.SlCan.ttyBaudrate",
    "UNIQUE_HARDWARE_ID": "Transport.Can.Ixxat.unique_hardware_id",
    "RX_FIFO_SIZE": "Transport.Can.Ixxat.rx_fifo_size",
    "TX_FIFO_SIZE": "Transport.Can.Ixxat.tx_fifo_size",
    "DRIVER_MODE": "Transport.Can.Kvaser.driver_mode",
    "NO_SAMP": "Transport.Can.Kvaser.no_samp",
    "SINGLE_HANDLE": "Transport.Can.Kvaser.single_handle",
    "USE_SYSTEM_TIMESTAMP": "Transport.Can.Neovi.use_system_timestamp",
    "OVERRIDE_LIBRARY_NAME": "Transport.Can.Neovi.override_library_name",
    "BAUDRATE": "Transport.Can.Serial.baudrate",
    "SLEEP_AFTER_OPEN": "Transport.Can.SlCan.sleep_after_open",

for example, this code:

can_config_hard_coded = {
            "TRANSPORT": "CAN",
            "ALIGNMENT": 8,
            "TIMEOUT": 3.0,
            "CREATE_DAQ_TIMESTAMPS": False,
            "DISCONNECT_RESPONSE_OPTIONAL": False,
            "LOGLEVEL": "DEBUG",
            "CAN_DRIVER": "slcan",
            "CAN_USE_DEFAULT_LISTENER": True,
            "CHANNEL": "COM10",
            "BITRATE": 500000,
            "CAN_ID_MASTER": 0x03,
            "CAN_ID_SLAVE": 0x04,
            "CAN_ID_BROADCAST": 0xF4,
            "MAX_DLC_REQUIRED": False,
            "DAQ_IDENTIFIER": [0x5, 0x6, 0x7],
        }

        try:
            # 1. Convert legacy dict to nested Config
            converted_cfg = convert_config(can_config_hard_coded, logger=self.logger)

            self.logger.debug("Converted legacy config to traitlets Config successfully")

            # 2. Create XCP Master
            self.master = Master("CAN", config=converted_cfg)
            self.logger.warning("Created XCP Master with converted config")

leads to
ERROR XCPDriver:xcp_driver.py:276 Error during CAN bus initialization: 'LazyConfigValue' object has no attribute 'disable_error_handling'

The error message is not very informative, and seems to be even somewhere inside the traits library. 😫

[fross]

The device I'm using has a dedicated port for XCP communication. It is not a problem if pyxcp manages the bus since it is not shared. Note that the code of my previous message is not working. I basically created an empty configuration and gave it a shot. I added missing parameters based on errors returned by pxycp and/or python-can. I still need to work on it, but it's not a priority at the moment. I'm still on 0.21. I"m confident I'll figure it out though.

[SaxElectronics]

Sure it is possible and I am confident that "I'll figure it out, too"... at some point. The problem is the time spent on it.
I thought in 1-2 days the update will be over, now almost a week on it. This is a bit frustrating.

Update: It is not that simple, once you try to directly pass a traits configuration you get ton's of errors. It seems that "create_application()" function parses many things and fills up missing configurations (for example filling up certain parameters with None's in order for Master to accept it). Even all the AI out there is not that helpfull. 😉

If @akermu really made it work, I would really appreciate to apply directly a working solution. 🙏

[SaxElectronics]

Update: finally I made it work! 🥳🥳🥳
I am pasting here full working configuration for slcan without using "create_application" for anyone interested.

from pyxcp.config import PyXCP, General, Transport

def create_direct_pyxcp_can(
    channel: str         = "COM10",
    bitrate: int         = 500_000,
    can_id_master: int   = 0x03,
    can_id_slave: int    = 0x04,
    can_id_broadcast: int= 0xF4,
    daq_identifier: list = (0x5, 0x6, 0x7),
    timeout: float       = 3.0,
    alignment: int       = 8,
    disconnect_response_optional: bool = False,
    create_daq_timestamps: bool        = False,
) -> PyXCP:
    """
    Return a PyXCP app pre-configured for CAN-SLCAN.
    You can then do:
        app = create_direct_pyxcp_can(...)
        master = Master(app.transport.layer, config=app)
    """
    app = PyXCP()
    # replicate what start() would do internally
    app.general   = General(config=app.config, parent=app)
    app.transport = Transport(parent=app)

    # General settings
    app.general.disconnect_response_optional = disconnect_response_optional

    # Transport settings
    app.transport.layer             = "CAN"
    app.transport.timeout           = timeout
    app.transport.alignment         = alignment
    app.transport.create_daq_timestamps = create_daq_timestamps

    # CAN-interface settings
    cancfg = app.transport.can
    cancfg.interface       = "slcan"
    cancfg.channel         = channel
    cancfg.bitrate         = bitrate
    cancfg.can_id_master   = can_id_master
    cancfg.can_id_slave    = can_id_slave
    cancfg.can_id_broadcast= can_id_broadcast
    cancfg.daq_identifier  = list(daq_identifier)

    return app

Usage:

  # use defaults
    app = create_direct_pyxcp_can()
    master = Master(app.transport.layer, config=app)

[akermu]

@SaxElectronics nice 👍 Your solution is actually cleaner, I just use a dummy config file:

        empty_config_file = NamedTemporaryFile(delete=False, suffix=".py")
        empty_config_file.close()

        config = PyXCP()
        config.config_file = empty_config_file.name
        config._read_configuration(config.config_file)

[SaxElectronics]

Hi guys,

I have a small but important update for you, perhaps usefull for some of you. I have been testing my scripts with direct configuration and also observed an inconcistency while executing in different pyxcp versions. I managed to track this down to this very latest change causing it:

This is a key difference, if you are working without a context managet you need to call transport.connect() before calling master.connect()! Otherwise your code will fail with an error that the transport layer instance instance is not available:

INFO     Connecting...
ERROR    Test failed: 'PythonCanWrapper' object has no attribute 'can_interface'
INFO     Test complete

@christoph2 introduced this fix to enable multiple connecting and disconnecting from the slave in the very same session (example: connect -> disconnect -> connect).

So correct order instantiate (currently) a connection would be:

master.transport.connect()
logger.info("Connected successfully to TRANSPORT LAYER")
response = master.connect()
logger.info(f"Connection successful to SLAVE! Response: {response}")

I think properly defined APIs in the master class with clear error handling would be very helpful for everyone.
As the normal user (who is not that deep in to the library) has difficulties understanding the error messages.

1. API for connecting and disconnecting to the BUS
2. API for connecting and disconnecting to the slave: 1) is (of course) required for 2)

proper error handling can inform the user what he did wrong, for example in case he tries to connect to the slave without a transport layer connection established first.

[SaxElectronics]

@fross @akermu if you are interested in the examples there are added here in my fork:
https://github.com/SaxElectronics/pyxcp/tree/master/pyxcp/examples

I can try to bring them in the upstream/master if @christoph2 wants them.