Error restarting tpg docker after unplugging and replugging the 122.8 MHz clock

[jlashner]

When we unplug the 122.8 MHz clock from the timing master and plug it back in (difficulty in the timing system) I've gotten this error:

cryo@smurf-srv-SO9:~/docker/tpg/v5.0.1$ docker logs -f tpg_ioc

IP address was defined. Ignoring shelfmanager and slot number.
Starting IOC...
#!../../bin/rhel6-x86_64/tpg
# Start of timing_dev/Tpg/iocBoot/sioc-smrf-ts01/st.cmd
## You may have to change tpg to something else
## everywhere it appears in this file
## tidair/smurf-tpg-ioc release version
# SMURF_TPG_IOC_RELEASE=v5.0.1
## SMuRF tpg-ioc versions packaged into this release
# LCLS2_TIMING_PATTERNS_HASH=3d2437f
# LCLS2_TIMING_PATTERNS_BRANCH=ESCRYODET-472
# TIMING_DEV_HASH=0ef6f57
# TIMING_DEV_BRANCH=dev
< envPaths
epicsEnvSet("IOC","sioc-smrf-ts01")
epicsEnvSet("TOP","/root/Tpg")
epicsEnvSet("EPICS_BASE","/")
cd /root/Tpg
## Register all support components
dbLoadDatabase("dbd/tpg.dbd")
tpg_registerRecordDeviceDriver(pdbbase)
# ====================================================================
# Setup some additional environment variables
# ====================================================================
# Setup environment variables
epicsEnvSet("ENGINEER","Kukhee Kim")
epicsEnvSet("LOCATION","LI00")
epicsEnvSet("IOC_PV", "SIOC:SMRF:TS01")
epicsEnvSet("IOC",    "sioc-smrf-ts01")
# tag log messages with IOC name
# How to escape the "ioctpg" as the PERL program
# will try to repplace it.
# So, uncomment the following and remove the backslash
#epicsEnvSet("EPICS\_IOC\_LOG_CLIENT_INET","${IOC}")
# ========================================================
# Support Large Arrays/Waveforms; Number in Bytes
# Please calculate the size of the largest waveform
# that you support in your IOC.  Do not just copy numbers
# from other apps.  This will only lead to an exhaustion
# of resources and problems with your IOC.
# The default maximum size for a channel access array is
# 16K bytes.
# ========================================================
epicsEnvSet("EPICS_CA_MAX_ARRAY_BYTES", "80000000")
# END: Additional environment variables
# ====================================================================
########################################################################
# BEGIN: Load the record databases
#######################################################################
# TPG driver DB
dbLoadRecords("db/tpg_smrf.db", "DEV=TPG:SMRF:1")
#dbLoadRecords("db/crossbarCtrl.db", "DEV=${PREFIX},PORT=crossbar")
# BSA driver DB
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=PIDU,SECN=PIDU")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=PIDL,SECN=PIDL")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=TSU,SECN=TSU")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=TSL,SECN=TSL")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=RATES,SECN=RATES")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=BEAMREQ,SECN=BEAMREQ")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=BEAMNRG01,SECN=BEAMNRG01")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=BEAMNRG23,SECN=BEAMNRG23")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=PHWAVELEN,SECN=PHWAVELEN")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=MPSSTAT,SECN=MPSSTAT")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=MPSCLASS01,SECN=MPSCLASS01")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=MPSCLASS23,SECN=MPSCLASS23")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=BSAINITL,SECN=BSAINITL")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=BSAINITU,SECN=BSAINITU")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=BSAACTL,SECN=BSAACTL")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=BSAACTU,SECN=BSAACTU")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=BSAAVGL,SECN=BSAAVGL")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=BSAAVGU,SECN=BSAAVGU")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=BSADONEL,SECN=BSADONEL")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=BSADONEU,SECN=BSADONEU")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=CTRLSEQ01,SECN=CTRLSEQ01")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=CTRLSEQ23,SECN=CTRLSEQ23")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=CTRLSEQ45,SECN=CTRLSEQ45")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=CTRLSEQ67,SECN=CTRLSEQ67")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=CTRLSEQ89,SECN=CTRLSEQ89")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=CTRLSEQAB,SECN=CTRLSEQAB")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=CTRLSEQCD,SECN=CTRLSEQCD")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=CTRLSEQEF,SECN=CTRLSEQEF")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=ACVADCA,SECN=ACVADCA")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=ACVADCB,SECN=ACVADCB")
#dbLoadRecords("db/bsa.db", "DEV=TPG:SMRF:1,PORT=bsaPort,BSAKEY=ACVADCC,SECN=ACVADCC")
#
# =====================================================================
# Load iocAdmin databases to support IOC Health and monitoring
# =====================================================================
dbLoadRecords("db/iocAdminSoft.db","IOC=SIOC:SMRF:TS01")
dbLoadRecords("db/iocAdminScanMon.db","IOC=SIOC:SMRF:TS01")
# The following database is a result of a python parser
# which looks at RELEASE_SITE and RELEASE to discover
# versions of software your IOC is referencing
# The python parser is part of iocAdmin
dbLoadRecords("db/iocRelease.db","IOC=SIOC:SMRF:TS01")
# =====================================================================
# Load database for autosave status
# =====================================================================
dbLoadRecords("db/save_restoreStatus.db", "P=SIOC:SMRF:TS01:")
# =====================================================================
#Load Additional databases:
# =====================================================================
## Load record instances
#dbLoadRecords("db/dbExample.db","user=khkimHost")
# END: Loading the record databases
########################################################################
# =================================
# Load YAML
# =================================
#cd yamlConfig_0x000C
cd yamlConfig_0x0000000E-20170420000855
#cd yamlConfig_0x03090000
cpswLoadYamlFile("000TopLevel.yaml", "NetIODev", "", "10.0.1.102")
WARNING: (YamlPreprocessor) #once line lacks a 'tag' -- in : <stream> (will use filename for now but future version escalates to ERROR)
CONNECTED to 10.0.1.102:8193
CONNECTED to 10.0.1.102:8193
Starting up 'UDP RX Handler (UDP protocol module)'
Starting up 'RSSI Thread'
C SYN received, good checksum (state CLNT_WAIT_SYN_ACK)
Starting up 'TDEST VC Demux'
Starting up 'SRP VC Demux'
Starting up ''Depacketizer' protocol module'
CONNECTED to 10.0.1.102:8194
CONNECTED to 10.0.1.102:8194
Starting up 'UDP RX Handler (UDP protocol module)'
Starting up 'RSSI Thread'
C SYN received, good checksum (state CLNT_WAIT_SYN_ACK)
Starting up ''Depacketizer' protocol module'
Starting up 'SRP VC Demux'
Starting up 'TDEST VC Demux'
# ===================================
# Load ADC configuration from YAML file
# ====================================
cd /root/Tpg
cpswLoadConfigFile("config/configDump.yaml", "mmio/AmcCarrierTimingGenerator/ApplicationCore", "")
cd /root/Tpg
# ====================================
# Setup TPG Driver
# ====================================
#tpgAsynDriverConfigure("tpgPort", "192.168.2.10")
#tpgAsynDriverConfigure("tpgPort", "10.0.3.102")
# TPG driver for yaml
tpgAsynDriverSetKind("SOUTH_POLE")
tpgAsynDriverConfigure("tpgPort")
terminate called after throwing an instance of 'BadStatusError'
  what():  SRP Read (off 0x80000001) terminated with bad status
./start_ioc.sh: line 120:     8 Aborted                 (core dumped) PREFIX=${prefix} FPGA_IP=${fpga_ip} ./st.cmd

So far its seemed to go away by power cycling and restarting a bunch of times... @swh76 do you have any idea what causes this / how to fix without power cycling?

[jlashner]

oh wait tomoki mentioned that we should deactivate / reactivate slot two using:

    ssh shm-smrf-sp01
    clia deactivate board 2
    clia activate board 2

That sounds like the right way to solve this.

[msilvafe]

Yes 122.88 MHz clock must be present when the board boots but if you have to reset restarting just the card and not the whole crate sounds correct