Begin forwarded message:

 

From: Maciek Konstantynowicz <mkonstan@...>

Subject: number of csit-vpp jobs failing on vpp master branch - RCA wip

Date: 6 March 2018 at 14:54:39 GMT

To: csit-dev@...

Cc: Peter Mikus <pmikus@...>

 

RCA Update from Peter Mikus on two issues behind the failing csit-vpp jobs due to recent VPP patches:-

Side note - honeycomb jobs are also affected.

 

1. errors during VPP install due to new vpp-lib dependencies - observed from last Friday

 
Root cause
----------
 
VPP patch [1] introduce new external package dependencies (libmbedcrypto0 libmbedtls10 libmbedx509-0). Without these packages VPP service is not starting results in following log messages:
      vpp[30760]: /usr/bin/vpp: error while loading shared libraries: libmbedtls.so.10: cannot open shared object file: No such file or directory
 
This was not caught by vpp-csit-verify-virl-master job due to the bug in evaluating the robot framework results from multiple parallel simulation (see full log in [2]). Root cause is error in writing into /run/shm/rc_test_run${N} where CSIT bootstrap.sh is storing the results of partial runs. This results into faulty evaluation of total number of passed/failed test cases.
 
There are several steps needed to fix this issue:
#1 - Fix bootstrap to proper evaluate the results https://gerrit.fd.io/r/#/c/10981/
#2 – Install the missing dependencies in VIRL Ubuntu image .. TODO (patch https://gerrit.fd.io/r/#/c/10980/)
-   Rebuild of Ubuntu image needed
#3 – Install the missing dependencies in VIRL Centos image .. TODO (patch https://gerrit.fd.io/r/#/c/10980/)
-   Rebuild of Ubuntu image needed
#4 – Install the missing dependencies on PhyTB .. DONE (patch https://gerrit.fd.io/r/#/c/10980/)
 
[1] https://gerrit.fd.io/r/10768
[2] https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-csit-verify-virl-master/9755/console.log.gz
 
 
2. startup.conf broken - observed from last Friday
    impacting all csit-vpp jobs and hc-vpp jobs
        wasn't detected in vpp-csit-verify
 
Root cause
----------
 
This issue is caused by api-segment { gid vpp } to be not recognized in default startup.conf installed by VPP, resulting into VPP service not started. Root cause to be confirmed. Work in progress. Need to fix the vpp-lib dependencies first.