Re: CSIT failing perf tests for week 39 (09/12 – 09/18)
Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
We’ve got issue with all ARM testbeds failing to start VPP. Assigned the highest priority. ARM is aware and are working on fix.
1) error: ALL ARM testbeds are failing with VPP failed to start
testbed: 2n-tx2, 3n-alt, 3n-tsh
From: Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
1) error: 3n-icx: all NFV density-DCR memif-Chain ipsec tests failing with no traffic forwarded
test: all chain ipsec
2) error: 2n-clx: half of the packets lost on PDR tests (re-opened)
test: e810Cq ip4base, ip6base
NOTE: happened again but only on ip6base e810Cq - build #158.
3) error: 2n-icx: all tests failed with parent suite setup time-out
rca: SSHTimeout: Timeout exception during execution of command: fgrep docker /proc/1/cgroup
4) error: 3n-icx: NDR tests failing with ~1700 packets lost
test: IP4 tunnels with E810Xxv nic
NOTE: TRex doesn't support E810Xxv (Columbiaville).
5) error: 2n-clx, 2n-zn2, 2n-icx: QEMU NF failed to run on vppl2xc
rca: svm_region_map(mmap open): No such file or directory
test: MRR: VM VHOST vppl2xc
NDRPDR: VM VHOST vppip4
testbed: 2n-clx, 2n-zn2, 2n-icx
NOTE: 2n-zn2 started with build #603 on 31st of August. Build #602 on 30th of August passed.
6) error: 2n-clx: X710 NICs interfere with TRex
rca: i40e interface 0000:18:00.0 is under Linux and will interfere with TRex interface 0000:18:00.2
test: X710 (ip4base, ip6base, l2bd)
7) error: 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1
test: 2n-tx2: all Container Memif
testbed: 2n-tx2, 3n-tsh
NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.
NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.
8) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss
test: 1518B crypto
9) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions
test: 1518B tput
NOTE: #1240 all tput test passed
10) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss
test: all AVF crypto
testbed: 3n-skx, 3n-icx
11) error: NDR sporadic packet lost
test: af-xdp multicore tests
testbed: 2n-skx, 2n-clx
12) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.
tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)
testbed: 3n-tsh, 3n-alt, 2n-clx
13) error: T-Rex STL runtime error
rca: VPP code - X557 speed_capability set 1GE instead of 10GE
testbed: 2n-dnv and 3n-dnv
TODO: VPP to fix speed_capability.
14) error: failed creating AVF interface
rca: issue in Intel FVL driver
test: multicore AVF
testbed: all testbeds
NOTE: A long standing issue without a final permanent fix.
TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782
15) error: Not all DET44 sessions have been established: 4128767 != 4128768
test: nat44det udp 4m and 16m (64k and 1m are ok)
frequency: very sporadic. It failed in 1 out of 8 runs.
testbed: 2n-skx, 2n-icx, 2n-clx