CSIT failing perf tests for week 38 (09/04 – 09/11)
Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
1) error: 3n-icx: NDR tests failing with ~1700 packets lost
test: IP4 tunnels with E810Xxv nic
2) error: 2n-clx, 2n-zn2, 2n-icx: QEMU NF failed to run on vppl2xc
rca: svm_region_map(mmap open): No such file or directory
test: MRR: VM VHOST vppl2xc
NDRPDR: VM VHOST vppip4
testbed: 2n-clx, 2n-zn2, 2n-icx
NOTE: 2n-zn2 started with build #603 on 31st of August. Build #602 on 30th of August passed.
3) error: 2n-clx: X710 NICs interfere with TRex
rca: i40e interface 0000:18:00.0 is under Linux and will interfere with TRex interface 0000:18:00.2
test: X710 (ip4base, ip6base, l2bd)
4) error: 2n-tx2, 3n-tsh: Failed to create container DUT1_CNF1
test: 2n-tx2: all Container Memif
testbed: 2n-tx2, 3n-tsh
NOTE: 2n-tx2 started with build #383 on 31st of August. Build #382 on 30th of August passed.
NOTE: 3n-tsh started with build #674 on 31st of August. Build #673 on 30th of August passed.
5) error: 3n-icx: All 1000Tnlsw Fixtnlip non AVF tests failing. 1518B with no traffic forwarded, IMIX with excessive packet loss
test: 1518B crypto
6) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions
test: 1518B tput
7) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss
test: all AVF crypto
testbed: 3n-skx, 3n-icx
8) error: NDR sporadic packet lost
test: af-xdp multicore tests
testbed: 2n-skx, 2n-clx
9) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time.
tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)
testbed: 3n-tsh, 3n-alt, 2n-clx
10) error: T-Rex STL runtime error
rca: VPP code - X557 speed_capability set 1GE instead of 10GE
testbed: 2n-dnv and 3n-dnv
TODO: VPP to fix speed_capability.
11) error: failed creating AVF interface
rca: issue in Intel FVL driver
test: multicore AVF
testbed: all testbeds
NOTE: A long standing issue without a final permanent fix.
TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782
12) error: Not all DET44 sessions have been established: 4128767 != 4128768
test: nat44det udp 4m and 16m (64k and 1m are ok)
frequency: very sporadic. It failed in 1 out of 8 runs.
testbed: 2n-skx, 2n-icx, 2n-clx
13) error: 2n-clx, 2n-zn2: Failed to create AVF interfaces on 710 NICs
test: AVF 710
testbed: 2n-clx, 2n-zn2
NOTE: 2n-clx X710-AVF tests failing with issue #3
NOTED: FIXED on 09th of September 2n-clx build #1182
14) error: 2n-clx: half of the packets lost on PDR tests
test: e810Cq ip4base, ip6base