Re: CSIT failing perf tests for week 31 (07/25 – 07/31)
Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
1) PM to apply workaround until Fan & Intel team fix the issue. Issue happened after change in compatibility matrix in CSIT 22.06 env space.
2) Frequency is now higher. VP to fix.
3) No progress since last week. VP to fix.
4, 5, 6, 7, 8 – long term issues with no permanent fix.
From: Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco)
1) error: 2n-clx, 3n-skx: First port is down when running traffic profile
testbed: 2n-clx, 3n-skx
NOTE: Issue had been observed on 2n-clx and fixed with workaround. Now (7/3) it appeared on 3n-skx. The same x710 NIC.
2) error: 2n-dnv: sporadic 1518B tput tests failing to establish required sessions
test: 1518B tput
3) error: 3n-icx, 3n-skx: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss
test: all AVF crypto
testbed: NDRPDR: 3n-skx
NOTE: After a while failed on MRR daily tests. Only 1518B tests failed, IMIX passed.
NOTE: MRR 3n-icx failing always (not sporadic)
4) error: 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra): NDR tests failing from time to time.
tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)
testbed: 3n-tsh, 3n-alt, 2n-clx
5) error: NDR sporadic packet lost
test: af-xdp multicore tests
testbed: 2n-skx, 2n-clx
NOTE: Not observed this week.
6) error: T-Rex STL runtime error
rca: VPP code - X557 speed_capability set 1GE instead of 10GE
testbed: 2n-dnv and 3n-dnv
TODO: VPP to fix speed_capability.
7) error: failed creating AVF interface
rca: issue in Intel FVL driver
test: multicore AVF
testbed: all testbeds
NOTE: A long standing issue without a final permanent fix.
TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782
8) error: Not all DET44 sessions have been established: 4128767 != 4128768
test: nat44det udp 4m and 16m (64k and 1m are ok)
frequency: very sporadic. It failed in 1 out of 8 runs.
testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-dnv
NOTE: During week28 1st time happened on 2n-icx
NOTE: During week29 1st time observed on 2n-dnv
9) 3n-alt: failed to set port1 up
NOTE: only 1 test failing, but it's failing in each run since Wednesday (07/19)
10) error: 2n-skx, 2n-clx, 2n-icx: ALL 1518B TCP tput tests failing with big packet loss
test: 1518B TCP tput
testbed: 2n-skx, 2n-clx, 2n-icx
11) 2n-skx: UDP 16m tput tests fail to create all sessions
test: UDP 16m TPUT
NOTE: not the same issue as https://jira.fd.io/browse/CSIT-1795 where only 1 session not created.
NOTE: This is not observed on daily trending.