CSIT failing perf tests for week 48 (11/21 – 11/27)
|
Viliam Luc -X (vluc - PANTHEON TECH SRO at Cisco) <vluc@...>
=====SUMMARY===== New issues - 0 Unfixed issues - 15 Fixed issues - 3
===NEW ISSUES===
===OUTSTANDING UNFIXED===
1) [H] 2n-icx: NFV density tests breaks VPP which fails to start (re-opened) rca: test: all subsequent frequency: sporadic testbed: 2n-icx
TICKET: https://jira.fd.io/browse/CSIT-1881 NOTE: host stopped responding when running 1pl-2mif-4dcr1t-vppip4 tests and all subsequent tests failed with multiple error messages.
2) [M] 3n-snr: All hwasync wireguard tests failing when trying to verify device rca: Failed to bind PCI device 0000:f4:00.0 to c4xxx on host 10.30.51.93 test: hwasync wireguard frequency: always testbed: 3n-snr
TICKET: https://jira.fd.io/browse/CSIT-1883
3) [M] 1n-aws: TRex NDR PDR ALL IP4 scale and L2 scale tests failing with 50% packet loss rca: test: ip4scale2m frequency: always testbed: 1n-aws
TICKET: https://jira.fd.io/browse/CSIT-1876 NOTE: The root cause can be shared environment in aws cloud.
4) [H] 2n-clx, 2n-zn2: all RDMA tests failing with cli_inband clear runtime command rca: 37429: rdma: fix ipv4 checksum check in rdma-input node | https://gerrit.fd.io/r/c/vpp/+/37429 test: RDMA with CX556A NIC frequency: always testbed: 2n-clx, 2n-zn2
TICKET: https://jira.fd.io/browse/CSIT-1882 FIX (not merged yet): 37720: rdma: fix for-loop initialization in scalar path | https://gerrit.fd.io/r/c/vpp/+/37720
5) [M] 3n-tsh: all VM tests failing to boot VM rca: test: all VM frequency: sporadic testbed: 3n-tsh
TICKET: https://jira.fd.io/browse/CSIT-1877 NOTE: 3n-alt testbed was fixed. 3n-tsh still failing
6) [M] 3n-snr: 25GE interface between SUT and TG/TRex goes down randomly rca: test: all frequency: sporadic testbed: 3n-snr
TICKET: https://jira.fd.io/browse/CSIT-1871 NOTE: Sometimes 'TwentyFiveGigabitEthernetec/0/0' goes down and all subsequent tests fail. NOTE: All tests passed in #50 (except hwasync)
7) [H] 2n-clx: half of the packets lost on PDR tests (re-opened) rca: test: e810Cq ip4base, ip6base frequency: sporadic testbed: 2n-clx
TICKET: https://jira.fd.io/browse/CSIT-1864
8) [M] 3n-alt, 3n-snr: testpmd tests fail with no traffic rca: test: testpmd frequency: always testbed: 3n-alt, 3n-snr
TICKET: https://jira.fd.io/browse/CSIT-1848
9) [L] 2n-dnv: sporadic 1518B tput tests failing to establish required sessions rca: test: 1518B tput frequency: sporadic testbeds: 2n-dnv
TICKET: https://jira.fd.io/browse/CSIT-1850
10)[H] 3n-icx, 3n-skx, 3n-snr: all 1518B AVF crypto tests failed with no traffic, all IMIX AVF crypto with excessive packet loss rca: test: all AVF crypto with Xxv710 frequency: sporadic testbed: 3n-skx, 3n-icx, 3n-snr
TICKET: https://jira.fd.io/browse/CSIT-1827
11)[L] all testbeds: AF-XDP - NDR tests failing from time to time rca: test: af-xdp multicore tests frequency: low testbed: 2n-clx, 2n-skx, 2n-tx2, 2n-icx
TICKET: https://jira.fd.io/browse/CSIT-1802 NOTE: This is mainly observed in iterative and coverage. It's very low frequency ~ 1 out of 100
12)[M] 3n-tsh, 3n-alt, 2n-clx testbed (Taishan, Altra, Cascade-lake): NDR tests failing from time to time. rca: tests: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected) frequency: medium testbed: 3n-tsh, 3n-alt, 2n-clx
TICKET: https://jira.fd.io/browse/CSIT-1804
13)[L] T-Rex STL runtime error rca: VPP code - X557 speed_capability set 1GE instead of 10GE test: sporadic frequency: always testbed: 2n-dnv and 3n-dnv
TODO: VPP to fix speed_capability. TICKET: https://jira.fd.io/browse/VPP-2010
14)[L] failed creating AVF interface rca: issue in Intel FVL driver test: multicore AVF frequency: sporadic testbed: all testbeds
NOTE: A long standing issue without a final permanent fix. TICKET: multicore AVF tests are failing when trying to create interface, https://jira.fd.io/browse/CSIT-1782
15)[L] Not all DET44 sessions have been established: 4128767 != 4128768 rca: unknown test: nat44det udp 4m and 16m (64k and 1m are ok) frequency: very sporadic. It failed in 1 out of 8 runs. testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-clx
TICKET: https://jira.fd.io/browse/CSIT-1795
===OUTSTANDING FIXED===
===FIXED ISSUES=== #) [H] 2n-aws, 2n-icx: All tests fail to start VPP rca: Module uio_pci_generic not found in directory /lib/modules/5.4.0-1009-aws test: all frequency: always testbed: 2n-aws
NOTE: intermitent
#) [H] 2n-clx, 2n-icx: All tests fail to initialize interface rca: '/sys/bus/pci/devices/0000:3b:00.0/virtfn0/driver/unbind': No such file or directory test: all frequency: always testbed: 2n-clx, 2n-icx
NOTE: intermitent
#) [H] 2n-clx: dpdk tests failing to unbind PCI device rca: DRV_VFIO_PCI test: all DPDK frequency: always testbed: 2n-clx
NOTE: intermitent
Best regards, Viliam Luc |
|
|