Topics

[vpp-dev] VPP Performance Trending - the Dashboard is out


Ray Kinsella <mdr@...>
 

W00t!

Great work - another big step in trace-ability.

Ray K

On 29/04/2018 22:08, Maciek Konstantynowicz (mkonstan) wrote:
Hi,
Per updates on VPP and CSIT project calls, few of us in CSIT worked on
getting the VPP performance trending into a usable shape. We got there
in the end, initial (experimental!) version is out. Thanks to Peter,
Tibor and Vratko for great work, I was just complaining :)
The results are published as follows:
*1. VPP Performance Dashboard [1]* - provides the latest VPP throughput
trend, trend compliance and detected anomalies, all on a per VPP test
case basis. Test cases are clickable, leading to associated trendline
graph. Regressions are listed first (red), followed by progressions (green),
outliers (grey) and normal.
*2. Trending Methodology [2] *- describes used performance test metrics
and calculations for performance trend and anomaly classification
(progress, regression, outlier).
*3. Trendline Graphs [3]* - show graphically sequence of measurements
against the trendline, and anomaly classification, with associated CSIT
test jobs and VPP builds.
Note this is still in experimental phase, so don’t be alarmed if you see
your VPP code packet path being listed in red - it could always be a
false positive :) Same for code paths listed in green..
All feedback welcome, especially re anomaly detection accuracy,
usability and clarity of presentation and drill-down. We know there is a
lot that could be done here to improve, so constructive feedback is
golden :)
Below also a list of few caveats.
-Maciek
[1] https://docs.fd.io/csit/master/trending/introduction/index.html
[2] https://docs.fd.io/csit/master/trending/methodology/index.html
[3] https://docs.fd.io/csit/master/trending/trending/index.html
Caveats:
1. Update frequency - all dynamic content is to be updated every ~12hrs,
   but right now it’s every other day due to LF testbeds occupied with
   rls1804 testing work, back to daily next week.
2. Following test cases are currently not present in trendline graphs:
      (listing only 1t1c variants, 2t2c, 4t4c variants are also affected)
      10ge2p1x520-imix-1t1c-dot1q-l2bdbasemaclrn-eth-2vhostvr1024-1vm-mrr
      10ge2p1x520-64b-1t1c-eth-l2xcbase-eth-4vhostvr1024-2vm-ndrdisc-mrr
      10ge2p1x520-imix-1t1c-eth-l2xcbase-eth-4vhostvr1024-2vm-ndrdisc-mrr
      10ge2p1x520-64b-1t1c-eth-l2bdbasemaclrn-eth-4vhostvr1024-2vm-mrr
      10ge2p1x520-imix-1t1c-eth-l2bdbasemaclrn-eth-4vhostvr1024-2vm-mrr
      10ge2p1x710-64b-1t1c-eth-l2xcbase-eth-4vhostvr1024-2vm-ndrdisc-mrr
      10ge2p1x710-imix-1t1c-eth-l2xcbase-eth-4vhostvr1024-2vm-ndrdisc-mrr
      10ge2p1x710-64b-1t1c-eth-l2bdbasemaclrn-eth-4vhostvr1024-2vm-mrr
      10ge2p1x710-imix-1t1c-eth-l2bdbasemaclrn-eth-4vhostvr1024-2vm-mrr
      10ge2p1xl710-64b-1t1c-eth-l2xcbase-eth-4vhostvr1024-2vm-ndrdisc-mrr
      10ge2p1xl710-imix-1t1c-eth-l2xcbase-eth-4vhostvr1024-2vm-ndrdisc-mrr
      10ge2p1xl710-64b-1t1c-eth-l2bdbasemaclrn-eth-4vhostvr1024-2vm-mrr
      10ge2p1xl710-imix-1t1c-eth-l2bdbasemaclrn-eth-4vhostvr1024-2vm-mrr
3. Following test cases shouldn’t be listed in dashboard tables:
      10ge2p1x520-64b-1t1c-eth-l2bdscale1mmaclrn-ndrdisc
      10ge2p1x520-64b-2t2c-eth-l2bdscale1mmaclrn-ndrdisc
      10ge2p1x520-64b-4t4c-eth-l2bdscale1mmaclrn-ndrdisc