Regression Cycle, Debug & Tracking

Transcription

1 Regression Cycle, Debug & Tracking Balasubramanian Gopalakrishnan, Daniel Rigby, Mark Daniel - restricted -

2 The Regression Cycle DUT TB Regression Failures/ Cover Holes Daily regression cycle Fixed version of DUT & TB Regression 1000s of tests/random seeds This gives us 100s of failures/coverage holes We debug and fix issues in DUT and/or TB Repeat with new DUT/TB Debug Fix How many regression cycles 150 cycles to close coverage 25%->100% 40 cycles to close coverage 89%->100% 2

3 Debugging, Debugging and more Debugging Thousands of seeds run each day Can give hundreds of failures to debug on a daily basis Regression execution is cheap vs debug that is expensive With limited resources Do you keep the failures or throw them away? One of those hundreds of failures could be the critical RTL bug? How do you focus debug to the right failure? How do you stop people debugging the same issues? 3

4 Triage & Issue Reporting DUT TB Regression Failures/ Cover Holes Triage Debug Issue Reporting Fix Triage regression test failures Automatic Process Categorise the failures E.g. 100s of new failures can match to 5 categories, only 1 one which is new Target debug to the 1 new issue Match failures to existing Reported Issues failure category and test name stored in Issue ticket Can have known failures in a regression, and easily trace why they fail No need to over constrain to achieve 100% pass rate Can write an assertion to identify specific issue (e.g. automatic debug) Track who is debugging what 4

5 Regression Enrichment DUT TB How do we improve our regression content? Regression Failures/ Cover Holes Triage Debug Issue Reporting Fix Session Cover Ranking Failures Ranked coverage, add contributing testcases to regression Keeps corner case coverage hitting tests Initially 300 tests per cycle closing to 10 per cycle Add failures to next regression If not debugged on this cycle, then they are not lost Ensures fixed bugs are not reintroduced All failures or one from each category? One regression session with all of this content Automatically updated for the next cycle 5

6 Maintaining High Quality Code Changes Large teams all making changes to RTL or Verif Code High probability that changes from two people will not be compatible and give a catastrophic overnight regression How to ensure good quality code commits? How to avoid losing the result of one 24 hour cycle? Avoid wasting resources on unnecessary avoidable failures. 6

7 Quality Control Regression DUT QC Regression TB Regression Failures/ Cover Holes Triage Debug Issue Reporting Fix QC Session Session Cover Ranking Failures Mini, Coffee-Break, Gate-keeper Should run in ~10 minutes Used to qualify changes in RTL or TB If this fails changes are not committed Automatically added to with failing testcases (e.g. Interesting RTL/TB bugs) Ensures good quality commits Short High coverage yielding testcases Typical number of tests <400 tests ~50% are fast best coverage tests 7

8 Tracking Progress How do we know we are making progress with each cycle? We track a number of metrics Coverage Functional & Code Coverage Tests ranked and adding to coverage (# added to regression) New Failing Tests New Failure Categories (requiring debug) Categories Debugged (or not Debugged) Failures added to regression Issues Raised (Bugs Found) Bugs Fixed 8

9 Tracking Dashboard Track Metrics, Reference logfiles, reports, results Collected on a per cycle basis Scripts post data into database via API Results viewable via a configurable web based interface 9

10 Summary Small changes can make a big difference Don t over-constrain your random testbenches. Passes are cheap, Failures are valuable, they point towards a bug It s ok to have failing testcases (as long as they are not unknown RTL bugs) By automating Triage of failures we reduce the debug effort Adding traceability of failing tests, we can keep valuable test failures and ensure they are analysed, tracked and fixed 10

11