Canary analysis: promote or roll back?

A new StudyBuddy release is rolled out in stages. At each stage the canary (new version) and the baseline (old version) serve traffic at the same time. Decide automatically whether to continue.

Input: lines stagePercent baselineRequests baselineErrors canaryRequests canaryErrors canaryP95ms baselineP95ms until the end of input.

For each stage compute error rates (errors / requests, as percentages). The canary fails the stage if:

• its error rate is greater than 2 × the baseline's error rate and greater than 0.5% (absolute), or
• its p95 latency is more than 1.2 × the baseline p95.

Output per stage:

Stage 5%: baseline 0.40% errors, canary 0.50% errors, p95 310 vs 300 ms -> PROMOTE

or -> ROLLBACK (errors), -> ROLLBACK (latency) or -> ROLLBACK (errors, latency). Rates with two decimals. After the first ROLLBACK, stop processing and print Result: rolled back at X%. If every stage passes, print Result: fully released.

If a group has 0 requests, its error rate is 0.

Input:

1 10000 40 100 0 290 300
10 9000 36 1000 5 305 300
50 5000 20 5000 60 320 300
100 0 0 10000 50 300 300

Output:

Stage 1%: baseline 0.40% errors, canary 0.00% errors, p95 290 vs 300 ms -> PROMOTE
Stage 10%: baseline 0.40% errors, canary 0.50% errors, p95 305 vs 300 ms -> PROMOTE
Stage 50%: baseline 0.40% errors, canary 1.20% errors, p95 320 vs 300 ms -> ROLLBACK (errors)
Result: rolled back at 50%