THINK FIRST·CODE LATER

← All labs

Detect data drift with the population stability index

Problem

Input: a reference CSV, a line ---, and a current CSV with the same numeric columns.

For each column compute the PSI with 4 bins defined by the reference quantiles (np.unique(np.quantile(ref, [0, .25, .5, .75, 1])), with the first and last edges replaced by −∞ and +∞), proportions per bin clipped below at 0.0001, and PSI = Σ (cur − ref) · ln(cur / ref). Print temperature: mean 11.5 -> 23.9, PSI 5.852 (SIGNIFICANT shift - check the model) — no significant shift if PSI < 0.1, moderate shift if < 0.25, otherwise the significant message.

Data for this lab

Your program reads its data from standard input. Download the file, then give it to your program as shown below. Start with data set 1: its expected output is given.

v13-4-data-1.txt — first 14 lines of 403

temperature,trips,pm25

12.0,238.0,41.0

13.2,343.0,43.0

10.9,325.0,37.0

8.4,344.0,48.0

10.2,394.0,42.0

8.0,374.0,30.0

12.2,243.0,33.0

17.4,216.0,46.0

10.0,341.0,44.0

9.5,249.0,25.0

14.0,299.0,51.0

13.4,342.0,45.0

12.4,218.0,51.0

What your program must print for v13-4-data-1.txt
temperature: mean 11.5 -> 23.9, PSI 5.852 (SIGNIFICANT shift - check the model)
trips: mean 296.0 -> 313.2, PSI 0.128 (moderate shift)
pm25: mean 38.5 -> 45.4, PSI 0.632 (SIGNIFICANT shift - check the model)

Run your program first, then open this and compare line by line: your output must contain these lines exactly.

How to give the file to my program

Terminal (macOS, Linux, or Windows Command Prompt), in the folder that holds both files:

python my_lab.py < v13-4-data-1.txt

Windows PowerShell (the default terminal of VS Code on Windows) does not understand <:

Get-Content v13-4-data-1.txt | python my_lab.py

PyCharm: Run → Edit Configurations → tick Redirect input from and choose the file.

Jupyter: put this at the top of the cell that holds your program:

import sys
sys.stdin = open("v13-4-data-1.txt", encoding="utf-8")

Write it here, in Jupyter or in your IDE, then paste it. Run it on the sample input yourself before comparing. Your code stays in your browser — it is never sent to or stored on the server.