BlyxDocumentation
Browse chapters
Documentation / Practical projects

Evaluate models with Blyx packages

Compare the cost of two classifiers, check their probability forecasts, and score a ranked retrieval result. Run the comparison in CI or import the same Blyx functions into your application.

Start with a release decision

You have predictions from two versions of a classifier on the same held-out observations. The current model raises one false alarm. Its replacement misses one positive. Both get three out of four right. Decide what a false alarm and a miss cost before you look at which model wins.

Prepare a paired CSV

Save the four rows below as predictions.csv in UTF-8. Each row must refer to the same observation for both models. The label is 0 or 1. Each prediction is an unquoted number between 0 and 1. Keep the header and column order shown. Rows can use LF or CRLF line endings.

label,baseline,candidate
1,0.9,0.4
1,0.8,0.9
0,0.6,0.2
0,0.1,0.1

Run ModelGate

ModelGate 0.1.2 ships as a native executable. On Apple Silicon macOS or x86-64 Linux use the Homebrew command below. On x64 Windows, extract the release ZIP from the ecosystem catalog and run .\modelgate.exe predictions.csv --fn-cost 5 in PowerShell. A release archive also contains fixtures/improved.csv and fixtures/regression.csv.

brew install rahulchaube1/blyx/modelgate
modelgate predictions.csv --fn-cost 5

Read the decision

At the default threshold of 0.5, a score equal to the threshold counts as positive. The baseline’s one false alarm costs 1. The candidate’s missed positive costs 5. Divide by four observations to get costs per row of 0.25 and 1.25. The JSON pass field is false. You can retain the report as a CI artifact using shell output redirection.

Exit codeMeaning
0Candidate meets the cost and Brier criteria.
1The candidate exceeds at least one permitted limit.
2The CSV, a value or a command option is invalid. Check the diagnostic.
4The native host could not complete an I/O operation.

Measure probability quality

Brier score averages the squared difference between each probability and its binary label. Two models with identical class decisions can have different Brier scores. ModelGate accepts --brier-tolerance for the largest allowed increase in this mean. The calibration library also provides cal_bin for counts, forecast totals and outcome totals, and cal_ece for a count-weighted summary of the gaps. Compare ECE on the same data using the same bin count.

Import a versioned library

Install Blyx 0.2.1, Rust 1.85 or later and a platform linker for a source build. In an empty project directory, run the clone command below. It puts the release beside the entry file. For a shared project, record the commit in version control, for example as a Git submodule.

git clone --depth 1 --branch v0.1.1 https://github.com/Rahulchaube1/blyx-classification.git vendor/classification

Use the library in a native program

Save this code as main.blyx beside vendor and run blyxc native-run main.blyx. It prints 6, the total cost of one false alarm and one miss. To keep the executable use blyxc build main.blyx -o evaluate, or -o evaluate.exe on Windows. The editable browser program below contains the same library in one file.

use vendor::classification::classification;

fn main() {
    println!(class_cost(1.0, 1.0, 1.0, 5.0));
}

Evaluate a retriever

Use the retrieval package to score an ordered list of numeric document IDs for one judged query. With results [7, 7, 9], judgments [7, 9] and k of 3, unique hits occur at ranks 1 and 3. Precision is 2/3, recall is 1, reciprocal rank is 1, and average precision is (1 + 2/3) / 2. A repeated document still uses a rank. Record which queries have judgments and average the per-query metrics over that set.

Keep future data out of a time-series backtest

Blyx Backtest measures forecasts in chronological folds. For demand [10, 20, 30, 40, 50, 60, 70, 80], set initial history to 3, forecast horizon to 2, gap to 1 and training window to 2. The first fold trains on 20 and 30, skips 40 and scores against 50 and 60. The next trains on 40 and 50, skips 60 and scores against 70 and 80. Across those four observations the last-value baseline has mean absolute error of 25, the training-mean baseline 30, and custom forecasts [45, 55, 65, 75] score 5. Use the same fold bounds when fitting preprocessing and model parameters.

let scores = bt_baselines([10, 20, 30, 40, 50, 60, 70, 80], 3, 2, 1, 2);

Choose what to test

Use held-out paired data and keep the error costs, threshold and tolerance fixed when comparing runs. Inspect the individual errors and the number of observations. The four-row fixture illustrates the decision rule. Statistical uncertainty, subgroup performance and business impact require evidence from your own evaluation.

Try the program

Output Ready
Expected output
0.5
0.5
6
Open in the full playground →

Practice

In the worked program, raise the cost of a missed positive from 5 to 10. Predict the three lines of output and then run it.

Hint

Only the total cost uses that argument. The confusion counts stay the same.

Worked solution

Precision and recall both remain 0.5. One false alarm costs 1 and one missed positive now costs 10, so the final line is 11.

Find

at(table, 3), 1.0, 5.0

Replace with

at(table, 3), 1.0, 10.0

Output after the change

0.5
0.5
11
Get the Blyx packages