Lessons in Risk – Foundations | The ARC Lab

What a Risk Score Represents

Wed, 29 Apr 2026 00:00:00 +0000

Module 1 · Risk Score Foundations

Risk scores are not labels.

A risk score is often treated like a label—low, moderate, or high. But that is not really what the score means. A risk score does not tell you exactly what will happen to one person. It tells you what tends to happen among people with similar scores.

Key takeaway

Risk scores are group-level probabilities, not exact predictions.

The Common Misunderstanding

Risk scores are often treated as fixed categories. But they are better understood as positions along a continuum.

The idea: A risk score is not a label—it is a position within a distribution.

How Risk Scores Usually Look

Scores tend to form a distribution that looks like a bell curve. Most people cluster in the middle, with fewer at the extremes.

Illustration

Risk scores form a distribution

Most people cluster in the middle of the score range, with fewer at the low and high ends. Each part of this distribution is associated with a different observed outcome rate.

The score places a person within this distribution—linking them to what tends to happen among others with similar scores.

How to Read the Figure

Lower scores → lower outcome rates (~10%)
Middle scores → moderate rates (~25%)
Higher scores → higher rates (~50%)

The score does not predict an individual outcome. It tells you what tends to happen among people with similar scores.

Bottom Line

Risk scores are group-level probabilities—not exact predictions.

← Back to Modules Next Module How Risk Scores Are Built →

How Risk Scores Are Built

Fri, 01 May 2026 00:00:00 +0000

Module 2 · Risk Tool Lessons

Where risk scores come from.

In the last module, we saw that a risk score reflects what tends to happen among people with similar scores. So where do those scores come from? They are built from a set of factors, or “items,” that are combined into a single score.

Key takeaway

Risk scores are built from multiple items, and how those items are combined matters.

What Goes Into a Risk Score

Each item captures something about a person. Common examples include:

prior history
age
substance use
employment
peer associations

These items are then combined into a single score.

The idea: A risk score is built by combining information across multiple items.

From Items to a Score

The figure below shows a simplified example of how items are combined into a score.

Illustration

How risk scores are built

Risk scores are constructed by combining information from multiple items into a single value. The same inputs can be combined in different ways.

Different tools use different scoring rules—some treat all items equally, while others weight items based on their relationship to outcomes.

Two Common Approaches

There are two common ways to combine items into a score:

1. Burgess-style scoring (e.g., LS/CMI; ORAS)

Items are typically binary (0/1)
Each item contributes equally
Simple and easy to hand score

2. Statistically weighted scoring (e.g., COMPAS; STRONG-R)

Items can have different weights
Weights reflect relationships with outcomes, such as recidivism

Why This Matters

The same inputs can produce different scores depending on how they are combined.

Burgess-style tools emphasize simplicity and transparency. Weighted approaches often achieve higher predictive accuracy, though the magnitude of that improvement varies.

Bottom Line

Risk scores are built from a set of items. The way those items are combined shapes what the score means and how well it performs.

← Back to Modules Next Module Same Score, Different Person →

Same Score, Different Person

Mon, 04 May 2026 00:00:00 +0000

Module 3 · Risk Tool Lessons

Same score. Different profiles.

Two people can have the same risk score and still be very different. This happens because risk scores combine multiple factors into a single number.

Key takeaway

Risk scores summarize information—they do not fully represent it.

Same Score Does Not Mean Same Person

The figure below shows a simple example. Both individuals receive the same total score, but they get there in different ways.

One person has greater prior history
The other has more recent needs, such as substance use or employment

The idea: Different pathways can produce the same score.

Illustration

Same score, different pathways

Two people can receive the same total score even when the factors behind that score look very different.

The total score summarizes the profile, but it does not show every difference that produced that score.

Why This Happens

Risk tools combine several factors into one total score. That total score can be useful, but it also compresses information.

Because the same score maps to the same predicted probability of recidivism, two people with the same score may be treated as equally “risky,” even when their underlying profiles are very different.

Why This Matters

Individuals with the same score may need different interventions
A single number can hide meaningful differences
Interpretation requires looking beyond the total score

Bottom Line

Same score does not mean same person. Risk scores summarize information, but they do not fully represent the individual profiles behind the number.

← Back to Modules Next Module What "Accuracy" Really means →

What “Accuracy” Really Means

Wed, 06 May 2026 00:00:00 +0000

Module 4 · Risk Tool Lessons

Accuracy is meaningful separation.

When people talk about a “more accurate” risk assessment tool, what do they mean? It is not just about getting individual cases right. A stronger tool separates lower- and higher-risk individuals more clearly.

Key takeaway

Accuracy is about how well a tool separates outcomes—not just how often it is “right.”

Two Tools Can Behave Differently

Both tools in the figure below assign risk scores. But they behave very differently.

One creates greater separation between lower- and higher-risk individuals
The other compresses people into a narrower range of probabilities

The idea: The difference is separation.

Illustration

Accuracy means separation

Two tools can assign risk scores, but differ in how much they separate lower- and higher-risk individuals.

A stronger tool creates clearer separation across risk levels. A weaker tool compresses people into a narrower range of predicted probabilities.

How to Read the Figure

The same score can correspond to very different predicted probabilities depending on the tool.

At a score of 20, one tool corresponds to about 20% risk, while the other corresponds to about 40%
At a score of 80, one tool corresponds to about 75% risk, while the other corresponds to about 45%

A stronger tool spreads people out meaningfully across risk levels. A weaker tool groups people closer together, even when outcomes differ.

Why This Matters

Better separation supports more informed decisions
Poor separation limits how useful a tool can be
Two tools can look similar on the surface but behave very differently

Bottom Line

Think of accuracy as meaningful separation. A useful risk tool does not just assign scores—it meaningfully distinguishes between different levels of risk.

← Back to Modules Next Module What AUC Actually Measures →

What AUC Actually Measures

Mon, 11 May 2026 00:00:00 +0000

Module 5 · Risk Tool Lessons

AUC is about ranking.

Assessment tools are often described using something called “AUC.” AUC stands for “Area Under the Curve,” but conceptually it answers a simple question: how often does the tool correctly rank people?

Key takeaway

AUC measures ranking—not how risk scores behave in practice.

What AUC Means

Imagine randomly picking two people from two groups:

one person who reoffended
one person who did not reoffend

The AUC is the probability that the tool assigns a higher score to the person who reoffends.

The idea: AUC asks how well the tool orders people from lower to higher risk.

Illustration

AUC compares pairs

AUC asks whether the tool gives a higher score to the person who reoffends when compared with a person who does not.

Higher AUC values mean the tool more often ranks people in the expected order.

How to Read AUC Values

AUC = 0.50 → no better than chance
AUC = 0.75 → correctly ranks pairs 75% of the time
AUC = 1.00 → perfect separation

What AUC Is Not About

AUC is not about exact probabilities. It is also not about individual predictions.

It tells you how well the tool ranks people from lower to higher risk, not whether a specific score corresponds to a useful or well-separated probability in practice.

Bottom Line

AUC measures ranking. It tells you how often a tool orders people correctly, but it does not fully explain how risk scores behave in real decision settings.

← Back to Modules Next Module Same AUC, Different Behavior →

Same AUC, Different Behavior

Fri, 15 May 2026 00:00:00 +0000

Module 6 · Risk Tool Lessons

Same AUC does not mean same tool.

In the last module, we saw that AUC measures how well a tool ranks people from lower to higher risk. But two tools can have the same AUC and still behave very differently.

Key takeaway

AUC measures ranking—not how scores map to real-world decisions.

Same Ranking Ability, Different Scores

Both tools in the figure below have the same AUC. That means they are equally good at ranking individuals from lower to higher risk.

But their scores behave very differently in practice.

The idea: Two tools can rank equally well while producing very different risk estimates.

Illustration

Same AUC, different behavior

Two tools can have identical AUC values while producing very different risk estimates across the score range.

AUC captures ranking ability, but it does not fully describe how scores behave in practice.

How the Scores Differ

At a score of 20, one tool corresponds to about 20% risk while the other corresponds to about 40%
At a score of 80, one tool corresponds to about 75% risk while the other corresponds to about 45%

The tools have the same ranking ability, but they produce very different risk estimates.

Why This Matters

Decisions are based on scores and probabilities—not just ranking
Tools with the same AUC can lead to different classifications
Policy outcomes can differ even when AUC is identical

← Back to Modules Next Module Base Rates Change Everything →

Base Rates Change Everything

Wed, 06 May 2026 00:00:00 +0000

Module 7 · Risk Tool Lessons

Same tool. Same score. Different population.

A risk assessment tool does not operate in a vacuum. Its predictions depend heavily on the outcome’s base rate: how common the outcome is for a population.

Key takeaway

Risk scores are not universal constants—they are tied to populations and outcomes.

What a Base Rate Means

A base rate is simply how common an outcome is for a population.

The base rate of a probation population will likely be lower than the base rate of a parole population.

The idea: The same score can correspond to very different probabilities across populations.

Same Tool, Different Meaning

The figure below shows a simple example.

Both groups use the same risk tool. Both individuals receive the same score. But the predicted probabilities are very different because the underlying outcome rates differ across populations.

Illustration

Base rates change score meaning

The same tool and same score can correspond to different predicted probabilities when the outcome is more or less common in different populations.

A score only makes sense in relation to the population and outcome it is tied to.

How the Scores Differ

In one population, a score of 50 might correspond to about 15% risk
In another population, the exact same score might correspond to about 45% risk

Same tool. Same score. Different population.

Why This Matters

Risk estimates depend on the sample the tool was developed on
Tools may behave differently across jurisdictions
Base rates affect how useful scores are for decision-making

Why Revalidation Matters

This is one reason tools often need revalidation and recalibration when moved to new settings.

If your state or agency has adopted a tool developed elsewhere, has it been evaluated recently?

Bottom Line

Risk scores are not universal constants. They are tied to populations, outcomes, and the settings where they are used.

← Back to Modules Next Module Calibration →

Calibration

Thu, 21 May 2026 00:00:00 +0000

Module 8 · Risk Tool Lessons

Ranking well is not enough.

A risk assessment tool can rank people correctly and still produce inaccurate probabilities. That is where calibration comes in.

Key takeaway

A tool can rank people well and still produce misleading probabilities.

What Calibration Means

Calibration asks a different question than AUC:

The idea: Do predicted probabilities actually match observed outcomes?

In a well-calibrated tool, predicted risk closely matches what actually happens. For example, a predicted 30% risk corresponds to about 30% observed recidivism.

Illustration

Calibration compares prediction to reality

A well-calibrated tool produces predicted probabilities that closely match observed outcomes.

Calibration asks whether predicted risk matches what actually happens across different score levels.

How Tools Mis-Calibrate

Tools can mis-calibrate in different ways:

Under-prediction → actual outcomes are higher than predicted
Over-prediction → predicted probabilities are too high
Mis-specified slopes → errors change across risk levels

Why This Matters

Decisions are often based on probabilities, not just rankings
Poor calibration can distort supervision and classification decisions
Tools may calibrate differently across populations and jurisdictions

Why Recalibration Matters

If a state or agency has adopted a tool developed elsewhere, has its calibration been evaluated?

Tools may perform differently when moved to new settings, populations, or jurisdictions.

Bottom Line

A tool can rank people well and still produce misleading probabilities. Calibration asks whether predicted risk actually matches observed outcomes.

← Back to Modules

Why Risk Scores Are Misunderstood

Mon, 27 Apr 2026 00:00:00 +0000

Module 0 · Start Here

Risk scores shape real decisions.

In corrections, classification, and pretrial settings, risk scores influence decisions about incarceration, supervision, and programming. But one basic issue is often overlooked: not all risk scores mean the same thing.

Key takeaway

A risk score is only useful if it meaningfully distinguishes between outcomes.

Where Risk Scores Show Up

Risk-needs assessment tools are used every day in:

adult and juvenile corrections
prison classification
pretrial release decisions

They shape decisions about incarceration, supervision, and programming.

The Basic Problem

Two tools can assign the same “risk score," but behave very differently.

That matters because a risk score is only useful if it meaningfully distinguishes between outcomes.

The idea: Not all risk scores carry the same practical meaning.

See the Difference

Same Score, Different Performance

Both tools below assign a risk score. But they do not perform the same way.

More Accurate Tool

Clear separation between lower- and higher-risk individuals.

Less Accurate Tool

Scores vary, but outcomes are not well distinguished.

Why the Difference Matters

Both tools generate “risk scores.” But only one meaningfully separates outcomes.

That difference—how clearly a tool distinguishes between outcomes—is what we mean by accuracy in practice.

Why This Is Easy to Miss

Many risk assessment tools still in use today are built on empirical foundations that predate the 1980s.

However, understanding of how these tools work—and how to evaluate them—has not kept pace.

As a result, risk scores are often interpreted as if they carry the same meaning, even when they do not.

Bottom Line

Risk scores are widely used but often poorly understood. Understanding what they represent is a prerequisite for using them well.

Next Module What a Risk Score Represents →