Statistics: rocmetrics

Class Definition: `rocmetrics`

statistics: rocmetrics

Receiver operating characteristic (ROC) metrics for classifier output.

The rocmetrics class evaluates a classifier’s performance by computing, for each class, a one-versus-all ROC curve together with a set of threshold-dependent performance metrics. It stores the results in the Metrics table and the per-class area under the curve in AUC, and provides the addMetrics, average, and plot methods for follow-up analysis.

For a problem with $K$ classes and scores supplied as an $N$ -by- $K$ matrix, the discriminant score used for class k is the one-versus-all margin Scores(:,k) - max (Scores(:,j)) over j != k, matching MATLAB’s rocmetrics. Every metric is evaluated at each distinct value of that margin.

See also: perfcurve, confusionmat, confusionchart

Source Code: rocmetrics

The rocmetrics class contains the following properties:

Table of performance metrics, vertically concatenated across the classes in ClassNames order with one row per distinct threshold. The standard variables are ClassName, Threshold, FalsePositiveRate, and TruePositiveRate, followed by one variable for each metric requested through 'AdditionalMetrics'. This property is read-only.

One-versus-all ROC curves for a three-class problem

 labels = [1 1 2 2 3 3]';
 scores = [0.9 0.05 0.05; 0.6 0.3 0.1; 0.2 0.7 0.1; ...
           0.1 0.6 0.3; 0.2 0.2 0.6; 0.1 0.3 0.6];
 rocObj = rocmetrics (labels, scores, [1 2 3]);
 disp (rocObj.AUC)

   1   1   1

 plot (rocObj);

Binary ROC metrics with additional performance metrics

 labels = [1 1 1 1 0 0]';
 p = [0.9 0.7 0.4 0.3 0.8 0.2]';
 scores = [1-p p];
 rocObj = rocmetrics (labels, scores, [0 1], ...
                      "AdditionalMetrics", {"Accuracy", "ExpectedCost"});
 head = rocObj.Metrics(1:4,:);
 disp (head);

  4x6 table

    ClassName    Threshold    FalsePositiveRate    TruePositiveRate    Accuracy    ExpectedCost    
    _________    _________    _________________    ________________    ________    ____________    

            0          0.6                    0                   0    0.666667       0.0740741    
            0          0.6                    0                 0.5    0.833333        0.037037    
            0          0.4                 0.25                 0.5    0.666667       0.0740741    
            0          0.2                  0.5                 0.5         0.5        0.111111

Row vector holding the area under the one-versus-all ROC curve for each class, in ClassNames order. This property is read-only.

One-versus-all ROC curves for a three-class problem

 labels = [1 1 2 2 3 3]';
 scores = [0.9 0.05 0.05; 0.6 0.3 0.1; 0.2 0.7 0.1; ...
           0.1 0.6 0.3; 0.2 0.2 0.6; 0.1 0.3 0.6];
 rocObj = rocmetrics (labels, scores, [1 2 3]);
 disp (rocObj.AUC)

   1   1   1

 plot (rocObj);

Binary ROC metrics with additional performance metrics

 labels = [1 1 1 1 0 0]';
 p = [0.9 0.7 0.4 0.3 0.8 0.2]';
 scores = [1-p p];
 rocObj = rocmetrics (labels, scores, [0 1], ...
                      "AdditionalMetrics", {"Accuracy", "ExpectedCost"});
 head = rocObj.Metrics(1:4,:);
 disp (head);

  4x6 table

    ClassName    Threshold    FalsePositiveRate    TruePositiveRate    Accuracy    ExpectedCost    
    _________    _________    _________________    ________________    ________    ____________    

            0          0.6                    0                   0    0.666667       0.0740741    
            0          0.6                    0                 0.5    0.833333        0.037037    
            0          0.4                 0.25                 0.5    0.666667       0.0740741    
            0          0.2                  0.5                 0.5         0.5        0.111111

Class names for which the ROC metrics are computed, in the column order of Scores. This property is read-only.

One-versus-all ROC curves for a three-class problem

 labels = [1 1 2 2 3 3]';
 scores = [0.9 0.05 0.05; 0.6 0.3 0.1; 0.2 0.7 0.1; ...
           0.1 0.6 0.3; 0.2 0.2 0.6; 0.1 0.3 0.6];
 rocObj = rocmetrics (labels, scores, [1 2 3]);
 disp (rocObj.AUC)

   1   1   1

 plot (rocObj);

Binary ROC metrics with additional performance metrics

 labels = [1 1 1 1 0 0]';
 p = [0.9 0.7 0.4 0.3 0.8 0.2]';
 scores = [1-p p];
 rocObj = rocmetrics (labels, scores, [0 1], ...
                      "AdditionalMetrics", {"Accuracy", "ExpectedCost"});
 head = rocObj.Metrics(1:4,:);
 disp (head);

  4x6 table

    ClassName    Threshold    FalsePositiveRate    TruePositiveRate    Accuracy    ExpectedCost    
    _________    _________    _________________    ________________    ________    ____________    

            0          0.6                    0                   0    0.666667       0.0740741    
            0          0.6                    0                 0.5    0.833333        0.037037    
            0          0.4                 0.25                 0.5    0.666667       0.0740741    
            0          0.2                  0.5                 0.5         0.5        0.111111

Square misclassification-cost matrix, with zero diagonal and unit off-diagonal entries by default. It is used by the ExpectedCost metric. This property is read-only.

One-versus-all ROC curves for a three-class problem

 labels = [1 1 2 2 3 3]';
 scores = [0.9 0.05 0.05; 0.6 0.3 0.1; 0.2 0.7 0.1; ...
           0.1 0.6 0.3; 0.2 0.2 0.6; 0.1 0.3 0.6];
 rocObj = rocmetrics (labels, scores, [1 2 3]);
 disp (rocObj.AUC)

   1   1   1

 plot (rocObj);

Binary ROC metrics with additional performance metrics

 labels = [1 1 1 1 0 0]';
 p = [0.9 0.7 0.4 0.3 0.8 0.2]';
 scores = [1-p p];
 rocObj = rocmetrics (labels, scores, [0 1], ...
                      "AdditionalMetrics", {"Accuracy", "ExpectedCost"});
 head = rocObj.Metrics(1:4,:);
 disp (head);

  4x6 table

    ClassName    Threshold    FalsePositiveRate    TruePositiveRate    Accuracy    ExpectedCost    
    _________    _________    _________________    ________________    ________    ____________    

            0          0.6                    0                   0    0.666667       0.0740741    
            0          0.6                    0                 0.5    0.833333        0.037037    
            0          0.4                 0.25                 0.5    0.666667       0.0740741    
            0          0.2                  0.5                 0.5         0.5        0.111111

Row vector of prior class probabilities, in ClassNames order, summing to one. This property is read-only.

One-versus-all ROC curves for a three-class problem

 labels = [1 1 2 2 3 3]';
 scores = [0.9 0.05 0.05; 0.6 0.3 0.1; 0.2 0.7 0.1; ...
           0.1 0.6 0.3; 0.2 0.2 0.6; 0.1 0.3 0.6];
 rocObj = rocmetrics (labels, scores, [1 2 3]);
 disp (rocObj.AUC)

   1   1   1

 plot (rocObj);

Binary ROC metrics with additional performance metrics

 labels = [1 1 1 1 0 0]';
 p = [0.9 0.7 0.4 0.3 0.8 0.2]';
 scores = [1-p p];
 rocObj = rocmetrics (labels, scores, [0 1], ...
                      "AdditionalMetrics", {"Accuracy", "ExpectedCost"});
 head = rocObj.Metrics(1:4,:);
 disp (head);

  4x6 table

    ClassName    Threshold    FalsePositiveRate    TruePositiveRate    Accuracy    ExpectedCost    
    _________    _________    _________________    ________________    ________    ____________    

            0          0.6                    0                   0    0.666667       0.0740741    
            0          0.6                    0                 0.5    0.833333        0.037037    
            0          0.4                 0.25                 0.5    0.666667       0.0740741    
            0          0.2                  0.5                 0.5         0.5        0.111111

True class labels supplied at construction, one per observation. This property is read-only.

One-versus-all ROC curves for a three-class problem

 labels = [1 1 2 2 3 3]';
 scores = [0.9 0.05 0.05; 0.6 0.3 0.1; 0.2 0.7 0.1; ...
           0.1 0.6 0.3; 0.2 0.2 0.6; 0.1 0.3 0.6];
 rocObj = rocmetrics (labels, scores, [1 2 3]);
 disp (rocObj.AUC)

   1   1   1

 plot (rocObj);

Binary ROC metrics with additional performance metrics

 labels = [1 1 1 1 0 0]';
 p = [0.9 0.7 0.4 0.3 0.8 0.2]';
 scores = [1-p p];
 rocObj = rocmetrics (labels, scores, [0 1], ...
                      "AdditionalMetrics", {"Accuracy", "ExpectedCost"});
 head = rocObj.Metrics(1:4,:);
 disp (head);

  4x6 table

    ClassName    Threshold    FalsePositiveRate    TruePositiveRate    Accuracy    ExpectedCost    
    _________    _________    _________________    ________________    ________    ____________    

            0          0.6                    0                   0    0.666667       0.0740741    
            0          0.6                    0                 0.5    0.833333        0.037037    
            0          0.4                 0.25                 0.5    0.666667       0.0740741    
            0          0.2                  0.5                 0.5         0.5        0.111111

Classification scores supplied at construction, as an $N$ -by- $K$ matrix. This property is read-only.

One-versus-all ROC curves for a three-class problem

 labels = [1 1 2 2 3 3]';
 scores = [0.9 0.05 0.05; 0.6 0.3 0.1; 0.2 0.7 0.1; ...
           0.1 0.6 0.3; 0.2 0.2 0.6; 0.1 0.3 0.6];
 rocObj = rocmetrics (labels, scores, [1 2 3]);
 disp (rocObj.AUC)

   1   1   1

 plot (rocObj);

Binary ROC metrics with additional performance metrics

 labels = [1 1 1 1 0 0]';
 p = [0.9 0.7 0.4 0.3 0.8 0.2]';
 scores = [1-p p];
 rocObj = rocmetrics (labels, scores, [0 1], ...
                      "AdditionalMetrics", {"Accuracy", "ExpectedCost"});
 head = rocObj.Metrics(1:4,:);
 disp (head);

  4x6 table

    ClassName    Threshold    FalsePositiveRate    TruePositiveRate    Accuracy    ExpectedCost    
    _________    _________    _________________    ________________    ________    ____________    

            0          0.6                    0                   0    0.666667       0.0740741    
            0          0.6                    0                 0.5    0.833333        0.037037    
            0          0.4                 0.25                 0.5    0.666667       0.0740741    
            0          0.2                  0.5                 0.5         0.5        0.111111

Non-negative observation weights, one per observation. Defaults to a vector of ones. This property is read-only.

One-versus-all ROC curves for a three-class problem

 labels = [1 1 2 2 3 3]';
 scores = [0.9 0.05 0.05; 0.6 0.3 0.1; 0.2 0.7 0.1; ...
           0.1 0.6 0.3; 0.2 0.2 0.6; 0.1 0.3 0.6];
 rocObj = rocmetrics (labels, scores, [1 2 3]);
 disp (rocObj.AUC)

   1   1   1

 plot (rocObj);

Binary ROC metrics with additional performance metrics

 labels = [1 1 1 1 0 0]';
 p = [0.9 0.7 0.4 0.3 0.8 0.2]';
 scores = [1-p p];
 rocObj = rocmetrics (labels, scores, [0 1], ...
                      "AdditionalMetrics", {"Accuracy", "ExpectedCost"});
 head = rocObj.Metrics(1:4,:);
 disp (head);

  4x6 table

    ClassName    Threshold    FalsePositiveRate    TruePositiveRate    Accuracy    ExpectedCost    
    _________    _________    _________________    ________________    ________    ____________    

            0          0.6                    0                   0    0.666667       0.0740741    
            0          0.6                    0                 0.5    0.833333        0.037037    
            0          0.4                 0.25                 0.5    0.666667       0.0740741    
            0          0.2                  0.5                 0.5         0.5        0.111111

The rocmetrics class offers the following public methods:

rocmetrics: obj = rocmetrics (labels, scores, classnames)
rocmetrics: obj = rocmetrics (…, Name, Value)

labels is a vector of true class labels with one element per observation; it may be numeric, logical, a character matrix, or a cell array of character vectors. scores is an $N$ -by- $K$ numeric matrix of classification scores, where scores(i,k) is the score of observation i for the class classnames(k). classnames lists the $K$ classes in the column order of scores.

The following Name-Value pairs are supported:

`Name`	`Value`
`'AdditionalMetrics'`	A character vector or cell array of metric names to append to `Metrics`. Supported names are `'TruePositives'`, `'FalseNegatives'`, `'FalsePositives'`, `'TrueNegatives'`, `'SumOfTrueAndFalsePositives'`, `'RateOfPositivePredictions'`, `'RateOfNegativePredictions'`, `'Accuracy'`, `'FalseNegativeRate'`, `'TrueNegativeRate'`, `'PositivePredictiveValue'`, `'NegativePredictiveValue'`, `'ExpectedCost'`, and `'f1score'`.
`'Prior'`	Prior class probabilities, given as `'empirical'` (default), `'uniform'`, or a numeric vector with one value per class.
`'Cost'`	A $K$ -by- $K$ misclassification-cost matrix used by the `ExpectedCost` metric. The default has zero diagonal and unit off-diagonal entries.
`'Weights'`	A vector of non-negative observation weights. The default is a vector of ones.
`'NaNFlag'`	How to treat `NaN` scores: `'omitnan'` (default) drops the affected observations, while `'includenan'` treats them as always classified negative.
`'FixedMetricValues'`	`'all'` (default) to use every distinct threshold, or a numeric vector of threshold values at which to report the curve (nearest actual thresholds are returned).

Construction from a trained model object, the 'FixedMetric' grids other than thresholds, and bootstrap confidence intervals are not implemented.

One-versus-all ROC curves for a three-class problem

 labels = [1 1 2 2 3 3]';
 scores = [0.9 0.05 0.05; 0.6 0.3 0.1; 0.2 0.7 0.1; ...
           0.1 0.6 0.3; 0.2 0.2 0.6; 0.1 0.3 0.6];
 rocObj = rocmetrics (labels, scores, [1 2 3]);
 disp (rocObj.AUC)

   1   1   1

 plot (rocObj);

Binary ROC metrics with additional performance metrics

 labels = [1 1 1 1 0 0]';
 p = [0.9 0.7 0.4 0.3 0.8 0.2]';
 scores = [1-p p];
 rocObj = rocmetrics (labels, scores, [0 1], ...
                      "AdditionalMetrics", {"Accuracy", "ExpectedCost"});
 head = rocObj.Metrics(1:4,:);
 disp (head);

  4x6 table

    ClassName    Threshold    FalsePositiveRate    TruePositiveRate    Accuracy    ExpectedCost    
    _________    _________    _________________    ________________    ________    ____________    

            0          0.6                    0                   0    0.666667       0.0740741    
            0          0.6                    0                 0.5    0.833333        0.037037    
            0          0.4                 0.25                 0.5    0.666667       0.0740741    
            0          0.2                  0.5                 0.5         0.5        0.111111

rocmetrics: obj = addMetrics (obj, metrics)

metrics is a metric name or a cell array of metric names, chosen from the list supported by the 'AdditionalMetrics' constructor argument. The named metrics are appended as new variables of the Metrics table; metrics already present are left unchanged.

One-versus-all ROC curves for a three-class problem

 labels = [1 1 2 2 3 3]';
 scores = [0.9 0.05 0.05; 0.6 0.3 0.1; 0.2 0.7 0.1; ...
           0.1 0.6 0.3; 0.2 0.2 0.6; 0.1 0.3 0.6];
 rocObj = rocmetrics (labels, scores, [1 2 3]);
 disp (rocObj.AUC)

   1   1   1

 plot (rocObj);

Binary ROC metrics with additional performance metrics

 labels = [1 1 1 1 0 0]';
 p = [0.9 0.7 0.4 0.3 0.8 0.2]';
 scores = [1-p p];
 rocObj = rocmetrics (labels, scores, [0 1], ...
                      "AdditionalMetrics", {"Accuracy", "ExpectedCost"});
 head = rocObj.Metrics(1:4,:);
 disp (head);

  4x6 table

    ClassName    Threshold    FalsePositiveRate    TruePositiveRate    Accuracy    ExpectedCost    
    _________    _________    _________________    ________________    ________    ____________    

            0          0.6                    0                   0    0.666667       0.0740741    
            0          0.6                    0                 0.5    0.833333        0.037037    
            0          0.4                 0.25                 0.5    0.666667       0.0740741    
            0          0.2                  0.5                 0.5         0.5        0.111111

rocmetrics: [FPR, TPR, Thresholds, AUC] = average (obj, type)

type selects the averaging method: 'macro' (unweighted mean of the per-class curves), 'micro' (a single curve pooling every one-versus-all instance), or 'weighted' (mean of the per-class curves weighted by Prior). The function returns the averaged false and true positive rates FPR and TPR, the corresponding Thresholds, and the area AUC under the averaged curve.

The averaged curve is evaluated on the union of the per-class thresholds. MATLAB inserts additional staircase points when building the averaged curve, so the exact rows and the averaged AUC may differ slightly from MATLAB.

One-versus-all ROC curves for a three-class problem

 labels = [1 1 2 2 3 3]';
 scores = [0.9 0.05 0.05; 0.6 0.3 0.1; 0.2 0.7 0.1; ...
           0.1 0.6 0.3; 0.2 0.2 0.6; 0.1 0.3 0.6];
 rocObj = rocmetrics (labels, scores, [1 2 3]);
 disp (rocObj.AUC)

   1   1   1

 plot (rocObj);

Binary ROC metrics with additional performance metrics

 labels = [1 1 1 1 0 0]';
 p = [0.9 0.7 0.4 0.3 0.8 0.2]';
 scores = [1-p p];
 rocObj = rocmetrics (labels, scores, [0 1], ...
                      "AdditionalMetrics", {"Accuracy", "ExpectedCost"});
 head = rocObj.Metrics(1:4,:);
 disp (head);

  4x6 table

    ClassName    Threshold    FalsePositiveRate    TruePositiveRate    Accuracy    ExpectedCost    
    _________    _________    _________________    ________________    ________    ____________    

            0          0.6                    0                   0    0.666667       0.0740741    
            0          0.6                    0                 0.5    0.833333        0.037037    
            0          0.4                 0.25                 0.5    0.666667       0.0740741    
            0          0.2                  0.5                 0.5         0.5        0.111111

rocmetrics: plot (obj)
rocmetrics: h = plot (obj)

Each class in ClassNames contributes one true-positive-rate versus false-positive-rate curve. A handle to the line objects is returned in h when requested.

One-versus-all ROC curves for a three-class problem

 labels = [1 1 2 2 3 3]';
 scores = [0.9 0.05 0.05; 0.6 0.3 0.1; 0.2 0.7 0.1; ...
           0.1 0.6 0.3; 0.2 0.2 0.6; 0.1 0.3 0.6];
 rocObj = rocmetrics (labels, scores, [1 2 3]);
 disp (rocObj.AUC)

   1   1   1

 plot (rocObj);

Binary ROC metrics with additional performance metrics

 labels = [1 1 1 1 0 0]';
 p = [0.9 0.7 0.4 0.3 0.8 0.2]';
 scores = [1-p p];
 rocObj = rocmetrics (labels, scores, [0 1], ...
                      "AdditionalMetrics", {"Accuracy", "ExpectedCost"});
 head = rocObj.Metrics(1:4,:);
 disp (head);

  4x6 table

    ClassName    Threshold    FalsePositiveRate    TruePositiveRate    Accuracy    ExpectedCost    
    _________    _________    _________________    ________________    ________    ____________    

            0          0.6                    0                   0    0.666667       0.0740741    
            0          0.6                    0                 0.5    0.833333        0.037037    
            0          0.4                 0.25                 0.5    0.666667       0.0740741    
            0          0.2                  0.5                 0.5         0.5        0.111111

rocmetrics: disp (obj)

One-versus-all ROC curves for a three-class problem

 labels = [1 1 2 2 3 3]';
 scores = [0.9 0.05 0.05; 0.6 0.3 0.1; 0.2 0.7 0.1; ...
           0.1 0.6 0.3; 0.2 0.2 0.6; 0.1 0.3 0.6];
 rocObj = rocmetrics (labels, scores, [1 2 3]);
 disp (rocObj.AUC)

   1   1   1

 plot (rocObj);

Binary ROC metrics with additional performance metrics

 labels = [1 1 1 1 0 0]';
 p = [0.9 0.7 0.4 0.3 0.8 0.2]';
 scores = [1-p p];
 rocObj = rocmetrics (labels, scores, [0 1], ...
                      "AdditionalMetrics", {"Accuracy", "ExpectedCost"});
 head = rocObj.Metrics(1:4,:);
 disp (head);

  4x6 table

    ClassName    Threshold    FalsePositiveRate    TruePositiveRate    Accuracy    ExpectedCost    
    _________    _________    _________________    ________________    ________    ____________    

            0          0.6                    0                   0    0.666667       0.0740741    
            0          0.6                    0                 0.5    0.833333        0.037037    
            0          0.4                 0.25                 0.5    0.666667       0.0740741    
            0          0.2                  0.5                 0.5         0.5        0.111111

Examples

 labels = [1 1 2 2 3 3]';
 scores = [0.9 0.05 0.05; 0.6 0.3 0.1; 0.2 0.7 0.1; ...
           0.1 0.6 0.3; 0.2 0.2 0.6; 0.1 0.3 0.6];
 rocObj = rocmetrics (labels, scores, [1 2 3]);
 disp (rocObj.AUC)

   1   1   1

 plot (rocObj);

 labels = [1 1 1 1 0 0]';
 p = [0.9 0.7 0.4 0.3 0.8 0.2]';
 scores = [1-p p];
 rocObj = rocmetrics (labels, scores, [0 1], ...
                      "AdditionalMetrics", {"Accuracy", "ExpectedCost"});
 head = rocObj.Metrics(1:4,:);
 disp (head);

  4x6 table

    ClassName    Threshold    FalsePositiveRate    TruePositiveRate    Accuracy    ExpectedCost    
    _________    _________    _________________    ________________    ________    ____________    

            0          0.6                    0                   0    0.666667       0.0740741    
            0          0.6                    0                 0.5    0.833333        0.037037    
            0          0.4                 0.25                 0.5    0.666667       0.0740741    
            0          0.2                  0.5                 0.5         0.5        0.111111

Categories &

Functions List

Descriptive Statistics

Descriptive Statistics

Data Manipulation

Data Manipulation

Hypothesis Testing

Hypothesis Testing

Experimental Design

Experimental Design

Regression

Regression

Supervised Learning

Supervised Learning

Clustering

Clustering

Nearest Neighbors

Nearest Neighbors

Anomaly Detection

Anomaly Detection

Dimensionality Reduction

Dimensionality Reduction

Model Evaluation

Model Evaluation

Markov Models

Markov Models

Random Sampling

Random Sampling

Distribution Classes

Distribution Classes

Distribution Fitting

Distribution Fitting

Distribution Functions

Distribution Functions

Distribution Statistics

Distribution Statistics

Distribution Wrappers

Distribution Wrappers

Plotting

Plotting

I/O

I/O

Utilities

Utilities

Class Definition: rocmetrics

statistics: rocmetrics

Metrics

AUC

ClassNames

Cost

Prior

Labels

Scores

Weights

rocmetrics

rocmetrics: obj = rocmetrics (labels, scores, classnames)

rocmetrics: obj = rocmetrics (…, Name, Value)

addMetrics

rocmetrics: obj = addMetrics (obj, metrics)

average

rocmetrics: [FPR, TPR, Thresholds, AUC] = average (obj, type)

plot

rocmetrics: plot (obj)

rocmetrics: h = plot (obj)

disp

rocmetrics: disp (obj)

Examples

Class Definition: `rocmetrics`

`Metrics`

`AUC`

`ClassNames`

`Cost`

`Prior`

`Labels`

`Scores`

`Weights`

`rocmetrics`

`rocmetrics: obj = rocmetrics (labels, scores, classnames)`

`rocmetrics: obj = rocmetrics (…, Name, Value)`

`addMetrics`

`rocmetrics: obj = addMetrics (obj, metrics)`

`average`

`rocmetrics: [FPR, TPR, Thresholds, AUC] = average (obj, type)`

`plot`

`rocmetrics: plot (obj)`

`rocmetrics: h = plot (obj)`

`disp`

`rocmetrics: disp (obj)`