Diagnostic research: a quick overview
Diagnosis Vs Screening
A diagnostic test is done on sick people
- Patient presents with symptoms
- Pre-test probability of disease is high (i.e. disease prevalence is high)
A screening test is usually done on asymptomatic, apparently healthy people
- Healthy people are encouraged to get screened
- Pre-test probability of disease is low (i.e. disease prevalence is low)
Evaluating a diagnostic test
- Define gold standard
- Recruit consecutive patients in whom the test is indicated (in whom the disease is suspected)
- Perform gold standard and separate diseased and disease free groups
- Perform test on all and classify them as test positives or negatives
- Set up 2 x 2 table and compute:
- Sensitivity
- Specificity
- Predictive values
- Likelihood ratios
Evaluating a diagnostic test
- Diagnostic 2 X 2 table*:
*When test results are not dichotomous, and then can use ROC curves [see later]
Disease + | Disease - | |
---|---|---|
Test + | True Positive | False Positive |
Test - | False Nagative | True Nagative |
Sensitivity: The proportion of patients with disease who test positive, this is known as true positive rate also.
This is the formula we can calculate the true positive rate of sensitivity bases on following:
Specificity: The proportion of patients without disease who test negative, this known as true negative rate also.
The formula of the specificity:
,where TP = True positive, FN = False Negative
Predictive value of a positive test: Proportion of patients with positive tests who have disease
PPT = TP / (TP+FP)
Predictive value of a negative test: Proportion of patients with negative tests who do not have disease
= TN / (TN+FN)
Diagnostic Odds Ratio (DOR): Odds of positive test result in persons with the target condition compared to those without the target condition
DOR = (a/c) / (b/d)
DOR = ad / bc
Example of all measurement calculation through formula using the serological test for TB
Disease Yes | Disease No | Total | ||
---|---|---|---|---|
Serological Test | Positive(+) | 14 | 3 | 17 |
Negative ( - ) | 54 | 28 | 82 | |
Total | 68 | 31 | 99 |
- Sensitivity = TP / (TP+FN) = 14/ (14+54) = 21%
- Specificity = TN / (TN + FP) = 28 / (3+28) = 90%
- PPT = TP / (TP+FP) = 14/ (14+3) = 82%
- PNT = TN / (TN+FN) = 28/ (28+54) = 34%
- DOR = ad/bc = 14*28/3*54 = 392/162 = 2.41
ROC Curve
- A receiver operating characteristic curve, i.e. ROC curve, is a graphical plot that illustrates the diagnostic ability of a binary classifier system as its discrimination threshold is varied.
- The diagnostic performance of a test or the accuracy of a test to discriminate diseased cases from normal cases is evaluated using Receiver Operating Characteristic (ROC) curve analysis
- A Receiver Operating Characteristic (ROC) Curve is a way to compare diagnostic tests. It is a plot of the true positive rate against the false positive rate.
Interpretation of ROC curves
a. A ROC curve of a random classifier
- A classifier with the random performance level always shows a straight line from the origin (0.0, 0.0) to the top right corner (1.0, 1.0).
- Two areas separated by this ROC curve indicate a simple estimation of the performance level. ROC curves in the area with the top left corner (0.0, 1.0) indicate good performance levels, whereas ROC curves in the other area with the bottom right corner (1.0, 0.0) indicate poor performance levels
- A ROC curve represents a classifier with the random performance level. The curve separates the space into two areas for good and poor performance levels.
ROC curve of a perfect classifier
- A classifier with the perfect performance level shows a combination of two straight lines – from the origin (0.0, 0.0) to the top left corner (0.0, 1.0) and further to the top right corner (1.0, 1.0).
- It is important to notice that classifiers with meaningful performance levels usually lie in the area between the random ROC curve (baseline) and the perfect ROC curve.
ROC curves for multiple models
- Comparison of multiple classifiers is usually straight-forward especially when no curves cross each other. Curves close to the perfect ROC curve have a better performance level than the ones closes to the baseline.
- Two ROC curves represent the performance levels of two classifiers A and B. Classifier A clearly outperforms classifier B in this example.
AUC (Area under the ROC curve) score
- Another advantage of using the ROC plot is a single measure called the AUC (area under the ROC curve) score. As the name indicates, it is an area under the curve calculated in the ROC space.
- One of the easy ways to calculate the AUC score is using the trapezoidal rule, which is adding up all trapezoids under the curve.
- The AUC score can be calculated by the trapezoidal rule, which is adding up all trapezoids under the curve. The areas of the three trapezoids 1, 2, 3 are 0.0625, 0.15625, and 0.4375. The AUC score is then 0.65625.
- Although the theoretical range of AUC score is between 0 and 1, the actual scores of meaningful classifiers are greater than 0.5, which is the AUC score of a random classifier.
- It shows four AUC scores. The score is 1.0 for the classifier with the perfect performance level (P) and 0.5 for the classifier with the random performance level (R). ROC curves clearly shows classifier A outperforms classifier B, which is also supported by their AUC scores (0.88 and 0.72).
Tests with continuous results:
ROC curve analysis:
Blood sugar level (2-hour after food) in mg/100 ml |
Sensitivity (%) | Specificity (100%) |
---|---|---|
70 80 90 100 110 120 130 140 150 160 170 180 190 200 |
98.6 97.1 94.3 88.6 85.7 71.4 64.3 57.1 50.0 47.1 42.9 38.6 34.3 27.1 |
8.8 25.5 47.6 69.8 84.1 92.5 96.9 99.4 99.6 99.8 100 100 100 100 |
Note - For each cut-off, we can generate a 2 x 2 table with estimates of sensitivity and specificity
- Area under the curve (AUC) can range from 0.5 (random chance, or no predictive ability; refers to the 45 degree line in the ROC plot) to 1 (perfect discrimination/accuracy.
- The closer the curve follows the left-hand border and then the top-border of the ROC space, the more accurate the test. The closer the curve comes to the 45-degree diagonal of the ROC space, the less accurate the test.
Studies internet links
- http://www.statisticshowto.com/receiver-operating-characteristic-roc-curve/
- http://gim.unmc.edu/dxtests/roc3.htm
- https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Basic_concept
- https://www.medcalc.org/manual/roc-curves.php
- https://classeval.wordpress.com/introduction/introduction-to-the- roc-receiver-operating-characteristics-plot/