Sorting Framework

We used the following criteria to sort the indices by accuracy (calibration and discrimination), generalizability, bias, usability, and clinical efficacy. Please note that this framework was developed by the authors of ePrognosis. In this sense, it represents expert opinion. It has not undergone peer review and should be viewed a preliminary starting point. Users may weigh certain factors differently than us in sorting prognostic indices.

Excellent: Prospectively tested in a clinical setting demonstrating changes in clinician behavior with beneficial patient outcomes (a measure of clinical efficacy).

Very Good: Maintains accuracy when tested in a national sample or in heterogeneous populations.

• observed mortality range 50% or greater (a measure of usability) 10% points or fewer between the predicted mortality rate and the observed mortality rate (a measure of calibration)
• c-statistic of 0.7 or greater (a measure of discrimination)
• used data other than ICD-9 codes (a measure of potential bias)

Good: Accuracy suffers when tested in heterogeneous settings, or usability is limited.

• observed mortality range 30% or greater (a measure of usability) 15% points or fewer between the predicted mortality rate and the observed mortality rate (a measure of calibration)
• c-statistic of 0.65 or greater (a measure of discrimination)
• used data other than ICD-9 codes (a measure of potential bias)

Moderate: Does not meet above "good" criteria, but at a minimum tested some form of reproducibility or transportability (a validation study was conducted).

Use/Usefulness

We use the following algorithm to determine user-rated use and usefulness of these indices. The index's "bubble" size corresponds to the following factors:

• Use: number of times an index is selected ("clicked") by users
• Usefulness: a user rating that the index was "useful" (responses can be submitted on the results page for each index). A "not useful" rating contributes negatively to the use/usefulness calculation.
• Usefulness is weighted more heavily than use.