Degradation Category Rating (DCR) Methodology

• Degradation Category Rating (DCR) methodology is a double-stimulus protocol that compares a pristine reference with a degraded signal to assess perceptual annoyance.
• It employs a rigorous experimental workflow with structured trial presentation, rater qualification, and randomized design to minimize bias.
• DCR uses statistical aggregation methods like DMOS and ICC to validate its reproducibility, making it ideal for codec comparison and algorithm tuning.

The Degradation Category Rating (DCR) methodology is a paired-stimulus subjective quality assessment protocol originating from ITU-T Recommendations P.800 (speech) and P.910 (video). At its core, DCR requires human raters to judge the perceptual annoyance or severity of degradation by directly comparing a pristine reference signal to its processed counterpart. This fundamental double-stimulus design enables DCR to discriminate fine-grained differences, particularly those missed by single-ended evaluations such as Absolute Category Rating (ACR). In contemporary research, DCR has been adapted for both controlled laboratory studies and large-scale crowdsourcing deployments, with rigorous procedures for rater qualification, trial structure, statistical estimation, and data validation to ensure reliability and reproducibility across a wide range of contexts (Naderi et al., 2022, Naderi et al., 2020).

1. Conceptual Principles and Scale Definitions

DCR is a double-stimulus protocol: each trial presents a reference (clean) signal followed immediately by a degraded (processed) signal. Raters assess the perceptual difference using a five-point degradation category scale:

Category Index	Video Label (P.910)	Speech Label (P.800/P.808)
5	Imperceptible	Inaudible
4	Perceptible but not annoying	Perceptible but not annoying
3	Slightly annoying	Slightly annoying
2	Annoying	Annoying
1	Very annoying	Very annoying

In contrast to ACR’s single-stimulus, unanchored judgments, DCR’s explicit reference anchoring reduces inter-rater and session variability, enhances sensitivity to subtle artifacts, and provides more reliable measurements for tasks such as codec comparison and tuning (Naderi et al., 2022, Naderi et al., 2020).

2. Experimental Workflow and User Interface Design

DCR trials are structured as follows:

• For each trial, the undistorted reference is presented, followed immediately by the processed test signal, both at matched resolution and sample rate (video: spatial/frame-rate; audio: codec/bandwidth).
• Raters are instructed to judge “how annoying the difference is” between the presented pair, with interface cues (radio buttons and/or labeled colored bars) reinforcing both category names and scale mapping.
• Sessions are composed of a fixed number of pairs: in video, typically 20 unique test sequences covering a range of compression or degradation levels; in speech, a random selection of reference–degraded pairs plus one or more “trap” pairs (where the reference and degraded signals are identical).
• To minimize skipping or arbitrary responses, the interface requires full sequential playback and disables submission until a category is selected.
• Orders of pairs and locations of traps or gold-standard items are randomized to reduce positional and expectancy biases (Naderi et al., 2022, Naderi et al., 2020).

3. Rater Qualification, Environmental Gating, and Data Quality Controls

Both lab and crowdsourcing DCR implementations deploy multi-layered quality assurance mechanisms to assure data integrity:

• Hardware, Environment, and Network Validations (Video): Enforcement of minimum screen size (≥1280×720), browser compatibility (latest Chrome/Firefox), minimum CPU and network bandwidth, and mid-tone ambient lighting verified by webcam-derived histograms.
• Audio Hardware and Environment Checks (Speech): Adaptive tone-detection (Levitt, 1992), four-item just-noticeable-difference test, WebRTC-based headset enumeration (flagging two-ear headsets as more reliable).
• Trap and Gold-Standard Questions: Each session injects at least one “trap” where reference and test are identical; any rating <5 triggers suspicion or rejection. Gold-standard pairs, anchored to laboratory means, assess rater fidelity; large deviations result in disqualification.
• Rating Pattern Analysis: Flat-line responses, failure to engage playback, or rating all pairs identically are flagged and filtered.
• Raters failing more than one gold or trap question are disqualified, typically eliminating <5% of participants but substantially reducing label noise.
• Temporal Environment Certificates (Speech): After passing JND, raters are certified for 30 minutes to reduce repeated qualification latency (Naderi et al., 2022, Naderi et al., 2020).

4. Data Aggregation, Statistical Estimation, and Scaling

DCR raw ratings are numerically mapped ($s_{i, j} \in \{1, 2, 3, 4, 5\}$), with aggregation producing Degradation Mean Opinion Scores (DMOS):

$\text{DMOS}_j = \frac{1}{N} \sum_{i=1}^N s_{i, j}$

Lower DMOS values indicate greater perceived degradation. For each clip $j$, error is quantified as:

$$\text{SE}(\text{DMOS}_j) = \frac{\sigma_j}{\sqrt{N}}, \quad \sigma_j^2 = \frac{1}{N-1}\sum_{i=1}^N (s_{i, j} - \text{DMOS}_j)^2$$

$$\text{CI}_{95}(j) = \text{DMOS}_j \pm 1.96 \frac{\sigma_j}{\sqrt{N}}$$

A session normalization can be optionally applied:

$\tilde{s}_{i,j} = \frac{s_{i,j} - \mu_k}{\sigma_k}$

where $\mu_k$, $\sigma_k$ are session mean and SD, and $S_k$ is the set of trials for session $k$. This step is rarely necessary with robust qualification but may mitigate session-wise bias (Naderi et al., 2022).

Post-processing scripts also compute intra-class correlation coefficients (ICC) for reproducibility and can output diagnostic logs, bonus assignments, and per-worker accept/reject summaries (Naderi et al., 2022, Naderi et al., 2020).

5. Validation against Laboratory Standards and Reproducibility Metrics

DCR-based crowdsourcing protocols have been assessed against formal lab implementations to verify both accuracy and stability:

• Video Quality (P.910 adaptation): Pearson correlation of 0.93 and Spearman rank correlation of 0.91 between crowdsourced DMOS and gold-standard lab DMOS (across 50 reference–test pairs; $p < 0.001$ for both). The paired DMOS difference (<0.05 on a 5-point scale, $p = 0.12$) suggests statistical equivalence. The typical absolute deviation (0.08 DMOS points) is well below the perceptual just-noticeable-difference threshold (0.2 in P.910). Within-session split-half ICCs reach 0.89; across independent pools, ICC = 0.87 (Naderi et al., 2022).
• Speech Quality (P.808 adaptation): Five independent crowdsourced DCR runs on the INTERSPEECH 2020 corpus (≈90 unique raters per run, 5 votes per file) produced average Pearson correlation PCC = 0.994 and SRCC = 0.94. The two-way random-effects ICC for absolute agreement was ICC = 0.907 (“excellent” per P.1401 benchmarks). RMSE against lab MOS was ±0.24 points (Naderi et al., 2020).

These results indicate crowdsourced DCR, with rigorous gating and statistical filtering, can match lab studies in both absolute and relative measures of perceptual quality.

6. Implementation Enhancements and Methodological Variants

Several enhancements have improved DCR’s practical deployment:

• Monolithic HIT Design (Speech): Collapsing eligibility tests into the rating task reduces latency by 4–5× compared to two-stage approaches.
• Temporal Qualification: Storing session “certificates” enables raters to bypass repeated environmental validation, reducing attrition and test time.
• Automated Data Cleansing: End-to-end scripts for computing DMOS, confidence intervals, ICCs, and rater diagnostic logs safeguard against operational errors and facilitate integration into continuous evaluation pipelines (Naderi et al., 2020).
• Support for Comparison Category Rating (CCR): Some systems implement CCR alongside DCR, providing expanded options for paired-comparison protocols (Naderi et al., 2020).

A plausible implication is that these efficiencies make DCR feasible for large-scale evaluation or iterative development cycles (e.g., codec or enhancement algorithm optimization) at a fraction of lab-based cost and logistics.

7. Domain Applications and Limitations

DCR is widely adopted for scenarios demanding fine discrimination in perceptual quality—most notably, codec comparison, parameter tuning, and benchmarking of new compression or enhancement algorithms in both speech and video domains. Its sensitivity to subtle artifacts and robustness to rater and session variability position it as the standard for subjective paired-comparison testing.

One limitation is the need for pristine references, restricting DCR’s applicability to scenarios where clean source material is available. Additionally, absolute scales may require mapping or normalization across studies for meta-analyses. Nevertheless, as demonstrated by Naderi, Cutler, and others, DCR-based crowdsourcing, when governed by ITU-standard methodology and rigorous validation, achieves reproducibility and accuracy on par with formal experiments, enabling its use as a reliable surrogate for traditional laboratory subjective tests (Naderi et al., 2022, Naderi et al., 2020).