In addition, the CCI estimate from a reliability study is only an expected value of the true ICC. It makes sense to determine the degree of reliability (i.e. mediocre, moderate, good and excellent) by testing whether the value of the ICC obtained is significantly higher, using statistical conclusions, than the values proposed above. This type of analysis can be easily implemented using SPSS or other statistical software. As part of the reliability analysis, SPSS calculates not only an ICC value, but also its 95% confidence interval. Table 4 shows an example of a background analysis of SPSS. In this hypothetical example, the CCI obtained was calculated by a single scoring model with absolute match and 2-way random effects with 3 advisors in 30 subjects. Although the ICC value obtained is 0.932 (which indicates excellent reliability), its 95% confidence interval is between 0.879 and 0.965, which means that the probability that the real ICC will be at each point between 0.879 and 0.965 is 95%. Therefore, on the basis of statistical conclusions, it would be preferable to conclude the degree of reliability as “good” to “excellent”.” To assess the advisors` agreement, we first calculated two reliable variation indices (RPI), one based on the test reliability of the ELAN manual, the second taking CCI into account for our study population.
Note that while the two reliability indicators can be used to calculate the ROI, they are not equivalent in terms of accuracy and rigor. Test-test correlations represent a very accurate estimate of the reliability of the instrument (compared to a stable construction over time), the reliability of the Interrater reflects rather the accuracy of the evaluation process. The share of the (reliable) agreement was assessed on the basis of the two estimates of honourability to show the impact of the choice of the insurance measure on the evaluation and interpretation of the agreement. In addition to the absolute proportion of compliance, information on the magnitude of the (reliable) differences and on a possible systematic orientation of the differences is also relevant to the full evaluation of the agreement of the raters. Thus, this report takes into account three aspects of agreement: the percentages of ratings that differ reliably, if any, to what extent they differ, and the direction of the difference (i.e. a systematic tendency of the two groups of advisors to react to the other). In the analyses presented here, we also refer to the size of the differences based on factors that may influence the likelihood of divergent assessments in our sample: the sex of the assessed child, the bilingual family environment, and the subgroup of raters. In summary, CCI is a reliability index that reflects both the degree of correlation and the consistency between the measures. It has often been used in conservative care medicine to evaluate interrater, test-retest and intrarater reliability of numerical or continuous measurements. Given that there are 10 different forms of the ICC and that each form contains different hypotheses in their calculations and will give rise to different interpretations, it is important for researchers and readers to understand the principles of selecting an appropriate form of the ICC. Since the CCI estimate from a reliability study is only an expected value of the true CCI, it is preferable to assess the level of reliability based on the 95% confidence interval of the CCI estimate, not the ICC estimate itself.