About the miou metric question of background

[BubbleSai]

Is it logically equivalent to 【1】calculate miou (remove background) using a model trained in all categories, and to 【2】calculate miou (ignore background) using only a model trained in foreground categories?

Suppose the dataset has a total of 10 classes: 9 foreground classes + 1 background class.

【1】The definition of the dataset contains 10 classes (i.e. CLASSES=('background', 'class_1', ..., 'class_9')), and the num_classes of decoder heads is 10.

And the model will get the iou result: ['background': 10, 'class_1': 20, 'class_2': 30, ..., 'class_9':100].

The final result is : Result_1 = sum('class_1' + 'class_2' + ... + 'class_9') / 9

【2】The definition of the dataset only contains 9 classes(i.e. CLASSES=('class_1', ..., 'class_9'), and use reduce_zero_label = True both in dataset definition and LoadAnnotations ), and the num_classes of decoder heads is 9.

And the model will get the iou result: ['class_1': 21, 'class_2': 32, ..., 'class_9':92].

The final result is : Result_2 = sum('class_1' + 'class_2' + ... + 'class_9') / 9

If the data set clearly states that the result does not need to calculate the background class, are Result_1 and Result_2 logically equivalent?

[MeowZheng]

No, it is not equivalent, as for the 9 foreground classes, in 【2】the predicated segment will be masked by the ignored area i.e. background in gt, and the predicted segment of 9 foreground classes of 【2】is different from【1】.

[BubbleSai]

No, it is not equivalent, as for the 9 foreground classes, in 【2】the predicated segment will be masked by the ignored area i.e. background in gt, and the predicted segment of 9 foreground classes of 【2】is different from【1】.

But the dataset does not consider the background class, should the prediction results in the background area be ignored?

[MeowZheng]

yes, but the results 【1】and 【2】of other classes expect background are different.

Ignoring background or keeping background depends on your task or your dataset. For cityscapes dataset, we keep the label 0 when evaluation, but for ADE dataset, we ignore the label 0.