arXiv:1711.08536 · 中英对照阅读
No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World
中文速览
用于发展中国家的图像识别模型,可能会因为训练图片主要来自少数国家,而认不准当地的人和事。研究者检查了 ImageNet 和 Open Images 两个常用数据集的图片来源,并用众包图片和按国家收集的网络图片测试它们训练出的模型。结果发现,两个数据集都明显偏向美国和欧洲,而来自中国、印度等地的图片很少;模型对不同国家的图片表现也有明显差异,例如更容易认出美国的新郎,却常把其他地区的新郎认成盔甲或衣物。研究提醒我们,不能只看数据集规模或通用测试成绩,还要检查图片是否覆盖模型将要服务的人群,并在建库时把地域代表性当作重要目标。
摘要
Modern machine learning systems such as image classifiers rely heavily on large scale data sets for training. Such data sets are costly to create, thus in practice a small number of freely available, open source data sets are widely used. We suggest that examining the geo-diversity of open data sets is critical before adopting a data set for use cases in the developing world. We analyze two large, publicly available image data sets to assess geo-diversity and find that these data sets appear to exhibit an observable amerocentric and eurocentric representation bias. Further, we analyze classifiers trained on these data sets to assess the impact of these training distributions and find strong differences in the relative performance on images from different locales. These results emphasize the need to ensure geo-representation when constructing data sets for use in the developing world.
术语表
- machine learning (ML)
- 机器学习(ML)
- image classifier
- 图像分类器
- large-scale data set
- 大规模数据集
- open-source data set
- 开源数据集
- geo-diversity
- 地理多样性
- geo-representation
- 地理代表性
- representation bias
- 代表性偏差
- amerocentric bias
- 美洲中心偏差
- eurocentric bias
- 欧洲中心偏差
- ImageNet
- ImageNet
- Open Images
- Open Images
- Inception V3
- Inception V3
- pretrained model
- 预训练模型
- image classification
- 图像分类
- geo-location
- 地理位置
- country-level geo-location
- 国家级地理位置信息
- proxy information
- 代理信息
- human-verified label
- 人工核验标签
- crowdsourced data
- 众包数据
- stress-test data
- 压力测试数据
- geo-located web images
- 带地理位置标注的网络图像
- geo-dependent mis-classification
- 地理位置相关的误分类
- SmoothGrad
- SmoothGrad
- saliency map
- 显著性图
- standard evaluation data split
- 标准评测数据划分
- log likelihood
- 对数似然
- training distribution
- 训练数据分布
- developing world
- 发展中地区