Deep Sets
Manzil Zaheer1,2, Satwik Kottur1, Siamak Ravanbhakhsh1, Barnabás Póczos1, Ruslan Salakhutdinov1, Alexander J Smola1,2 1 Carnegie Mellon University 2 Amazon Web Services
Abstract
We study the problem of designing models for machine learning tasks defined on sets. In contrast to traditional approach of operating on fixed dimensional vectors, we consider objective functions defined on sets that are invariant to permutations. Such problems are widespread, ranging from estimation of population statistics [1], to anomaly detection in piezometer data of embankment dams [2], to cosmology [3, 4]. Our main theorem characterizes the permutation invariant functions and provides a family of functions to which any permutation invariant objective function must belong. This family of functions has a special structure which enables us to design a deep network architecture that can operate on sets and which can be deployed on a variety of scenarios including both unsupervised and supervised learning tasks. We also derive the necessary and sufficient conditions for permutation equivariance in deep models. We demonstrate the applicability of our method on population statistic estimation, point cloud classification, set expansion, and outlier detection.
中文速览
传统机器学习模型处理的是固定维度的向量，但现实中大量任务的输入是无序集合（比如一堆点云、一组客户样本），需要模型对元素的排列顺序完全不敏感。本文从理论上证明了一个关键定理：任何对集合具有排列不变性的函数，都可以被分解为"先对每个元素独立做变换φ、再对结果求和、最后经过变换ρ"这种形式，并在此基础上提出了深度网络架构 DeepSets——核心操作就是对集合元素的表示求和后再接非线性变换，同时还给出了排列等变层的充要参数共享条件。实验覆盖了统计量估计、点云分类、集合扩展（concept-set retrieval）和异常点检测等多类任务，DeepSets 均取得了与当前最优方法相当甚至更好的结果。这项工作不仅为"以集合为输入"的机器学习提供了统一的理论基础，还给出了可直接部署的通用网络设计方案，对处理点云、组合推荐、科学数据等无序集合场景具有重要实用价值。
原文 arXiv:1703.06114;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1703.06114v3