Self-Supervised Time Series Representation Learning via Cross Reconstruction Transformer
Wenrui Zhang*{}^{*}start_FLOATSUPERSCRIPT * end_FLOATSUPERSCRIPT, Ling Yang*{}^{*}start_FLOATSUPERSCRIPT * end_FLOATSUPERSCRIPT, Shijia Geng, Shenda Hong##{}^{\#}start_FLOATSUPERSCRIPT # end_FLOATSUPERSCRIPT Wenrui Zhang, Ling Yang and Shenda Hong are with the National Institute of Health Data Science, Peking University, and Institute of Medical Technology, Health Science Center of Peking University, Beijing, 100191, China (e-mail: Wenrui Zhang is also with the Department of Mathematics, National University of Singapore, Singapore, 119077, Singapore.Shijia Geng is with the HeartVoice Medical Technology, Hefei, 230027, China (e-mail: * end_FLOATSUPERSCRIPTEqual Contributions.##{}^{\#}start_FLOATSUPERSCRIPT # end_FLOATSUPERSCRIPTCorresponding author.
Abstract
Since labeled samples are typically scarce in real-world scenarios, self-supervised representation learning in time series is critical. Existing approaches mainly employ the contrastive learning framework, which automatically learns to understand similar and dissimilar data pairs. However, they are constrained by the request for cumbersome sampling policies and prior knowledge of constructing pairs. Also, few works have focused on effectively modeling temporal-spectral correlations to improve the capacity of representations. In this paper, we propose the Cross Reconstruction Transformer (CRT) to solve the aforementioned issues. CRT achieves time series representation learning through a cross-domain dropping-reconstruction task. Specifically, we obtain the frequency domain of the time series via the Fast Fourier Transform and randomly drop certain patches in both time and frequency domains. Dropping is employed to maximally preserve the global context while masking leads to the distribution shift. Then a Transformer architecture is utilized to adequately discover the cross-domain correlations between temporal and spectral information through reconstructing data in both domains, whic
中文速览
标注样本稀缺的现实痛点推动着时间序列自监督表示学习的发展，但现有对比学习方法依赖繁琐的正负样本构造策略，而重建类方法又因将片段直接置零（masking）导致训练与微调阶段的分布偏移，且两类方法都忽视了频域信息。为此，本文提出跨域重建Transformer（Cross Reconstruction Transformer，CRT），核心做法是把时间序列同时映射到时域和频域，并在两个域中随机"丢弃"（dropping，即直接舍去片段而非置零）部分片段，再让Transformer通过跨域重建任务同时恢复时域与频域数据，从而捕捉两个域之间的内在关联；此外还引入实例判别约束（Instance Discrimination Constraint）来压缩不同样本表示间的互信息，并采用课程学习策略在预训练中逐步提高丢弃比例以增强鲁棒性。在多个真实数据集上的实验表明，CRT相比现有最优方法稳定提升2%～9%，证明了同时利用时频信息并以"丢弃"代替"置零"这一设计的有效性，为标注资源匮乏场景下的时间序列表示学习提供了更可靠的解决方案。
原文 arXiv:2205.09928;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2205.09928v2