St a n z a : A Python Natural Language Processing Toolkit for Many Human Languages
Peng Qi* Yuhao Zhang* Yuhui Zhang Jason Bolton Christopher D. Manning Stanford University Stanford, CA 94305 {pengqi, yuhaozhang, {jebolton,
Abstract
We introduce St a n z a , an open-source Python natural language processing toolkit supporting 66 human languages. Compared to existing widely used toolkits, St a n z a features a language-agnostic fully neural pipeline for text analysis, including tokenization, multi-word token expansion, lemmatization, part-of-speech and morphological feature tagging, dependency parsing, and named entity recognition. We have trained St a n z a on a total of 112 datasets, including the Universal Dependencies treebanks and other multilingual corpora, and show that the same neural architecture generalizes well and achieves competitive performance on all languages tested. Additionally, St a n z a includes a native Python interface to the widely used Java Stanford CoreNLP software, which further extends its functionality to cover other tasks such as coreference resolution and relation extraction. Source code, documentation, and pretrained models for 66 languages are available at https://stanfordnlp.github.io/stanza/.
中文速览
斯坦福大学推出的开源自然语言处理工具包 Stanza，旨在解决现有工具普遍存在的三大痛点：支持语言少、精度不足、无法直接处理原始文本。Stanza 采用全神经网络流水线，从原始文本出发，一站式完成分词、词形还原、词性标注、句法依存分析和命名实体识别等核心任务，同时内置对 Java 版 Stanford CoreNLP 的 Python 接口，进一步支持共指消解和关系抽取。研究团队在通用依存树库等语料上训练了覆盖 66 种语言的 112 个模型，评测结果表明同一套架构在不同语言和文体上均能达到业界领先或持平的水准。这项工作的价值在于，它为全球多语言 NLP 研究提供了一个统一、高精度且易于调用的 Python 工具，大幅降低了非英语语言文本处理的门槛。
原文 arXiv:2003.07082;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2003.07082v2