Understanding HTML with Large Language Models
Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang Aakanksha Chowdhery, Sharan Narang, Noah Fiedel, Aleksandra Faust Google Research {izzeddin,ofirnachum,yingjiemiao,msafdari,austinvhuang
Abstract
Large language models (LLMs) have shown exceptional performance on a variety of natural language tasks. Yet, their capabilities for HTML understanding – i.e., parsing the raw HTML of a webpage, with applications to automation of web-based tasks, crawling, and browser-assisted retrieval – have not been fully explored. We contribute HTML understanding models (fine-tuned LLMs) and an in-depth analysis of their capabilities under three tasks: (i) Semantic Classification of HTML elements, (ii) Description Generation for HTML inputs, and (iii) Autonomous Web Navigation of HTML pages. While previous work has developed dedicated architectures and training procedures for HTML understanding, we show that LLMs pretrained on standard natural language corpora transfer remarkably well to HTML understanding tasks. For instance, fine-tuned LLMs are 12% more accurate at semantic classification compared to models trained exclusively on the task dataset. Moreover, when fine-tuned on data from the MiniWoB benchmark, LLMs successfully complete 50% more tasks using 192x less data compared to the previous best supervised model. Out of the LLMs we evaluate, we show evidence that T5-based models are ideal
中文速览
大型语言模型（LLM）在自然语言处理上表现出色，但它们能否直接理解网页原始 HTML 代码、用于网页自动化操作，此前几乎没有系统研究。研究团队在三个任务上对多种预训练 LLM 进行微调和评测：把 HTML 元素归类到语义类别（如用户名、密码框）、从 HTML 中生成元素的自然语言描述，以及根据自然语言指令自主完成多步网页导航。结果表明，在标准自然语言语料上预训练的 LLM 迁移到 HTML 任务时表现惊人：语义分类准确率比专用模型高 12%，在 MiniWoB 网页导航基准上仅用 192 分之一的数据就多完成了 50% 的任务，其中双向编码器-解码器结构的 T5 系列模型综合表现最佳。这项工作证明无需为 HTML 专门设计网络架构，通用 LLM 微调即可达到甚至超越专用模型的水平，为构建更通用的自主网页代理提供了新的方向。
原文 arXiv:2210.03945;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2210.03945v2