NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails
Traian Rebedea∗, Razvan Dinu∗, Makesh Sreedhar, Christopher Parisien, Jonathan Cohen NVIDIA Santa Clara, CA {trebedea, rdinu, makeshn, cparisien,
Abstract
NeMo Guardrails is an open-source toolkit111https://github.com/NVIDIA/NeMo-Guardrails for easily adding programmable guardrails to LLM-based conversational systems. Guardrails (or rails for short) are a specific way of controlling the output of an LLM, such as not talking about topics considered harmful, following a predefined dialogue path, using a particular language style, and more. There are several mechanisms that allow LLM providers and developers to add guardrails that are embedded into a specific model at training, e.g. using model alignment. Differently, using a runtime inspired from dialogue management, NeMo Guardrails allows developers to add programmable rails to LLM applications - these are user-defined, independent of the underlying LLM, and interpretable. Our initial results show that the proposed approach can be used with several LLM providers to develop controllable and safe LLM applications using programmable rails.
中文速览
大语言模型（LLM）在实际部署中容易跑偏话题、产生幻觉或被恶意提示词"越狱"，如何在不重新训练模型的前提下灵活管控其行为是一大难题。NeMo Guardrails 提出了一套开源工具包，通过一种名为 Colang 的自定义建模语言，让开发者像写对话剧本一样定义"护栏"规则，再由一个类对话管理器的运行时引擎在推理阶段实时拦截、引导 LLM 的输出，同时配合事实核查、幻觉检测和内容审核等执行型护栏。初步实验表明，该方案可与多家 LLM 供应商无缝配合，在不修改底层模型的情况下有效防止话题跑偏、有害输出和越狱攻击。这项工作的价值在于它把护栏从"训练期嵌入"变为"运行时可编程"，大幅降低了企业将 LLM 安全落地的门槛。
原文 arXiv:2310.10501;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2310.10501v1