RLChina 2023 WORKSHOP

Recent studies have discovered that Chain-of-Thought prompting (CoT) can dramatically improve the performance of Large Language Models (LLMs), particularly when dealing with complex tasks involving mathematics or reasoning. Despite the enormous empirical success, the underlying mechanisms behind CoT and how it unlocks the potential of LLMs remain elusive. In this talk, I will take a first step towards theoretically answering these questions. Specifically, I will examine the expressivity of LLMs with CoT in solving fundamental mathematical and decision-making problems. I will start by giving an impossibility result showing that bounded-depth Transformers are unable to directly produce correct answers for basic arithmetic/equation tasks unless the model size grows super-polynomially with respect to the input length. In contrast, autoregressive Transformers of constant size suffice to solve both tasks by generating CoT derivations using a commonly-used math language format. Moreover, I will show LLMs with CoT are capable of solving a general class of decision-making problems known as Dynamic Programming, thus justifying its power in tackling complex real-world tasks.

With infinitely many high-quality data points, infinite computational power, an infinitely large foundation model with a perfect training algorithm and guaranteed zero generalization error on the pretext task, can the model be used for everything? This question cannot be answered by the existing theory of representation, optimization or generalization, because the issues they mainly investigate are assumed to be nonexistent here. In this talk, we show that category theory provides powerful machinery to answer this question. We have proved three results. The first one limits the power of prompt-based learning, saying that the model can solve a downstream task with prompts if and only if the task is representable. The second one says fine tuning does not have this limit, as a foundation model with the minimum required power (up to symmetry) can theoretically solve downstream tasks for the category defined by pretext task, with fine tuning and enough resources. Our final result can be seen as a new type of generalization theorem, showing that the foundation model can generate unseen objects from the target category (e.g., images) using the structural information from the source category (e.g., texts). As the simplest case of our framework, we will demonstrate that SimCLR is doing spectral clustering on similarity graphs, which exploits RHKS as a special version of Yoneda lemma.

This talk delves into the capabilities of large language models (LLMs), specifically focusing on advancing the theoretical comprehension of chain-of-thought prompting. We investigate how LLMs can be effectively induced to generate a coherent chain of thoughts. To achieve this, we in- troduce a two-level hierarchical graphical model tailored for natural language generation. Within this framework, we establish a compelling geomet- rical convergence rate that gauges the likelihood of an LLM-generated chain of thoughts compared to those originating from the true language. Our findings provide a theoretical justification for the ability of LLMs to produce the correct sequence of thoughts (potentially) explaining performance gains in tasks demanding reasoning skills.

One way wonder what is “reality”, how to grasp elements of reality and how to describe them. The purpose of the talk will be to propose mathematical models of “reality” and of possible descriptions of elements of reality, based on the theory of Grothendieck toposes and its representations. This model suggests principles for designing mathematical languages adapted to describe some aspects of the world, such as for instance images.

We shall propose a research programme aimed at integrating syntactic structures into artificial learning systems. This, besides greatly reducing the amount of ‘brute-force’ computations, should lead to a significantly higher resilience, robustness and adaptability of the resulting systems. The unifying role of toposes as ‘bridges’ between geometric presentations and logical descriptions of data will be crucial in connection with this programme.

Compared to humans and animals, “intelligent” machines are very slow to learn, require a large amount of data, do not understand what they are doing, make stupid mistakes, do not reason have no common sense. Is there a way to overcome them? All these issues are related to the absence of semantics in nowadays machine learning. Is it possible, for a new architecture of machine to learn as quickly as a baby? With very few data? To “understand”? To reason? We aim at giving some answers to these questions by developing a theoretical point of view and applying it to experiments to show the effectiveness of our theory. This theory is based on a mathematical notion proposed by Alexander Grothendieck in the 60’s, the notion of topos. I will present how this notion helped us to develop a new theory of semantic information, how a generalized notion of equivariance just corresponds, in the right topos, to regular functions, how it is possible to learn by concepts and approach the way a baby can learn and finally, how toposes are related to logics. All these aspects will be presented both theoretically and experimentally. 

In the classical reinforcement learning (RL) formulation, reward is often the only performance metric for an algorithm. Accordingly, existing RL literature primarily focuses on developing algorithms that can achieve high rewards while rarely considering which solution is derived. However, we will show that (optimal) policies with the same reward can yield substantially different behaviors in popular RL testbeds. Moreover, many of the (optimal) behaviors can be hardly discovered by classical RL algorithms that only strive for rewards. In this talk, we present recent advances in developing diversity-driven RL algorithms, which not only optimize rewards but also aim to discover a broad spectrum of policies with visually distinct behaviors.

Training deep reinforcement learning (DRL) models usually requires high computation costs. Therefore, compressing DRL models possesses immense potential for training acceleration and model deployment. However, existing methods that generate small models mainly adopt the knowledge distillation-based approach by iteratively training a dense network. As a result, the training process still demands massive computing resources. Indeed, sparse training from scratch in DRL has not been well explored and is particularly challenging due to non-stationarity in bootstrap training. In this work, we propose a novel sparse DRL training framework, “the Rigged Reinforcement Learning Lottery” (RLx2), which builds upon gradient-based topology evolution and is capable of training a DRL model based entirely on sparse networks. Specifically, RLx2 introduces a novel delayed multistep TD target mechanism with a dynamic-capacity replay buffer to achieve robust value learning and efficient topology exploration in sparse models. It also reaches state-of-the-art sparse training performance in several tasks, showing 7.5×-20× model compression with less than 3% performance degradation and up to 20× and 50× FLOPs reduction for training and inference, respectively.

创造性是人类智慧区别于其他个体的关键特性之一；人工智能作为比拟人类智慧的科学技术，应具有创造性。本报告拟从智能规划的视角解释人工智能的创造性，并探讨如何基于智能规划实现人工智能的创造性，最后介绍基于智能规划实现创造性的案例。

强化学习是构建智能决策Agent的基石。然而，现有强化学习算法在样本和计算上不够高效，导致它们难以在真实世界落地应用。真实世界中的观察往往包含高价值的信息，有效驾驭它们是智能决策的关键。在本报告中，我们把智能决策形式化为有界最优的终身强化学习问题，并探讨智能决策Agent的设计及三方面的挑战：针对非独立同分布数据流的在线学习、使用有限资源的高效推理及探索-利用困境，最后围绕这些挑战介绍我们近期提出的一些深度强化学习算法。

本次报告我将讨论强化学习在短视频推荐系统应用的动机、挑战以及解决方案。首先我将介绍短视频推荐系统强化学习在线搜参算法，然后我将介绍应对短视频推荐落地强化学习三个方面挑战的3篇工作：多目标优化、大规模动作空间、留存优化。

以人为中心的推荐系统（Human-Centered Recommender Systems）强调分析和理解用户与推荐系统的特质以及两者间的复杂关联关系，聚焦解释和评估推荐系统对用户期望、价值与目标的影响、塑形与风险。大模型驱动的Agent建模具有系统演化过程可知、自由社会实验可控等特点，给以人为中心的推荐系统模拟提供了独特视角和全新思路，是深度探究伴随推荐系统涌现的曝光偏差、过滤气泡、回声壁、信息茧房等核心问题的利器。本报告将分享研究团队在大模型驱动的Agent建模推荐系统的最新工作，围绕以人为中心的推荐系统设计理念，所提出的基于大模型的微观模拟框架RHA!，从个体层面细粒度模拟了Recommender- Assistant- Human之间复杂互动关系与协同交互过程；所提出的预训练大模型驱动的宏观模拟工具SimuLine，从群体层面全生命周期模拟了Creator- Recommender- Consumer生态系统的复杂作用机理与动态演化过程。这些创新工作展示了其在未来用户体验评估、算法治理调控和网络监管实施等方面应用的巨大潜力。

随着预训练技术的不断发展，借助思维链等提示学习技术，大模型展现出了令人惊讶的推理能力，引起了学术界和工业界学者的广泛关注。然而，思维链方法在处理复杂推理、规划等任务时仍遇到了不少挑战。为了解决这些问题，研究人员开始探索基于大模型的自主智能体系统。本次报告将从基于提示学习的大模型思维链技术出发，介绍大模型智能体的自主规划学习技术，包含自主规划推理中的共性原理、基于执行反馈的工具学习新方法，以及相关的开源工具框架。

社会模拟是社会科学研究的基础工具，大模型智能体为社会模拟注入了新活力。大模型凭借其类人感知能力与决策过程，一方面能够对社会个体实现准确模拟，另一方面能够通过多智能体交互对真实社会宏观态势进行仿真复现。本次报告将围绕社会模拟，介绍基于大模型智能体的社交模拟系统S3与经济模拟系统两部分工作。在社交模拟方面，针对给定信息环境下对用户产生的交互行为、情绪变化、观点变化，基于大模型智能体实现对个体行为和认知状态的准确模拟；同时在宏观层面上，模拟出的信息传播趋势和情绪观点状态宏观变化趋势符合真实数据规律。在经济模拟方面，所构建的基于大模型智能体的经济模拟系统，与传统宏观经济模拟方法相比，展现出更强的稳定性和真实性，初步探索了宏观经济模拟的新型模拟范式。

近年来，大模型在自然语言处理、计算机视觉等诸多领域展现出惊人的应用价值。大模型通过在大规模无监督数据上进行预训练，在复杂交互环境中展现出了非凡的理解、推理与决策能力。然而，现有大模型在实际应用中的功能却主要局限于对用户输入的文本反馈，缺乏对环境的深入理解和主动交互能力、深层次的任务执行能力。本报告的内容为大模型工具学习，介绍大模型工具学习智能体如何理解任务，对给定任务目标进行分解，并调用各种工具来完成任务，包括其统一框架、主要挑战和未来方向。

随着大型语言模型的出现，关于是否会出现通用代理（generalist agent）的辩论重新兴起。这可能是一个过于雄心勃勃的目标。然而，为了取得任何进展，我们需要一个合适的测试环境和严格的评估协议。根据我们以往的研究结果，我们注意到以前用于测试智能体的环境大多被设计成只有一个具体的任务和目标（有时由一个特定奖励函数来指定）。这极大地限制了我们评估是否在构建通用代理方面取得了重大进展的能力。在这次分享中，我们将介绍我们组以及其他一些相关的知名研究实验室的各种在使用开放世界环境（例如Minecraft）来开放通用代理远行的努力。我们将深入探讨为什么现在是时候瞄准开放世界环境；这些环境的特点是什么；它们面临的独特挑战是什么，以及为什么解决这些挑战对于通用代理是不可或缺的；最后，我们将看一下这一领域的最新研究成果并对未来做一些展望。

以大模型为代表的最新人工智能技术成为引领世界新一代产业变革的核心驱动力，将极大推动智能机器的社会化应用，也为强调身心一体的具身智能研究带来了广阔的发展空间。然而，当前的具身智能体，虽然建立在神经网络强大的表达和学习能力基础之上，却面临着缺乏知识推理、难以与人沟通交互、无法解释决策机理等瓶颈。在人工智能领域新变革与大模型时代的背景之下，本次讲座围绕知识与大模型驱动的全新具身智能范式，介绍本团队近期开展的研究工作，介绍如何将人类知识以及大模型与具身智能体结合，提高具身智能体在推理、规划、决策等方面的智能水平，以及与人类沟通、协作的能力。

最近，以ChatGPT为代表的大语言模型受到了社会的广泛关注。本次报告将围绕大语言模型研发过程中的关键技术展开介绍，将从预训练、指令微调、轻量化部署等方面展开讨论，分析现有技术特点以及局限之处，并且给出报告人对于这些方面的相关思考与实践经验。

随着ChatGPT在人工智能领域崭露头角，大语言模型展现出卓越的人机交互和指令执行能力，为多个领域带来了新的机遇。然而，在推荐系统中，大语言模型的推荐准确度和用户满意度存在一些挑战。一方面，由于大语言模型的训练数据缺乏特定领域的物品信息和用户互动数据，它对领域内的模式和知识的理解相对不足，导致了较低的推荐精准度。另一方面，对于新加入系统的物品，大语言模型的更新和适应成本也难以承受。为了解决这些问题，我们提出了一种融合大语言模型与推荐领域工具的方法，构建了一种交互式的推荐智能体。其中，我们利用大语言模型的卓越指令执行、人机交互和工具使用能力，将其作为智能体的核心，通过在对话中理解和解析用户意图，制定工具使用计划，并结合个性化能力更强、准确度更高的推荐工具为用户提供推荐列表，从而综合提升了交互体验和推荐准确度。

大语言模型具备极强的生成能力，通过思维链等技术也能完成简单的决策问题。但是设计最优提示需要经验，而且难以泛化到复杂场景。为此，我们提出了动态存储增强的决策大模型，利用记忆模块来构建动态的提示，以指导模型做出正确决策。动态存储增强的决策大模型由记忆形成和记忆强化两个模块构成。记忆形成模型将大量的人类轨迹分解为状态-动作元组并储存在记忆模块中，同时构建索引。记忆强化模型采用树探索来寻找最优决策过程，并添加其中有价值的状态-动作元组到记忆形成模块中来实现整个模型的优化。在两个交互式环境中进行的实验表明我们提出的模型优于基准模型。

在游戏领域，文本游戏正在经历一场复兴，为玩家提供了独特而身临其境的游戏体验。本讨论关注如何将自然语言处理技术，尤其是语言智能体，融入文本游戏中，以使其表现得像人一样参与游戏。我们深入研究了语言智能体在知识表达、推理、决策等方面的运用，力求实现更强大的智能体。此外，我们还探讨了在游戏中应用语言智能体涉及的伦理问题，特别关注了与道德相关的潜在影响。我们讨论了缓解这些挑战以及在游戏中负责任地运用人工智能的策略。通过这些探讨，我们致力于推动游戏领域中语言智能体的发展，以提升智能体的语言理解、推理和决策能力。

This talk is dedicated to exploring responsible AI methods in decision and control systems through the lens of Implicit Networks. Implicit Networks, encompassing optimization layers, NeuralODEs, and deep equilibrium models, play a pivotal role in advancing reliability, interpretability, and memory efficiency in AI applications. The talk initiates with an exploration of differentiable optimization, elucidating how this approach contributes to enhancing the reliability of decision and control systems. The subsequent portion of the tutorial concentrates on memory-efficient neural network models, specifically NeuralODEs and deep equilibrium models. Attendees will delve into these advanced architectures, comprehending how they enable the modeling of temporal dependencies while maintaining memory efficiency. In this talk, we will also present our latest contributions to the field of responsible AI for decision and control, unveiling novel techniques that push the boundaries of reliability and efficiency.

Recent years have witnessed the great success of RL with deep feature representations in many challenging tasks, including computer games, robotics, smart city, and so on. Yet, solely focusing on the optimal solution based on a reward proxy and learning the reward-maximizing policy is not enough. Diversity of the generated states is desirable in a wide range of important practical scenarios such as drug discovery, recommender systems, dialogue systems, etc. For example, in molecule generation, the reward function used in in-silico simulations can be uncertain and imperfect itself (compared to the more expensive in-vivo experiments). Therefore, it is not sufficient to only search for the solution that maximizes the return. Instead, it is desired that we sample many high-reward candidates, which can be achieved by sampling them proportionally to the reward of each terminal state. The Generative Flow Network (GFlowNet) is a probabilistic framework proposed by Yoshua Bengio in 2021 where an agent learns a stochastic policy for object generation, such that the probability of generating an object is proportional to a given reward function, i.e., by learning a reward-matching policy. Its effectiveness has been shown in discovering high-quality and diverse solutions in molecule generation, biological sequence design, etc. The talk concerns my recent research works about how we tackle three important challenges in such decision-making systems. Firstly, how can we ensure a robust learning behavior and value estimation of the agent? Secondly, how can we improve its learning efficiency? Thirdly, how to successfully apply them in important practical applications such as computational sustainability problems and drug discovery?

大语言模型应用开发框架是一个灵活的方法，可以根据不同应用的需求进行定制，有助于开发者更好地利用大语言模型的潜力，同时考虑到伦理、可解释性和隐私等重要因素。本次分享探讨大语言模型在应用开发领域的潜在价值和挑战，并介绍了一种旨在提高可解释性的智能体设计方法。然后介绍CangChain（苍穹）开源项目在框架智能化、类似人类推理、隐私保护等领域的探索工作。

介绍深度强化学习（DRL）、基于Transformer的强化学习（TRL）、基于大语言模型（LLM-based）的智能体和多智能体等一系列关于AI Agent的探索工作，并分享一些研究心得体会。

大语言模型(LLMs)目前已展现出推理、规划、工具使用等诸多类人智能，可作为智能体(Agent)的大脑自动化地处理各种复杂任务。然而这些大语言模型智能体是否能够像人类一样学会沟通与分工，更快更好地进行任务协作，仍然是一个亟待探索的问题。本报告将介绍大语言模型智能体协作研究的最新进展，提出一种基于“招募-讨论-执行-反馈”模式的通用合作框架AgentVerse，并重点分析实验中发现的各类智能体合作涌现行为。

Large language models (LLMs) have enabled remarkable advances in automated task-solving with multi-agent systems. However, most existing LLM-based multi-agent approaches rely on predefined agents to handle simple tasks, limiting the adaptability of multi-agent collaboration to different scenarios. This talk will introduce AutoAgents, an innovative framework that adaptively generates and coordinates multiple specialized agents to build an AI team according to different tasks. Specifically, AutoAgents couples the relationship between tasks and roles by dynamically generating multiple required agents based on task content and planning solutions for the current task based on the generated expert agents. Experiments on various benchmarks demonstrate that AutoAgents generates more coherent and accurate solutions than the existing multi-agent methods. This underscores the significance of assigning different roles to different tasks and of team cooperation, offering new perspectives for tackling complex tasks.

Offline reinforcement learning has emerged as a promising framework for learning sequential policies from pre-collected datasets. It often assumes that offline datasets include rewards. In reality, reward-free data, like experiences from different tasks or from human players, is abundant and contains rich behavioral information. In this talk, we explore two questions: 1) How can we leverage reward-free data to improve the performance of offline RL algorithms in a principled way? 2) How can we effectively utilize the behavioral information in unsupervised offline data for rapid online learning? For the first question, we first investigate the theoretical benefits of utilizing reward-free data in linear Markov Decision Processes (MDPs) within a semi-supervised setting. We then present a novel, Provable Data Sharing algorithm (PDS) to utilize such reward-free data and enhance offline RL. For the second question, we propose an unsupervised approach to extract useful behaviors from offline reward-free datasets via diversified rewards and reuse them as candidate policies to facilitate the learning of new tasks. Perhaps surprisingly, we show that rewards generated from random neural networks are sufficient to extract diverse and useful behaviors, some even close to expert ones.