Tom Schaul Google DeepMind London, UK [email protected]

Annotated by @Hrishi Olickel

Abstract

An agent trained within a closed system can master any desired capability, as long as the following three conditions hold: (a) it receives sufficiently informative and aligned feedback, (b) its coverage of experience/data is broad enough, and (c) it has sufficient capacity and resource. In this position paper, we justify these conditions, and consider what limitations arise from (a) and (b) in closed systems, when assuming that (c) is not a bottleneck. Considering the special case of agents with matching input and output spaces (namely, language), we argue that such pure recursive self-improvement, dubbed 'Socratic learning,' can boost performance vastly beyond what is present in its initial data or knowledge, and is only limited by time, as well as gradual misalignment concerns. Furthermore, we propose a constructive framework to implement it, based on the notion of language games.

1 Introduction

On the path between now and artificial superhuman intelligence (ASI; Morris et al., 2023; Grace et al., 2024) lies a tipping point, namely when the bulk of a system's improvement in capabilities is driven by itself instead of human sources of data, labels, or preferences (which can only scale so far). As yet, few systems exhibit such recursive self-improvement, so now is a prudent time to discuss and characterize what it is, and what it entails.

We focus on one end of the spectrum, the clearest but not the most practical one, namely pure self-contained settings of 'Socratic' learning, closed systems without the option to collect new information from the external world. We articulate conditions, pitfalls and upper limits, as well as a concrete path towards building such systems, based on the notion of language games.

The central aim of this position paper is to clarify terminology and frame the discussion, with an emphasis on the long run. It is not to propose new algorithms, nor survey past literature; we pay no heed to near-term feasibility or constraints. We start with a flexible and general framing, and refine and instantiate these definitions over the course of the paper.

Definitions

Consider a closed system (no inputs, no outputs) that evolves over time (see Figure 1 for an illustration). Within the system is an entity with inputs and outputs, called agent, that also changes over time. External to the system is an observer whose purpose is to assess the performance of the agent. If performance keeps increasing, we call this system-observer pair an improvement process. The dynamics of this process are driven by both the agent and its surrounding system, but setting clear agent boundaries is required to make evaluation well-defined: in fact an agent is what can be unambiguously evaluated. Similarly, for separation of concerns, the observer is deliberately located outside of the system: As the system is closed, the observer's assessment cannot feed back into the system. Hence, the agent's learning feedback must come from system-internal proxies such as losses, reward functions, preference data, or critics.

The simplest type of performance metric is a scalar score that can be measured in finite time, that is, on (an aggregation of) episodic tasks. Mechanistically, the observer can measure performance in two ways, by passively observing the agent’s behaviour within the system (if all pertinent tasks occur naturally), or by copy-and-probe evaluations where it confronts a cloned copy of the agent with interactive tasks of its choosing.

data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAe0AAADdCAIAAAAZ2721AABQG0lEQVR4nO3dd0BTV9sA8HNv9iQkEEYIe29QFEFQoQ4cra2vbW3f1q6v1W5bte3b+XbX7r2sHepb29pa66iDKk5Q2bJlEzYEyJ73fH/EIqJFCEkugfP7i4STex9Gnpx77jnPwSCEAEEQBHFaONkBIAiCIOOC8jiCIIhzQ3kcQRDEuaE8jiAI4txQHkcQBHFuKI8jCII4N5THEQRBnBvK4wiCIM4N5XEEQRDnhvI4giCIc0N5HEEQxLmhPI4gCOLcUB5HEARxbiiPIwiCODeUxxEyPfroo19++aUND7h58+aHH37YJociCGL79u0Gg8EmR0MQ+0F5HCFTb2+vQqGw4QGVSmVPT49NDtXW1nb//fcTBGGToyGI/VDJDgCZQiCEOTk5OTk5Li4u//73v8Vi8dBvHT16NCcnh8/n33LLLVKpFADQ3d29ffv27u7uuLi4G2+8kUajAQBqa2t37typ1+uXLVuWmJhoeW1BQcG+ffu8vb2v2n0uLS3dvXs3QRDz58+fNWtWZWVlbW3t9ddfb/nugQMH/P39AwMDf/7554qKColEsmrVKldX1127dplMpi1btqxevZrNZp86derw4cM8Hm/VqlUSiQQAsGPHjtTU1J07d6rV6jvuuINKpf7www8UCuWee+5xc3NzwO8TQSxQfxxxnLfeeuuBBx5wd3evqqpKS0tTqVSD33rttdfuv/9+kUjU0dExffr08+fPGwyGuXPnNjQ0SKXSTZs2bdy4EQCQm5ubmpqq0+l4PN7y5ct/++03AMCff/65dOlSNptdXV29adOmYSctKytbuHAhk8nkcrnLly8/cOAAAOD222/v6+sDAGg0mjvuuINCoaxbt27r1q3+/v65ubnz5s0zmUwDAwMQQrlcThDEp59+euedd4pEIrlcPnPmzNraWgDAf/7zn2XLlikUiuLi4uuuu+62224DAOzfv3/16tWO+o0iCAAAAIggDqFWq3k8XlFREYSQIIj169dXVVXddtttmzZt6uvrY7FYpaWllpbr16+/5ZZbOjs7aTRaWVkZQRD19fX79u2DEM6fP/+NN96wNPvtt98iIyMJgkhJSdm8ebPlyXvvvfeWW24Zet4ffvghISGhr68PQnj8+PGSkhIIYXJy8pYtWyCEv/zyy+zZswmCSExM/OSTT8xms8Fg+O6777RabUtLC5vN1mq1arXa1dX11KlTlgM+/vjja9asgRAGBAR8+umnEMLe3l4Mw44fPw4hzM/Pd3V1te+vEkEuh/rjiIM0NjYCAKKjowEAGIa9/fbbYWFhlm9duHCBzWZHRkZaHqalpZWWlrq7u69du3batGnx8fFff/11VFSU2WwuKSnZvn37nDlz5syZ88Ybb1RWVmo0moqKiqSkJMtrU1JShp136dKlTCbT29s7MzOzpKQkODgYAHDXXXdt374dALB169bVq1djGPbUU089++yzfn5+a9euDQ4OZjAYg0doaGjo6+vbuHGj5bz79u0rKSmxfCsiIgIAwGKxaDRaYGAgAIDD4ZjNZjv9DhHkqtD4OOIgOI6bzWaTyUSlUgEAFy5cGBxEptPpBoMB/r3lt06no9PpGIZ98MEHTz311OHDh3/88cf09PTq6moqlbphw4YZM2YMHpbBYNBoNL1eb3l4ZQ51dXU9efJkeXn54cOHP/7444KCgm+//XblypUbN24sKys7efLkd999BwC4+eabs7Kyjh07tn///szMzFOnTnl4eFiOQKVSaTTap59+OpjcmUym5QvLkP3gD2jL3xeCjBr6z0McJDAwUCAQZGdnAwC0Wu3ixYtPnz5t+VZYWBibzd67dy8AwGw279ixIy0traKiIjQ0lMvlrl69+vPPP29ra9PpdGlpaSdOnAgLCwsPDz99+vSDDz6I43hGRsZPP/0EISQIYvfu3cPO+/nnn69atSo6OvqJJ554/PHH6+rqAABCoXDp0qUPP/zwokWLXF1dTSZTfHx8QUHB0qVLP/nkE39//+bmZgzDCIIwmUwBAQFeXl4lJSXh4eFhYWFvv/32119/7dhfHoKMhMz+uMFgkMlkHR0dAwMDWq0WQojjOIfDEQqF3t7enp6e1+zgQAjVanVLS0tnZ6dCoTCZTBiGMRgMgUDg6ekpkUiGXh0j5KLT6V999dV9992XmppaVVWVlJS0aNGi//3vfwAAJpP5zTff3HvvvT/88ENbWxuFQvn6669dXV2nTZs2bdq06dOnnz17duPGjS4uLps2bcrKykpPTxcKhbm5uTt27MBx/M0331y4cOH58+fVarVer+dyuUPPu3z58vfffz8tLc3Ly+vUqVPffPON5fnVq1cvXLjQctuTSqWuWbNm5cqV8+bNa2lp8fDwWLhwIY7jQqEwNjb24MGDmzdvvuOOO3bu3Nnf39/V1WX5NEKQCQIbvJh1GAhhT0/P+fPnFQqFj4+PWCwWCoVMJpNKpRIEoVKpuru729raurq6fHx8oqOjORzOVQ/S2dlZVFRkNpv9/Pw8PT1dXFxoNBqGYQaDYWBgoK2trbm5mc/nx8bGurq6OvhnRP5JX19fYWGhu7t7TEwMhmEdHR0MBsPyB+rv7y8qKnJ1dY2NjbV8hEMIy8vL29vbg4OD/f39MQwDABiNxsLCQrVanZiYKBAILIfV6/Vnz54ViUQ+Pj4ajcbT03PoSfV6fUFBgU6nS0hIGPxnqKioyMrKqq2tHRwbaW1traiocHNzi4uLswTQ19dXX18fERHBZrMVCkVhYSGdTp8+fTqdTgcANDY2enh4sFgsgiAaGhr8/PyoVKqld2IZK0cQx3B0HtdqtXl5eQaDIT4+XiwWW96ZV2U2mxsaGs6fP+/v7x8bG0uhUAa/pdPpLJMHpk2bJhAI/ukgEMKurq6CggJ3d/fExMShRxiB5f4vAADDsBHCQ5yXyWTas2fPl19+mZaW9uyzz5IdDoKMl0PzeHt7++nTp6dPn+7r6zvKFGk2m0tLS2UyWUZGhqVjLpfLc3Jypk2b5ufnN5ojQAjPnz/f3NyckZHBZrOv2kav17e2tra3t8vlcpPJNPhCKpXq5ubm4eHh4+ODhmgmDQjhkiVLpFLphx9+OHjHEkGcl+PyuKVznZmZedVxkhFACDs6OnJzczMzM00m07FjxzIyMgYvqEdJJpPl5+cvXLiQxWINPbJcLi8rKxsYGJBKpd7e3kKhkEqlWj5jIIQGg6G7u7ujo6O5uVkkEsXFxY31vMhVQQj1er1er3f8sB6CTEA0Go3JZI5yzOBKDsrj7e3t586dW7hwodW92p6enuPHjwMAMjMzXVxcrDiCTCYrKipavHix5Zel0WjOnj1rGeFxd3cf+foAQtja2moZvZ0xY4ZleBQZK4PBsGfPnt9+++3MmTOdnZ1qtRrlcQQBADAYDMvNvLlz5951110+Pj5jerkj8rharT506NCiRYuG9oWt0NHRcerUqfnz5/P5fOuOUFpaqtPpZsyYIZPJzp07l5SUJJFIRj8IDiGsqampqKhIS0tDBTTGBEK4e/fuxx57rLm5mc1mJyQkBAUFCQQCNOcaQQAAGo2ms7OzsLCwpaWFRqOtWbPm5ZdfHv3Vv93zOITw0KFDcXFxw6YQWKe9vb20tHTBggXW3YEkCGLfvn0+Pj7t7e0ZGRnWjY0qlcqjR4/GxcWNcoAeMZlMGzdufP/99319fZ966qnly5db/UmMIJMYQRD5+fkffPDBnj17QkJC9uzZM7jmeWR2z+NNTU0tLS2zZ8+21QFzcnJCQkIsBees0NLScubMmeXLl1tWFVrHYDAcOnQoNjbW19fX6oNMEQRBrF69etu2bbfeeut7773H4/HIjghBJjQI4a5du9auXcvn80+dOjWaOaz2XQcEISwpKcnMzLThMePj4xsbG63O4z4+PufOnTObzePJ43Q6ff78+QcPHuRyuUKhcOi3IIQKhaKzs7O/v1+lUlmWm+M4zmAwuFyuQCAQi8U8Hm/qzGh84403tm3btnbt2jfffBONoiDINWEYdtNNN/n6+i5btuyGG27Izc0dtrTtKi+xa3+8q6urvLx83rx59juFFc6ePevu7h4QEDDO4wwMDOTk5CxevNiykKS/v7+6urq1tZXD4Xh4eLi6uvJ4PAaDYakrotPpVCqVXC7v7OzUaDS+vr4hISGTfvZLRUVFYmKipUj30FIkCIJc0969e1etWvXUU0+9+eabI7e0bx7Pzc2VSqVjvfdqb3K5XKvVWt2jH6qyslKtVvv5+RUWFhIEER4eLpVKr9nTNxqNzc3NlqpP06ZNE4lE449kYlq1atXu3bvPnj3r7+9PdiwI4mQghLfffnt2dnZ9ff3I9xfteJ1rWU5pk9ubtiUUCm2SxAEAfn5+5eXl+fn5iYmJixYtCggIGM1wDY1GCwoKysrKio+Pz8vLO3ny5KTcBLKrq2vXrl0rVqxASRxBrIBh2OOPP67Vardu3TpyS7vkcQhhd3d3bm4ujuPjGYae4FpbW//888/09PRFixZdcwb6lTAME4vFixcv9vDw2Lt3b2dnp53iJEtOTo5er1+5ciXZgSCIs0pKSvLz8ztw4MDIAyc2TrKDJU3odPoknmENIaysrGxoaMjKyvqntf6jhGFYSEiIp6fn0aNHw8PDQ0JCJs0t0Pz8fCqVGh8fT3YgCOKsMAxLSkrKzs6GEI6QGWyZxw0GQ15enlarTUlJEQgEloolNjz+xNHQ0NDe3r5o0SKr19EOw+PxsrKyLB3YmJgYmxyTdHK5nMViWbf4FkEQCy8vr4GBAbPZPMJ0L5vl8YGBgaNHj0ZGRg72KEUi0WRd7uHn5+fv72/bWXQ0Gi0jI+PIkSN0Oj00NHQS9Mq1Wu14SkYgCAIAYLPZlnpEI8z4sk0mksvl2dnZs2fPHpqAmEzmZF30QaFQ7DEVmkKhzJs3r6ampqOjw+YHJ8Uk+DRCEHKN5k1kg2SkUqmOHj2akZHhgAHxwaKykxWVSs3IyMjNzdXpdGTHgiCIcxhvHjebzUeOHElJSXHMnjuHDh2a9Kmcw+HEx8efPXuW7EAQBHEO483jxcXFUqnUy8vLJtGMzGw27927t6KiwgHnIldAQIBGo+nu7iY7EARBnMC48rhSqZTJZHFxcbaKZmTl5eX9/f0nT550zOlIZJlsVFBQgMpzIwhyTePK40VFRQkJCQ4rfnTixAkAQH5+vlardcwZSSQUCjEMk8vlZAeCIMhEZ30K1ul0crlcKpXaMJoRaDSagoICy3ktX0xuGIZFRkZWVlaSHQiCIBOd9Xm8rq4uODjYYRPLCgsLB6dwnDx5cioMOEgkkq6uLrPZTHYgCIJMaNbn8ebmZkfWP7IMqliUlZX19fU57NRkwXFcLBZPvrorCILYlpXrOS0FtTlj3PneaiaTyd/f39fX99y5c/Hx8TQarb+/f9gGDpOSRCJpa2vz9vYmOxAEQSYuK/O4Uql05KY2VCp11apVAIC2trabb755nKWpnIhYLK6qqiI7CsRKA2dzDd1T9HKK4enNnzaD7CimCuvzOKp/NE4QQo1B1dxzobG7ikqhz4/515VtWCyWTqcbudQZMjGZddrqJx8ktBqyAyEJjoe8+aEoYwHZcUwJVuZxnU5n3WbzUxyEcEArb+quaeyuauyu7lS0QkgAAALFUVdN1pY5nSiPOyMKkyV9+Mmmt1/hhIT73HY3mEp/QEgQzd9+UffiRoaXNzcimuxwJj8r8zhBEGjP3NFTavur20sauqsau6v6VN0QQByjeLhIZgXPDxCH+7uHcRj8f8rUNBrNZDLR6XQHx4yMn+fNt2vrL3T9ukNVW+2xZPmU+jAOevK56hc31jz5YNS3PzM8JtymYJOMlXmcQqHo9XrbhnKl5uZmoVB4zb2iJzgzYf76yGvdyjYco0iEAdE+MwLcw33dgtmMURWDpFAoaOqhk8IwzP/JZ7WNDa0/fseUSAUJ08mOyHGYHp5B656uee256vUPRn21jcKaKve0SGFln5rNZms09h34gxB+/PHHx48ft+tZHKC85Wy3si0z6qYXV3y19rqXsuJXhUsSRpnEAbr0cXI4nR761ocMb5+GT97RtjSRHY5DccMi/e5/VFNVUfvCRoj6IvZkZYLg8XgKhcJWQSgUCssan7KyMoIgLE9iGPbiiy8uWODc90kISByr2sdh8NPCl9CpTCuurI1G4wj145GJjyZwDXv3M4CBuvdeNypt9q5xCsLUOZ433tJ39HDLZ+9PhbV7ZLEyj3M4HLVabas/zO+//37u3DkAwObNm4eOIXC5XGffprm243xbX0NK6EIGzZrbwhBCg8HQ0tJi76sfxK7YQSHBr7yj7+5s+GgTMdkLLw+FYZj3ilWus9Lbvv+6+49fyQ5n0rIyS+I4zuPx+vv7bVJ23N/ff8eOHXl5ef39/e+9997QYYQ5c+bMmOGss1AhhDkVfzCozOTg66w7gsFgwDCso6OjrKzMZDIFBQWFhoaie57OSJieIX3oiZaP3pZt3Sy964Gpc88Tw3G/Bx4x9HQ1vPkSw0fqMm0m2RFNQtb3dgMCAmpra5OSksYfRFpamru7e29vb21tbXJy8tAdHZ16KWNLb21Dd1Va2GI2w8pbtX19fd7e3jNnzgQAaLXa6urqP/74IzY2dnATVMSJeP/7Hm3dhe59vzN9fMXzF5MdjuNQGMzAdU9Xv7jxwsbHor7dwfL1Jzuiycb6PE6lUs+fPx8eHj7+TTgxDIuIiIAQQghnzZrl7GMpFpbOOJVCSw3LsvognZ2dnp4X52yxWKz4+PjIyMi8vLyWlpb09HQ0bu5cMBwPeOa/upamlu+/Ynr78KNiyY7IceiuoqAnn6v+79PV69ZEbdlBcxGQHdGkYuX4uMFgKCwsnDlzZklJia1CMZvN33///aSpuN050FLVVhTvl+rCtr4OTGRkpJ+f39Bn6HR6WlqaRCL5888/HTD1E7EtCpMZ+tZHdHdx/Ydv6TrayA7Hodh+AQEPPalrabrw9OOE0Uh2OJOKlXm8qKgoMjIyIiKir6/PVtuPUanU2bNnT5pNcI5X7QMYlh6+dDwHodFoV16dYBgWHh4eHR2dnZ2NppY7Hbq7OPSdT6HRWPfuaya1iuxwHEowbYbP7fcozuU2bHp5crzNJwhrRjDUanVHR0dSUhKO46mpqcePH1+yZIlNrvHpdPovv/xy5swZFxcXy/jv7Nmzp093vtUTclVXaVNutE+SG89eK9kCAwMVCsW5c+eSk5PtdArETrjhUYEvvlH7n3UNn74X/OSz2JAbQpOeOOt6XXtr966fWb4B3nfcQ3Y4k4Q1eby8vDw6Otoyq0QoFIaHh588eXLu3Lnjv/MWGBh45513Dn1GLBaP85ikOFG13wyJ9PCldr0bGRsb++eff3Z1dTnpb2kqc5ufpW2oa/3q49YdP0huu2vq3LXGMEy6+n59Z0fzx28zff2EczLJjmgyGPO4itlslslkQ3eQCAsL43A4ubm5479QSk5OTktLCwkJiYmJSU9PnzNnjq+v7ziP6XhK7UBBw/EQz2iJMMCuJ8JxPDk5OT8/H12iOiOf+x4UXpfVuW9X7/G/yI7FoXAqNfDRDUwvSe3zG1RVFWSHMxmMOY9btjUYOjXQsrk7j8czGAzjjKajo+M///nPyy+/fOjQoVOnTm3dunVweSe5BjRypXZglI1zLxw0mvVzIq53QCdLKBRSKJRJc3N4SsFwPOjF19lhEc3ffKaqmVobsVK5vKD1z+EUSs2Taw3dXWSH4/TGnMebmpqu3M4Nw7CYmBgGgzGeUCCEX3/99fXXX7927VoAwPTp01tbW8+cOTOeY46f0WT4q2zXu/ue/PKvl3WGay+q1BrUebXZUlFwoDjCAeFhGBYaGlpXV+eAcyE2R2Gxw975lOriWvfe6/qeqZXOmJ7egeueMcp7q59Ya9ZqyQ7HuY0tj0MIu7u77TQaazKZent7Z82aZenGslisjIwMEnfDgRBWtRV9eOCZ7LKd7nzvXlXH7/nfXnME41zdUa1BPSdimcNGPCUSSXt7u2POhdgcw9M7dNNHZq227t3XzX/vJD5F8CKife99UF1ZVvfSU3BiXHk7qbHlcYPBQKPR7FR+j0KhYBimUl2aidXR0WHDXYfMxBim6MlVXVtPvv/98XcMJt3KmWseXvBKWtjikubT+fU5I7zKaDKcrP5TzJdEeCeON9xRo9PpGIYZ0YRcp8WLTQh89mVtU33jFx9MtXQmSs/0vGGl/K+DLZ+9T3YsTmxs81UGBgbst50bjuPz589/7733wsPDOzs7d+3adeTIkZdeemn8RzaaDSeq9p+s/jPEMzotbIlEGDBCZ9loMhyv2nuscg9BmFNDF2VG38SicwAAC2Jvbuiu3lu41VcU4iHwueprCxtPKHX9C2NvcXClWR6Pp1KpbFLrBiGF2+IbNHW17T983f7bDq8Vq6bU9BXvm/+ta29r+/5rpl+AeNlNZEfklMaWx1UqlV13dcjKynJ1dT116pRKpeLxeM8995xIJBrPASGE9V2Vuwu+7Va0ifmSspZzpc15QR5R6eFLQzyjMQwf1ri6rXhv0dZeVWegOHJZ4h2egkuzZagU2q2zHvzk0HM7cj9ZO/+/dOrwmwFmwnSiar8r2z3Ob9Z4YrYCk8nUTbFL8kkGwzDpQ4/rGuvaf9vBlEiFs9LIjshxMBz3X/t4TW9XwxsvMrx9XNDuzGM3tjyu0+lYLJadQgEAYBg2a9as5ORkCOH4u7Qq3cCfxT8WNZ5kM3j/mvlAov/sfnXvqZoD+fXHvj32lpfALz18SbR0BpVCAwD0Kjv3FW2rbCt0YQlvnfVQrG/ysCwPABDxPG+Yds9PeZ/uLdx6Y9K9wzpNZS3nelUdyxLvtBzQkXAcnyATexCr4RRq0Mubyu9d1fTlhwwPL05gMNkROQ6FyQx64tmqF9ZfeOrRqC2okNaYjS2Pm0wmNtuO+zOZzeY//vjjxIkTKpXKx8dn5cqVERHWzPqAkMivP36gdIdGr5oekL4obhWHyQMAuHLdlybekRF149m6I6drDv6U99mh87+khi7SGdXHKvcSkEgPXzovajmT9o+fVfH+KXWdZefqjwZ7RMUO6XcTkDheuYfD4E0LmGNFwONkuW/h+PMitkXl8sLe/azsrpvr3ns9/JV36K7WV+ZxOnShKOjJZ2te/k/1urXR3+6g8u01fjspja3Pi2GYXZec/PzzzyUlJffff/8rr7wye/bsDz74QCaTjfUgHf0tXx159bdzX/OYggcynrtpxv9ZkvggNoM7N/L6jcs+uCnpPhqFsbdoa3bZb/7u4Y8uej0rftUISdxi2bQ7xXzJrvwtvcqOwScvtJe29TelhC6ybr+IcbL3eBfiMEwf35C3PjIp+uvff4MwTK06aJyAYP8H1+laGmueeowwjncxypQytjxOp9PHv9jnn5jN5pMnT65bty48PNzDwyMjIyMrK+vkyZOjP4LBpD9Q8tMnh55rlTcsjL3l4QWv+IvD/+mWEZVCSwqa99ii1+9Me/LOtCfvnrNRzJeM5ix0KvPWWQ+bCOOO3M9MZiOwlKit3MOgMmcGk7DImCAIrVZr1/EuxJFcps3wX/+c+kJV0+bPJtpKXXvXKRRMT5bculpxLrdh0ysT7WefyMaWx1kslv02GCMIAsOwodXM3dzcRvmxASGsaiv+8M+nj1X+EewR/XjWW3Mjrx/NODWOUyIkiRGSxDHNEPBy9V0S/2+ZvO5g6U8QwuaeC43dVTOCMjmj3j3Zhnp6eoRC4dSZ4TAmarXaMdV9NRqgsl3tQo8Vt3rceof8xJGOPb/Z7KDjpqgobfzyI7ueAsMwj6U3iuYt6N71c/u2LXY912Qytjzu4uIyMDDa5emjBCG0FF+l0WiRkZEnTpywfA7r9fpjx46NZr+hAY38x9Mff3/8bRNhui3l0dXp64Vcu9eNmhmcGe0z42T1geq24pxKy34Ri+x90qu6cOFCUFAQKaee+ORyeX5+fnl5eVdXl11r/F64AMLDsbVrwfHjwCYbcPo99pRLcmrbTz/0F1i/pBmaTYTBMLRjCyEkTMYrd6+HZjNhNA7rAkMICeOll+vb2wit3feJxTDM9+413KjY5k/eledk2/t0k8PY7nNyuVylUmnD06tUqnPnzgmFwri4OACASCTavHnzkSNHBAJBXV0djuPHjh07duwYAODGG2/08PAY9nKCMOfVZh8+v9Ng0s0KWTA/ZiWLbsfbsENhGHZT0n2tfQ0/5X2qN+qmB84dz34RVtPpdF1dXbNmOXqmoxOBEPb19fX19VEoFJFIJBaLB6si29bAALZtG7ZtG/D1hTffDG+5BYSFAavPg9Nowa+9V37PrQ2fvhv237fZUr9rv2YIs1bbvOVzZeV5aDRRuFy//3uEFx5p1uubt3ymKC3C6QzB9Jm6jraQDS9As7l1xw/y08cAAGz/IL8HHqXxXdp+/R+h0ynKSw1dHXQ3ccAjG6DZ3PbLdkKnq33v9eAn/mPlTzU6OI0W+NhT1S9urH1hY+RXW7nhUXY93SQwtjxOoVAYDIZareZwOOM8MYSwoaGhuLh42rRpg0UNPTw8brvtNsvX4eHhQ9tfWbxF1lu/u+Bbmbxe4hq4fPrdI6/usQcWg3PrrIe+/OsVgGFpEUsceepBhYWFMTExDl52ZCsGg6Gjo2PYk97e3oNbZ+j1+s7OzmENJBLJYJm2azYYymw2d3V1dXV1MRgMd3d3sVjMYrGqqrDm5suaicUgIeHSw/JyMOxeu4cHiI+/9PD8edDWBhoaLj3T3Iy98w727rswMRHceiu86SZgXSULmosg9J1Py+9dVffOq+GvvEMbyxSOrgN/mJQD0e99gVGosu1bOnbt4D3zcsfun439fdHvfwXNptq3/gsJMwCg88AfivLSyDc/orDZLdu/bfr6k6An/mNWq/vyToU89yrDXdzw2fsdf/zqv+YxrxW3DhTlBz32lDU/zBjRePyg9c9Vv/RUzRMPRn33M0M8vA+HDDXm+uNSqbSlpWVYkh0rCGF+fn5/f//ixYuZzEsTPObOnTual+uM2sPnf8m7kE2j0Jcm3JEcMp+Ck1OJ39ct5F8z7tcY1e48L8efvaOjo6+vz3k74yaTqXlYEgXAw8NjMI8bjcYrG3h5eQ3N41c2uObe3Hq9XiaTKZXKgICAzZt5X3112cf/4sXwp58uDS988QX23XeXNVi+HG7deqnBxx9jP/54lQ4EhFhBASgowF55BW7aBFetAlZ82rIDgoJfe7f6sQfqP9wU8vRL+Khnl3osWS5edL1Zq9V3tBN6vUmtBgD0nTktvfM+CpMJABBnXd+5bxcAoO/0CV5kjFbWDADgBoU2Ht5P6HUAAJfEJJa3DwCAHx3ff/Y0hmEYTsEw3GG7XrC8fQIf3Vj71ks16x+K/PIHCuvql9p5eeD22y/7/dNosKAADHY1T5wAd911WQMMg7W1lx5mZ4MHHrisAYcD8/MBnX7x4f794JFHLmvA5cLiYusvtmxuzHk8ICDg+PHjYWFhVnd+IYR5eXkAgOuuu86Kg5S1nPuraqdCK4+WzliacAcpoxlDJQTMJuW8KpXq9OnT8+fPd947nCwWa+bMmcOeHLqPHYfDGbkBl8u9ssFVO+OD2Gy2pTNuqUtz//1wyZLLBoXd3C5r/9BD8MYbL2swrHP9+OPw1lthXR144onL8jSOw6QksGoVvPFG4Opq/XtekDzbfdmN3bt39ufnjX6dp6apoemrj6HRyJT6DWZ/k2KAwrl4K57K4wMAIIQm5YCmrsbwd7VFl2kzockEAKAwL86AwigUAMiZOsKPjnPLWNh9aF/3/t2eK1ZdtQ2PB4bdRKNSL/vI5POHNxj2t3B1Hd6AxbqsjVA4vIE9V9FYY8x5nMPhYBimUCisLrRSUlJCEERKSooVCYgfDHcVfuXKFa9OWx/mHe+8KWyclEpldnZ2amrq0Ok9TgfDsJGXL12zAY7joxxTotFobm5uYrGYy+UO/bcJCwNhYSO9MDwcjHzxGRkJIiPBkPoRMCAA3HILvOUWEBRkgy7bQN7Jnn27Wf5BLnHTRv8q2bYtrrPSvG68BcOwroN79Z3tAACmt0TX0sQNDgUAaFuagOU3LHRzTZ4tXrgUAGBUDAwU51/s+V4tdAfPBVSUl/YcPcQKDBZl/uMkgqgosGPHSFHFxV2jwbRp12iQnHyNBqQbcx7HMCwqKqqsrCw1NdWK88lkso6ODqt7kZpWLC1lyXWxK64sbzJFqNXq3t7e/Pz8tLQ0d3d3ssOZ6HAct6RvgUBg77sIrq7whhvgqlUgOdmaIZSrUlWWX3jmcZqrMHj9c5SxdAIxGk3f1WGU9+raWzv3/47TGQAAj6Urmr/9HKNQoNnU8cdOusgNACBesKRl62aGpxdd5N764/cYhSpKy7jqMXE6Q9/eOlBa5BKbcNUGtqVuqKt79zWa0C384800wUSsAffjj+CLL7BffoGk76tozf6cUqm0pKREqVSOtTNoMBjOnTu3cOHCka98R2DSgrnhy6dgEocQ9vT0VFRUyOVytVp900032bVAwqTh7e39Tz0GCMGPPwJ/f5CSYoMThYSAqipo27+Jvr21+om1AILgDc9bcu7o+d33UOuOH+o/fIvlFxD46FNdh/YSRqNg+kycTus7c4rK4XkuW9F39jQAwHVWGsDxrj//MGs03NAIrxW3YhjG9g8a/Cyiiz24kTEAAJeEJEVZifzEEX6M3S+FdZ0ddW+/jNPpYR98yfAg4ebTaHR2gsJCzGgkv6tuTR7HMGzatGlnzpzJzMwc05+zqKgoMjISJaDRgBBWVlYCAPR6vUKhkMvlfD4/LCwsPT09Ly+vtrY2NjaW7BidwAj/nxCCJ57AVqwAKSk2eB/a/J/aqBioeux+U788+On/ssY46RAAwPT0Dnr86cGHAWvXAQBaf97GDgjyv/9RCGHLD5uZXhIAAIZhwuTZwuTLbvOI0i91yfmRMfzIGAAAlcsNWPu4tT/QGBgH+mvfetGsUYd/soUTMuKwFwIAsC6PAwC8vLxqamoaGhoCAwNH+RKVStXR0TGadT2IBZVKNZvNrq6ufn5+Q4cFZsyYsWfPHj8/P/vVgkfIRej1Nesf0jbUBTyywZJDbYITHNb05YfyU8fMGrWhuyv46f/a6sg2ZNZqaje9rO/qCt30ET9hOtnhOAcr87ilwOyff/4pEolGmU3Ky8ujo6OddKaz41k23rzqtygUyowZM/Lz8zMyMqbsnd5JDBJE3X+fURaek9x+t2uyLWdDCRKTOJs+0TTWU5hMdmAwTqNf+zWORZiM9R9t0jTU+j/9knAOCdWKnJT1WZXBYKSlpWVnZ4+m4orZbJbJZH5+Y748RK7Ky8tLq9WqbFjRA5kYIITNH7/Te2if+6JlHktutPnnNM1F4BKXyA2LnIBJHBJE4xcfKYoLfB541HPFrWSH40zG1TsWiUQzZ848dOiQWq0euWVHR4enp+fQmb/IeGAYFhISUl9fT3YgiI11/LStfes3ghkp0n8P36hkcoMQyn78ru9Ujse/bpPcs4bscEYlKQls2AAnwtTf8Y5y+Pj4TJ8+/cCBAyPXz5LJZD4+V9/TErGOr6+vFcXZkaFEIsjlkj/ZYFDvkUNN773OCYsMePAJhy2bnCA69//etXeXa8YC//XPYk4y+pqaCl54AfL5ZMdh9fj4UD4+PiwW68iRI3FxcQEBVy9y0tXVZamEhdgKi8UyGo0EQaBbDtbBMFBaOoGWViuKC+pe2Mj09A5a9x/8impCk1vP8SOt27/lT5sZ/PLbGLpqHzvbpACRSLR48eLGxsYjR45cOWgLITSZTEPrqCA2wWKxtFot2VE4KwwDFIrNFuyMk7axvmbDwziDEbThBdoUm4Y0UFLY/PXHrKDQ0E0fU1CWsIrN/osZDMa8efNCQ0P/+uuvvLw8pVI5uITXZDKh3SPtgclk6nQ6sqNAxsvQ21P12P2ERhO84QWm5wRd82In6tqa+g/eoLmLwz/6mjrFPsBsyJaXMBiGSaVSiUTS1NR04sQJDMMCAgIkEgmdTkfX/sMUFBT09PTMmjWLx+NZfTvL3tulIg5g1miqn1ij72gLeuJZTlAI2eE4lLZNVvvOKziTFf7RZmesTPvTT2DzZux//4OkF8iw/VAUjuMBAQEBAQFKpbKpqSk3N1ehUAiFU2jn79Ho7Ozctm3b9u3bY2Nj09LSEhIS6PQxzwMzGAxWvAqxgBDs3An8/MCMGaTFQJhMF55dpy4/73vfw4LEqbVEztAnr33rJcKgj/jkW3aAU25o1d4O8vIwg4H8vpQdbynweLzo6Ojo6OiBgYH8/HxbHfbUqVNX7inhdOrq6gAAJpOpsLCwsLDQUn81LS0tODh49NcuGo0Gba9sNQjBI49gK1aAGTPIeR9Cgmh8+9X+Ezmey292y1hASgxkMalVtW/91yDvDXv7U16cI6puTW6OuDXMYDBstddtVlZWa2vrJLi5Z7x833Gj0ajRaDQazejnn5hMJoIg0I0H59X6/Vddv/4oTM/0Xnn7lJoqThgM9R+8qW1uCHz+Nde0uWSHMxk4Io/T6XRb5XFLB98mhyKX2WzOz8/HcTwsLCwtLS0pKWmsW+W1t7d7enraKTzE3rr27pJ99gE/NtHvvoecZbq0TUDC3PD5+8qyEp8H14mvX0F2OJOEI/I4juNUKhUN5g4lFApvvvnmlJQUsbWliy9cuBATY7MKSogj9Z851fDa8yy/gIBHN45+q7ZJwFJnsT/vpOeqOyV3P0B2OJOHg6bci8Xijo6Owf2USdfT05OXl7d48WKyJtIkJyeP5+VKpVKtVru5ja0mNTIRqGsqLzz9GM1FELz+eeq49yt3Lh27f+k+uFc4P8vv8acnwVDStGngsccgl0t2HDacPz4yX1/fK/fDdbCurq729nbL193d3WfOnCEIgtyQrGPZpTo+fupua2crXC5kMBx6k1Pf2V79+BpgJoKfemmsW0M4NQhh95FDbT9t5SclB//3rclRdSAtDbz6KpwIs94d1B/38PDIzc01mUwklsqyTGm/6aabAAARERGvvPIKWZGMU1NTE0EQqF7NOOE4GLppugOYlIqqR+839slDnn6J5TNRrk0dY6DwbPOWz9jhUaGbPrZsMofYkIOyqmVSeW1tbfjIe9bazS+//JKTk4Pj+Pnz51euXCkQCHbu3Pnoo4/W1dWdPn0aAFBQUCAQCO66667S0tLjx49TqdQVK1ZYdr3o6enZvn17U1MTl8tdvnx5QkICiR1huVxeVFSUlZWFOuPOhdDrazY+qq2/4P/QkzzbbQ3hFFQ1lfUfvc3w8Ap//wsqbwKUlZp0HDc6HBERUVFRYTKZHHbGoRYsWBATE5OUlLR27drAwEC9Xt/Q0AAA0Gg0Bw4cEIvFr7/+enR09Kuvvtra2vrSSy/ddtttn3/+uVKpNBqNmzZtiomJeeONN+6+++7vv/++sbGRlB8BANDT05OTkzNv3jxUrMa5QIKof+15xblcya2rhSnpZIfjUNqWptq3X6FwueGffEN3J3tD4knKcaMcTCYzJCSktLQ0MTHRYScd5OLiwmazuVzulfNDRCLR/PnzcRyfP3/+rl27/vWvf3G53NjYWHd395aWFgghjuOhoaHd3d00Gi02NjYvLy8gIMDB8UMIa2pqKisrMzMzp+B2bnI5WLwYAHDZJcgPP8DBHZM6O8ENNwy/QPnxRzj4h5LJwL/+NbzBzp3QAaNTEMKWzz/o2b/bfeEyj2U3TakLKUNvd+2mlyEkwt/7nOXrT3Y4NrZzJ9iyBfv++8m4Ln8EkZGR+/bt8/PzE4lE4zmOVqvt6emRSqU2iYrP51tmrdDpdCqVyvu7LDyLxTKZTEqlUqlU7tq1a7C9zZM4QRANDQ1SqfSf5mW2tbUVFRW5urouWbJkai78wTBw5cyOoVONcPwaDa55BPvp/PXHtm+/FMxIkd4xtbaGMKlUF958ydgvD33vC170JCxbLZOBEycm+7r8K1EolPT09JycnKysrPGsrYcQVlRUjDWPYxh21QkqQ99aGIYNe6d5eHiwWKyHHnrIkuubmpr4tq4bT5jNSqXywIEDGIaFh4f7+vpCCM1mc19fX2dnZ3Nzs1AoTE1NdXFxmVJZYChXV/DXXwCAf3zDuLuDv/4a6e0kkVyjgZ3Ic7Ib336VExrh/+C6yTFJY5TMel3du6/q2mRBL73pmpJGdjiTnKNnjwgEgsTExOzs7AULFljdtWSxWFbUa3V3dz9x4oREIomKihr9qwIDA4VC4ZYtWzIzM2Uy2U8//fTSSy+N9dQjo9Jo8fHxcXFxvb292dnZtbW1EEIqleri4uLu7h4TE4PWTzkpVXVF7fMbgNlMYXOat3xOdjgOpWuTaWprfB/d4JZ1PdmxTH4kzAL09fU1Go2HDh3KyMiwrswThmEUCkWv14+pU5+ZmWk2mysrK/39/Xk8XkpKCgBAKBTO+LveHYVCmTdv3uDKoOnTpwuFQhzHn3jiif379//yyy8CgeCZZ56x3+qb/Pz8efPmeXg4XwFP+zEawZdfgrg4kOaEXTp9mwxnMnEGQ9NYR3Ysjod53/2A1xQbSiILObO5AwMD6XT6gQMHLAvTrfhLu7m5dXd3j2kONY1GW7JkyeDDlStXAgAkEolEIrE8Q6fTV69ePdhg6dKlli+YTKZl1rldVVdXX/U27BRnMoEXXsAeeACkpZE/CjlWwrnzhXMy/3k0aLLDcZTEHYOcPG7ZcUIgEJw4cUIkEo21+rZSqZTL5RwOZ9Kshent7a2url68eDH6v59MMAwD2BQaE59q4uPB2rVwItRWIHNLUx6Pl5WVVVNTs3fvXl9f3/DwcO6IpQoghHK5vKKior+/Pzo6uqyszPGFopRKJZVKtW3Vb5VKdezYsYyMjKk5FwVBnNTcuWDu3AlxtUXy1tQYhoWFhQUHB9fV1R07dgxC6O3t7e7uzuVyLUtdCILQarX9/f09PT2dnZ1cLjc8PNzb2xvDsAsXLvT39wsEAodFCyHMy8tLSEiwYR7X6XTZ2dkpKSmO/EEQBJlMSM7jFhQKJTQ0NDQ0VKfTtbe3d3R0KBQKS8lyKpXKZDL5fL6vr29SUhJlyMytiIiI8vLy1NRUh8U5MDBgMBjGOfl92AGPHDmSlJSEKokjCGK1CZHHBzGZTMvenqNp7OPjU1RUpFKpRh6NsRUIYUFBga2qDEII29ra8vLy0tLS0L3NEWAYCAiAaHtXBBmBE29EgmFYQkKCDXf+HJlcLjebzYOTW8bDbDafO3euqKho4cKFKImPjMkEBQVgwway40CQK+zZA1atwnp7yY7DqfM4AMDHx8dkMslkMnufCEJYXl6uUqlkMhmE1t/ZgBA2NTXt3r2bTqcvXrzYMVcSCILYQ10d2LsXG/uSRNubWOMqY4VhWEpKysGDB0UikV13jm9sbDQajZmZmfn5+SUlJVFRUb6+vqPZSwhCqFKp5HK50Wg0mUw1NTVcLnfBggUogyMIYivOnccBAGw2e+bMmUePHl2wYIGdNqmQy+UlJSWLFy+m0+kZGRkDAwPl5eUFBQXe3t6+vr7u7u40Gu3KQXOCIKqrqysqKlQqlV6vtwz9czicvr6+2traqKgoNMsQQRCbcPo8DgDw9vZWKpVHjx7NyMig2LoUkUKhOHbs2Lx58ywrlTAMEwgEqampRqOxvb29sbExPz+fIAgXFxcOh2OpmEgQhMFgkMlkvb29QqEwODjYz8/Py8vLEpulV75nz545c+bYcOrLZGUygS1bQHQ0SEkhOxQEmagmQx4HAISFhZnN5sOHD2dkZNiwqlRfX9/Ro0fT09OvnNxNo9F8fX0tO0dbytuq1WqDwWAymXQ6XVVVVUZGhqen55WfK1QqNTIyUiKRHD58eM6cOe6kly6e2IxG8PTT2AMPgJSUCbHgAkEmIOe+zzlUREREcHDw/v37+/v7x380CGFjY6Nl851rFsaiUqmurq4+Pj6BgYFBQUFNTU0LFiyQSCQjXBy4uLgsWLDgxIkTVhRuRBBkIoiNBffdN+XX5dsWhmHBwcEuLi5HjhwJCwuLiIgYzX3Iq9LpdGfOnDGZTIsXLx5rnfTq6mqJRDKamoV8Pj8+Pj4/P3/27NnWxYkgCIkyMkBGxoS4TJw8/XELd3f3ZcuWKZXKPXv2NDc3X3XjiBEYDIbi4uL9+/dLpdKMjIyxJnHL7mvR0dGjbB8QENDT04O65AiCjMfk6Y8PotFoycnJCoWiqKiooKAgKCgoICBgcLe2qyIIoqur68KFC93d3aGhoddff711U1/6+/t5PN7oB+gxDPP19W1tbQ0KCrLidAiCIGBS5nELPp8/Z84crVZbX19vGYYWiURCoZDL5dLpdBzHDQaDXq9XKBR9fX0KhcLd3T0oKGj27NnjWXbf29s71vuW7u7unZ2dVp9x0sMwIJHAqbezNIKMwaTN4xYsFisqKioqKspoNA4MDAwMDFgqcJnNZgaDwWAw3N3dQ0NDuVyu1YPpQxkMhrEOxTAYDDSuMgImE5w/T3YQCHI1+/aBn3/G3n+f/Po/kzyPD6LRaG5ubvbbks0Cx3GTyTSml5hMJjstX0IQxK4uXAC//Ya9/jr5tzon231OcvH5/IGBgTG9ZGBggM/n2ykeBEGmApTHbUksFnd2do5pkkxLS4uXl5f9QkIQZNJDedyWqFSqSCQaff3F/v5+o9GIdgIagckEvv0WnDlDdhwIMoGhPG5jiYmJhYWFBoPhmi0Jgjh9+nRiYiLaW3kERiN48kns99/RrwhB/hG6w2ZjXC43Li4uOzt7/vz5I1Q0VCgUeXl5vb29eXl5AAA2mx0RESGVSlFOn0wqK8H27Zf9Qel0+OyzYLBeQ1kZ2LHjsgYsFnzmGTA4eaq4GOzceVkDDgc+88ylhwUFYNcuDABAo8E1a8Ao1hEjNhMVBe68E7LZZMeB8rg9BAQEmM3m/fv3p6enu7q6DvsuhLCuru748eMxMTHXXXcdjuMQwoGBgaKioqamppSUFJuXbETI0tgIPvnksme4XPDMM5fyeG3t8AYCAXj66UsPq6uHN3B3B0PzeEXFxQZmM+7nR9x1l82CR65p/nwwfz75k1UAyuN2EhwcLBAIcnNzLWXHRSIRjUYzm819fX2NjY3t7e3z58+XSqWWxpZauHPnzi0sLDx37lxycjK5wSO2smAB6OgY/j4fOst02bKrNBj6Of6vf4EbbhgpU6xaBVauhJWVID0dG8dGVYhzQ3ncXtzc3LKysrq7u1taWurq6gwGA41G43A4jY2NK1euZDKZw9pjGJaYmPjnn3/29PTYe5474hgUChj54mr8DahUQKWCMS4+QyYblMftCMMwsVg8dCfltrY2COGVSXywfXR0dG1tLcrjQ3l4wBGr4yDAzQ08+SQRE0N2HAhJ0HwVh+rt7R2a1q8kFovlcrnD4pn4mExQXn7ZiLCz+OUX8MILDjqXWAxeeglMn+6g0yEWBw6A++7DJsL7FeVxhzKbzSPfxsRxfKy1dic3DAM4DpxxFs+RI9jXXzth3MioVVWBn37CtFqy40B53MF4PN7I2xUpFArORNhfBEEQ54HyuENJJJKWlpYRety1tbV+fn6ODAlBEGeH8rhDMZlMHx+foqKiK78FIWxubq6vrxeSUQSTMBk0bWXa9krHn3pkJhPYvh3k55Mdx8TW0AAWLsT27yc7DoQkKI87WkJCglwuP3v27NAKt0aj8eDBg7m5uREREadOncrLy4P2nwwMCbO+t6mv5PfWvS/Vf3t76+9Pywt/tvdJx8poBI88gv36KxpoHolWC06fxtB+JFMWmnfoaDiOX3fddefPn//jjz/c3d05HI7BYJDJZCKR6IYbbmAymfHx8Tk5OQ0NDYGBgfYIwKTq1ciKNbJiTWupWd178VkMZ7iHsDwjIISoNoBNhIbClBSyg0DsKTwc3HorZLHIjgPlcVJgGBYbGxsVFdXb26vRaOh0ukwmS09Pt0xlwTAsJiamvLzchnncrFdr28s1smJtS7Ghr3nweRrfk+UTz/aJZ/vE4gweyuA2tG4dWLeO7CAQe1q0CCxaNCEW0aI8ThoKhTI4l5xGoxEEMTglkSCI8adUSJh0XRc0LcUaWbG+qwaajZbncQaXLYljSePZPnE0vhfK3Qji7FAenxACAgJKS0unTZsGADCbzYWFhbGxsVYcB0Jo6JdpZSUaWbG2tZQwaIZ+l+kV5Z5yL8M9GLPFZqQIgkwQKI9PCNHR0bm5uXv37nVxcenp6QkJCRn9JkEQQrO2X9taaul6m1TdF7+BYXSRP9snni1NMOtUndlv6zqrTboBprMlcYFgQgxBTmRCIXjkERgVRXYcCElQHp8QcBxPTU3VaDQajYbP59Pp9Gu+hDDqtO0VljuWht4G8Pf8FirXjSWJY0sT2JJYKufSFEaTqrs377vO7Hfp/3qf7uI0O8mxWKC+nuwgrPLbb6C8HDz/vCPO5ekJJsJuv1PNoUPg99+x116DV1SndjSUxycQNpvNHrEoPTSb9L0NmpYijaxY11EFzRd3HcLpHJZXFEuawJbG0wUSDLtKj9s1YYWhr0VZ/VfH4Xd8bnwTp/zjHheITRw+jP3+O3j+eZReJ62KCrB1K/bssyiPI6NgHGhXy0q0siJNaxmhG7j4LEZhekZYhk0Y7sE49RpdeAzDxGlrdJ3V+q7q/tI9woSb7B43giAOgfL4BGXWDmjayixdb5Oi4++nMZrAh+0Tz5bGs7xjcDp7TLNNcDpLPPeR1t+fluf/jx86d+ioC4Igzgvl8QmEMBl0HVUaWbFGVqTvqQeE2fI8hSVg+cSxfeLZPnFUrvt4ZgqyvaN4YRnK6r968r73zHSC6c0mE/jtNxASAhISyA5lAmtqAg89hD38MFy0iOxQEDKgPE4ySJgN8iaNrETTUqRtr4AmneV5jMZkSeLYPvEsaTxD5H/VIW/riGbeoao9oao9YZx5B4070TesMBrBmjXYAw+AhAQ00PyP1Gpw7Bi2YoVDf0VGI1AoLj3EcTB0mFirBZrLZr1euwGFAgSCSw81GjCsJOw1G1CpwMXl0kO1Guh0Y2tAowE+/9JDlQro9Zc1oNPBBNzVBOVxEkAITaoezcUh71Kzpu/iNzAKwz2ELY1n+8QzPcNxql1266Jx3XhhGYqKA/2lf7in3GOPUyAAgMBAOLk3drj7brBnz6VLQ19feP78pe9u3Qo2bLjswjE4GBYUXHq4eTN47rnLGkRFwdOnLz38/HPw8suXNUhMhEePXnr4wQfgrbcua5CcDA8evPRw0ybwwQeXNZgzB/7xx6WHr74KPvvssgbz58OdOy89fOEF8M03lzVYuhRu337x68BAsHgx/IfdvRwK5XHHMevV2rYyjaxIIysx9skA+HumIN+TbVkcL4mhsFxGPohNCGKWKCoOKC/kuCXfieFO8D+wbRs4cACLi4PffXfpyY8/Blu2XPYeS0qCX3116eG774Jt2y5rkJICP/300sM33gA//3xZg7lz4fvvX3r48stg167LGixYAN9669LD554D+/Zd1mDJEvjqqwAAsGED2LBhVD+dk2puxjw8wK23Xvw3HtpTBgDExYHHHrvs+mDYZoWJicMbeHhc1iApaXgDieSyBsnJwxv4+l7WIDUVDKs3N6zURXo6oNEuaxASclmDefMAl3tZg4iIS18vWKBLT9fx+XzSCw46wXvYqREmg7679uJMwa4LgLhY4xBn8tneMSxpPNsnnsb3dPDieLrQn+EWrO+p1bSWcqSJjjz1WDGZ4K67YFMTBgDw9r7sW0Lh8LflsLVTItHwBp6elz10cxveYFgqubLBsF35xOJrNJjEfvwRgity66CZM8HMmSO9PDUVpKaO1CA9HaSnj9QgIwNkZIzUYMECsGDBSA2yskBW1kgNli0Dy5b943d7enoaGxuTkpIYZG90jfK47UEIDX0yjaxIKyvWtpUTBvXFb+BUliTW0vVmuAViFNJ++RiG8ULn6HtqVQ15EzyPYxh47z0weO0y1O23g9tvH2lE+K67wF13jdTg//4P/N//jdTgwQfBgw+O1ODRR8Gjj07Rgft/yuCI46E8bhsQQrNGrmk9r2kp0spKTOqei9/AcLoo4OJMQa8onDYBxtIAAACwfBIAAFrZeVSo1n5MJhOV6oi3mFAIHngADr3kR6YUlMetByGEJp22vULTUqyRFRnkzQBe3LCNynW7WA9WEkthu07ARMkQSnGmi7FfZtYOUNkCssOZnJqbm6VSKY1m96Wznp7gnXem6GUBAlAetwIkTLrueq2sWCMr0nVUXaoHS+ewvKPZ0niWJJ7u6jMBc/dQGE5huAdpWwoN8maUx+3BbDZ3dXWxWKzRlzxzLi0tAMOAjw/ZcSAoj48ShIRR0aGRlWhbijVt5wnd3/NmcSrTM9IyU5AhDnGuoiV0gUTbUmjob2X7WFMjFxlZX1+fyWTq6ury9HT0fWzHuP12jE4H2dlT+jpggvxlUR4fiVmr0LSWamTFGlmJSdE++DzNVWq5XcnyjqYwOCRGOB40ngcAwKhEuzraRVdXFwBAqVTqdDoWKrw7GXl7e3t6eg5u/0IilMeHg4RZ216uaSnStBTpe+oHh7wpHCFbEsv2iWf5xFM5ognyOTweOIMLAIBG3TVbImNlNBr7+i4u7+ru7vYdNrHZ1pqawLp12Jo1cORpdoht4TiOT4xq/iiPD6eoyu7K+XjwIU0gcYlYyPZNoLv6Yjj5H7w2hNNYAADCqL1mS2Ssenp6BlegdHd3S6VSu37wq9Xg8GFs2bIpPcQxlU2ID5MJhekZwfAIA3+nbGN/a1/Jrr7CX5U1OUZl97DlYZPB5PuJyAYhtAyqWGi1WqVSSWI8yKSH+uPDMYS+0pveMesU2tbzlt12TIoO5YUc5YUcgOF0gQ9bGs/ySWB5R+E0llOPrlh64jh9pJ0rECvodLphibu7u5s/tPwSMino9XqdTsfj8UgfXUF5/CowDKOyXHjBs3nBsyGExoF2S0LXtp039DUb+pr7S/8AFBpTHMqWJrCl8Uy3YBIXZ1rNstAU5XGbG9oZt+ju7g4ICCD93W5bP/wAnbkbYwPd3d1oXb5zwDCMLvCmC7wF0Yuh2aTrqdO0FGlbinVdNbr2cl17ufzsNpzBZXnHsKUJbJ94mounDWvM2pVxoB0AQOVNmYIgDgEhVCqVlje2Xq+nUCiWJZ1KpdLFxRFF0BzG35/sCJC/oTw+BhiFyvIIY3mEgem3mg1qbXuFtqVY01Jk6GtWN+SqG3IBADS+J8snju2TwJbEUlgT+lLa0N8KAKALUJkMW8IwLDo6GgBgNBrPnDnj7e3t5+dn75MKBOCee2BYmL3Pg0xQKI9biULncP2SuH5JEEKzWm4ZeNG0lhoVHcaKDkXFQYDhDLdASyed6RGO00i+8hoGEmZ9Tz0AGN1VSnYsyHh5e4MPP0T3q6culMfHC8MwKlfED8/kh2dCwmzoa9HIijUtxdq2Mn13rb67tq/wF4zGYnlGWAbT6UK/iTDwopc3E7oBmquvYyqeI5NPezvAsOGlgBFSoDxuSxhOYYj8GSJ/17jlhMmg66zStBRrZMX6njpNS6GmpRAAQGG7XlwL6hNL47qTFapWVgwAYEminXrKDUKiW25B6/InCpTH7QWn0tmSWLYkFsI7CJ1C01amaSnWyoqMik5lzVFlzVHLLEaWNJ4jTWB6RVEcOG8EQqioOQoA4AbMcthJpxoKhRIaGsrhOGvZBuSa0Lr8KQTDMArLhReUygtKhRAaFe1aWYm65dIsxoHSPzAKjSEOuziL0T3I3nutGfpaDD31FLaQLUEVsuwFx3GxozYHksnA889jq1fDuXMdc0IEALQuf8rCMIzu4k138XaJyoKESd9dr5EVaVqKdZ3VuvYyXXuZ/OxWnMFjSWIsW0/Q+HaZxdhX/BsAgBc6d5JVGpiyFAqwcyeWno7y+BSF8jhpMJzK9AhleoQKp91iNmh07RWWG6QGeaO6/rS6/jQAgMb3YvnEsX3i2T5xFCbPJuc1KruUNTkYhS6I+eedBxEEcR4oj08IFDqb4zed4zcdQmjW9GlkJZqWIm1bqVHRbqxoV1QcABjOcA+2dNKZHuE4lW71uXrPbAOEiRe1mMYj7S7rVDBYigfdSZ6s9Hq9wWDgcDikj66gPD6xYBhG5Qj5YfP4YfMgYTb0yyybxmnbyvVdNfqumr7CnzEai+UVaZmZThf6jmngRdNeoaw5itFYomm32O+nQAAAJpMpPz9fIpHYu2jtoPfew5KTL+3SKZeDm2/GhpVB++orGBR08evOTnDbbcM/Y779Fg7GK5OB1auHN9i2DVo2ONqyBa3LR+vykWvBcApD6McQ+rnG3QDNRm1ntaalSCMr1nfXapoLNM0FAAAKW8j2iWNLE9iSWMq1qqITRm1XzscAQOH0VVSuyFE/x9RlNpsdUyDT2xukpMCBAUAQl56EEKjVw8tZXtlgmNE3CA4eX9CI7aA87hwwCo3tHc32joYz/k3oVdq285aZ6caB9kuzGC27FEkTWF6RV1a/ghB2Hf/C2NfC8AgTxKKR8UlFIAAHDw7/wBCJQG7uSJ8inp4gL2+kBlLpNRogEwTK404GwzAKk8cNTOEGpkAIjYpObWuJpqVI01pqkDcZ5E39pbsxCp3pEcaWxrN9EhjuQZZJKX0lu5TVf+F0juf8Dc61jyiCICNDedyJYRhGd/Gku3i6RC601EuxzHjRdVZp285r2873ntmKM3hsn1iai1df0a8Ap3jMX0/no5XUCDKpoDw+SWA4hSkOYYpDhIkrCaNW2155cRZjb72q7tTFNlSGpvEcMBlZPrEUBpfcgBEEsRWUxychnMbi+CZyfBMhhGbtgEZWrJUVa2SlJlXXQPn+gfL9AGAMj1C2TwJbGs8Uh45nFiPyTyzr8tlstE3HpOXl5SUWi2k08kcpUR6fzDAMo7IF/NC5/NC5EBKG/lbt4CzGzmp9Z3VfwQ6MxmZ5R12cxejqMxFqMU4OjlyXj5CCQqFMhOIqAOXxqQPDcIarlOEqFcQug2aTrqvGMjNd13VB03RO03QOAOC56FleIKqchSBOBuXxqQijUFlekSyvSNGM28x6lbatXCMr0raeZ3tHkx0agkx0er1epVJd+bxQKCRr7S7K4zZj0vRDs9HpFrtTGFxuwExuwEyyA0EQp1FZWTnsGRcXF5GItLV1aDDUZhSVh3rPbiM7CgRB7IvBYFy5Zba7O5kdOJTHEQRBxmbYHWwcx0nsjAM0rmIdCKGy5uhAxQHCoGNL44TTb6PQWQAAs17V8df7+p46pmeEW/JqCoNrUst7cr/T99RRWAJB7PWW4Qtdd5383P+Mig6GW4Bo5p00nljXVatqyDOrew39rUzPMIYokB82z3LAziMfemQ+gWFYb/4ObUshRmMJYpdxA1MxDOs89ilTHDJQto8fuUgQlUXyLwVBpgyRSFRXV0f8XWtGKBSSO/sQ9cetobxwrPfMD6Lk1V6L/mMc6Oj8611LOSRNUz5bGu+9+EVCp+w8+jGAsPv0NziTK1n2iiD2+o7sd0zaAaOqp/WP5zgBsyTLXqUL/Vr3vECYDGbdQF/RrziTx49YwHQP6Sv+zXJAVX0uYdDgNGZnzsfGPpnXomfdZt3dfeoby1af2tbSgbL9rvE3Md1RySIEcRwqlSoUCgcfkj7BFOVxawyUH3BNXMn2iqK7eIrT16qb8k1qOQCA45/ED5lL47m7pdyrbjpr1qsAYTIqOo3KLo7vNP9/b6Yw+cqaHIZbADdgBkahukQshCaDtq0MAIDTmG6z7nKJuI4TOMuklut7GwEAyuoj/PDrzDqFqu6UMOk2nM6mCyT8sMyBioOWSAQxy3ghc5jiEPJ+GQgyFQ3mbhqNJhAISI0FjatYxaTuofG9LF9TWAKcxjRr5AAAGt8LYBgAgMIRYgCYdQq3lPt6cr9t2/siwCm80LluyXebVN36nvqWXU9ZXo5RaYRRg9NYFI6rZQ0OTqHxQtKVF45R6Gx9byM3INmg6ACEuSN7EwAYAABASBf6XTw7Z0KXn3VM1VYEcTyBQECj0YxGo5ubm133kRjNmwjlcWtQmC4mjdzyNWFQE0YdzuQBAMy6gYtPahUQApzBM6t7xXMfxnCqrqOy46/3GW5BOIPHliZ4LbiYx3XddTQXL11H5dCFlPzwzPaDb1LoHG7gLJzOojC4AMOlN26iMPkAAKOqGxBmS8uJvNcMk8nU6/UEQZC+WwqC2ByO425ubu3t7fYeVNFqtRiGjbxVBXqDWYMXkt5fvMuo6iZMht5z/2O4B9N4HgAAVX2erruOMBl68//HliZQmLyuE1/0FfwEAEYX+uJUOoXO5galqpvy1S3FkDCpGs7Ifn+a0A+v1c9wC8LpnL6S3/nh1wEAqFw3pmdET94PhFFn0vS17X9FeeE4CT/2GAmFQo1Go1AoyA4EQexCLBazWCwu174l5zo7O3k83sgFAFB/3BqCmKWEXin7bSOEZqZHhNeiZzAMw5l8Xujc7uOfGxUdTI9wj4zHMAwTz324K+eTxq13Y1Q6L3QeJ2AmhuEemet6Tn1t1g5QWC6eC56i8dyNyi7akHKyGIYJohcrLxxneoRbHnou2NB9/IvG7f8HAMYNnu0afxMAgMb3wqgkbyg1goSEBJPJVFpamp6eTnYsCGJ7XC7Xz8/PrtfEEML8/Pz4+PiRz4KhEUzETtra2vz9/e+8884PPviA7FgQxCkVFxenp6e/8sorzz777AjN0LgKYi/e3t5ZWVk//fRTW1sb2bEgiPOBEH700UdUKvWOO+4YuSXK44gdPffcczqdbt26dWazmexYEMTJHD16dOfOnffdd5+vr+/ILVEeR+xo+vTp69ev379//+uvv45G8BBk9CoqKu65557AwMDXXnvtmo3R+DhiX2azecWKFbt3716zZs2rr7468vQpBEEAAIcPH77nnnswDDt16lRERMQ126P+OGJfFArl559/vueee7744ovU1NS9e/cajUayg0KQiQhCWFtbu3bt2hUrVnC53JycnNEkcYD644hjEASxdevW9evX9/T0eHl5TZ8+PSAgQCAQTOR1TAjiMFqttqurq6SkpLS0FEJ4++23v/322x4eHqN8OcrjiOMolcqff/75119/LSgo6OnpGSwXhyAIm80ODw+fN2/evffeO8pu+CCUxxESmEwmjUbT39+PUjmCAAC4XC6Xy2UwGNZdoaI8jiAI4tzQfU4EQRDnhvI4giCIc0N5HEEQxLmhPI4gCOLcUB5HEARxbiiPIwiCODeUxxEEQZwbyuMIgiDODeVxBEEQ54byOIIgiHNDeRxBEMS5oTyOIAji3P4fKXOxG496s3AAAAAASUVORK5CYII=

Without loss of generality, the elements within an agent can be partitioned into three types: Fixed elements are unaffected by learning, such as its substrate or unmodifiable code. Transient elements do not carry over between episodes, or across to evaluation (e.g., activations, the state of a random number generator). And finally learned elements (e.g., weights, parameters, knowledge) change based on a feedback signal, and their evolution maps to performance differences (Lu et al., 2023).

We can distinguish improvement processes by their implied lifetime; some are open-ended and keep improving without limit (Hughes et al., 2024), while others converge onto their asymptotic performance after some finite time.1

2 Three Necessary Conditions For Self-Improvement

Self-improvement is an improvement process as defined above, but with the additional criterion that the agent's own outputs (actions) influence its future learning. In other words, systems in which agents shape (some of) their own experience stream, potentially enabling unbounded improvement in a closed system. This setting may look familiar to readers from the reinforcement learning community (RL; Sutton, 2018): RL agents' behaviour changes the data distribution it learns on, which in turn affects its behaviour policy, and so on. Another prototypical instance of a self-improvement process is self-play, where the system (often a symmetric game) slots the agent into the roles of both player and opponent, to generate an unlimited experience stream annotated with feedback (who won?) that provides direction for ever-increasing skill-learning.

From its connection to RL, we can derive necessary conditions for self-improvement to work, and help clarify some assumptions about the system. The first two conditions, feedback and coverage, are about feasibility in principle, the third (scale) is about practice.

2.1 Feedback

Feedback is what gives direction to learning; without it, the process is merely one of self modification. In a closed system where the true purpose resides in the external observer, but can not be accessed directly, feedback can only come from a proxy. This creates the fundamental challenge for system-internal feedback is be aligned with the observer, and remain aligned throughout the process. It places a significant burden on the system at set-up time, with the most common pitfall being a poorly designed critic or reward function that becomes exploitable over time, resulting in a process that deviates from the observer's intent. RL's famed capability for self-correction is not applicable here: what can self-correct is behaviour given feedback, but not feedback itself. Additionally, ideal feedback should be efficient, i.e., contain enough information (not too sparse, not too noisy, not too delayed) for learning to be feasible within the time horizon of the system.