Step 1: Run code in a sandbox
Fernhill Outdoor's managers ask questions like "which region sold most in March?". An LLM cannot add up 1,400 rows in its head, but it can write pandas code that does. Your agent will run that code, so the first job is a place to run it where a mistake or a trick cannot hurt anything.
sandbox.py sets the limits: TIMEOUT seconds, MAX_OUTPUT characters back to the model, and limits(), which
caps memory and CPU. PRELUDE loads the table as df before the model's code runs.
1. Write run_code(code): run PRELUDE + code in a new Python process inside a temporary folder that holds
copies of orders.csv and prep.py. Give the process only a minimal set of environment variables. Return
{"ok": True, "output": stdout}, cut to MAX_OUTPUT with a note. On an error, return ok False and the last
lines of the error. On a timeout, return ok False and say it was stopped.
2. Run it. The script tries a normal query, an endless loop, a memory bomb, a typo, a huge print and a look at the environment variables.
sandbox.py, the file you edit59 lines
"""Runs Python code written by the model, away from the agent, with limits."""
import ast
import os
import shutil
import subprocess
import sys
import tempfile
DATA = "orders.csv"
TIMEOUT = 5 # seconds of wall time per run
MAX_OUTPUT = 2000 # characters of output returned to the model
MEMORY = 600 << 20 # bytes of address space for the code
PRELUDE = "import pandas as pd\nimport numpy as np\nfrom prep import prepare\ndf = prepare(pd.read_csv('orders.csv'))\n"
# ---------- Step 1: run code in a separate process ----------
def run_code(code):
"""Run `code` after PRELUDE in a fresh Python process, in a temporary folder holding copies of the data and
prep.py.
Returns {"ok": bool, "output": text}: stdout on success; on an error, the error's last lines."""
# TODO (Step 1): in a tempfile.TemporaryDirectory(), copy DATA (as orders.csv) and prep.py, write PRELUDE + code
# to job.py, and run [sys.executable, "job.py"] there with subprocess.run: cwd=the folder, capture_output=True,
# text=True, timeout=TIMEOUT, a minimal env (PATH, HOME=the folder, OPENBLAS_NUM_THREADS=1, OMP_NUM_THREADS=1)
# and preexec_fn=limits.
# Timeout -> {"ok": False, "output": "Stopped: ..."}; non-zero exit -> {"ok": False, "output": last 3 stderr lines};
# success -> {"ok": True, "output": stdout cut to MAX_OUTPUT with a note saying it was cut}.
raise NotImplementedError("Step 1: write run_code()")
def limits():
"""Runs in the child before the code starts: cap its memory and CPU."""
import resource
resource.setrlimit(resource.RLIMIT_AS, (MEMORY, MEMORY))
resource.setrlimit(resource.RLIMIT_CPU, (TIMEOUT, TIMEOUT))
# ---------- Step 4: refuse dangerous code before it runs ----------
ALLOWED_IMPORTS = {"pandas", "numpy", "math", "statistics", "datetime", "collections", "itertools", "re"}
BLOCKED_NAMES = {"open", "exec", "eval", "compile", "__import__", "input", "globals", "locals", "vars", "getattr",
"setattr", "delattr", "breakpoint", "exit", "quit"}
def check_code(code):
"""None if the code may run, else the reason it may not. Rules: it must parse; imports only from
ALLOWED_IMPORTS; no use of a BLOCKED_NAMES name; no attribute starting with an underscore; no file access
through pandas or numpy (attributes in FILE_ATTRS or starting with read_)."""
return None
FILE_ATTRS = {"to_csv", "to_excel", "to_json", "to_parquet", "to_pickle", "tofile", "save", "savetxt", "to_sql", "to_hdf",
"load", "loadtxt", "fromfile", "genfromtxt"}
def safe_run(code):
"""check_code(), then run_code()."""
reason = check_code(code)
if reason:
return {"ok": False, "output": f"Refused before running: {reason}. Only analyse df with pandas and print results."}
return run_code(code)DATA_NOTES.mdagent.pyharness.pyorders.csvprep.pyquestions.jsontry_it.py