{ "cells": [ { "cell_type": "markdown", "id": "signal-quest-01", "metadata": {}, "source": [ "# Signal Quest: Build an Honest Machine with an LLM\n", "\n", "**Dr. Mallarapu · SEAS-8414 · Research and paper-trading only**\n", "\n", "This capstone teaches the complete *Signal Quest* textbook as one reproducible investigation. You will define a settlement rule, freeze an evidence boundary, build causal features, compare models, calibrate probabilities, replay a conservative policy, and monitor the system. The data are synthetic. A good conclusion may be `NO_TRADE`.\n", "\n", "**Non-negotiable rule:** an LLM may draft bounded code, but it never receives secrets, production data, execution authority, or permission to run its own output. Every candidate is reviewed and tested before use." ] }, { "cell_type": "markdown", "id": "signal-quest-02", "metadata": {}, "source": [ "## Learning map\n", "\n", "| Textbook movement | Notebook evidence | LLM role |\n", "| --- | --- | --- |\n", "| ML, labels, and clocks | Synthetic event ledger and label contract | Draft a pure feature function |\n", "| Metrics and fair tests | Baseline, chronological split, calibration | Explain a failed test, not invent a result |\n", "| Three model families | Tabular baseline, encoder scaffold, causal sequence scaffold | Draft isolated modules and tests |\n", "| Replay and paper trading | Conservative cost-aware `NO_TRADE` policy | Propose a test case only |\n", "| Agentic monitoring | Read-only health checks and reversible containment | Summarize evidence for a human |\n", "\n", "**Completion bar:** all tests pass, the manifest is complete, the LLM trace is saved, and every conclusion is labeled `simulated`, `illustrative`, or `design target` as appropriate." ] }, { "cell_type": "code", "execution_count": 1, "id": "signal-quest-03", "metadata": { "execution": { "iopub.execute_input": "2026-08-03T19:40:33.797847Z", "iopub.status.busy": "2026-08-03T19:40:33.797783Z", "iopub.status.idle": "2026-08-03T19:40:34.012590Z", "shell.execute_reply": "2026-08-03T19:40:34.012059Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Working directory: $TEXTBOOK_ROOT/btc-polymarket-ml/labs/signal_quest_lab\n", "Epistemic status: simulated teaching lab; no live market or order authority.\n" ] } ], "source": [ "# Optional student environment. Run once in a fresh notebook kernel if needed.\n", "# %pip install --upgrade numpy pandas scikit-learn\n", "# Optional model-family extensions: %pip install catboost torch\n", "\n", "import ast\n", "import hashlib\n", "import json\n", "import os\n", "import textwrap\n", "from dataclasses import asdict, dataclass\n", "from datetime import datetime, timezone\n", "from pathlib import Path\n", "\n", "import numpy as np\n", "import pandas as pd\n", "\n", "SEED = 8414\n", "rng = np.random.default_rng(SEED)\n", "WORKDIR = Path(\"signal_quest_lab\")\n", "WORKDIR.mkdir(exist_ok=True)\n", "print(f\"Working directory: {WORKDIR.resolve()}\")\n", "print(\"Epistemic status: simulated teaching lab; no live market or order authority.\")\n", "\n", "\n" ] }, { "cell_type": "markdown", "id": "signal-quest-04", "metadata": {}, "source": [ "## 1. The contract comes before the model\n", "\n", "At decision time `t`, a model may receive only events whose **receive time** is no later than `t`. The label resolves later. Here, `UP` means the synthetic settlement price at the end of a five-minute horizon is greater than or equal to the start price. This is a teaching contract, not a claim about a real venue or market." ] }, { "cell_type": "code", "execution_count": 2, "id": "signal-quest-05", "metadata": { "execution": { "iopub.execute_input": "2026-08-03T19:40:34.014012Z", "iopub.status.busy": "2026-08-03T19:40:34.013895Z", "iopub.status.idle": "2026-08-03T19:40:34.017081Z", "shell.execute_reply": "2026-08-03T19:40:34.016635Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"horizon_minutes\": 5,\n", " \"tie_rule\": \"UP\",\n", " \"receive_time_required\": true,\n", " \"research_only\": true\n", "}\n" ] } ], "source": [ "@dataclass(frozen=True)\n", "class DecisionContract:\n", " horizon_minutes: int = 5\n", " tie_rule: str = \"UP\"\n", " receive_time_required: bool = True\n", " research_only: bool = True\n", "\n", "CONTRACT = DecisionContract()\n", "print(json.dumps(asdict(CONTRACT), indent=2))\n", "\n", "def label_up(start_price: float, end_price: float) -> int:\n", " \"\"\"Settlement contract: equality resolves UP.\"\"\"\n", " return int(end_price >= start_price)\n", "\n", "assert label_up(100.0, 100.0) == 1\n", "assert label_up(100.0, 99.9) == 0\n" ] }, { "cell_type": "markdown", "id": "signal-quest-06", "metadata": {}, "source": [ "## 2. Build an event ledger and stop time travel\n", "\n", "The ledger deliberately contains some events that occurred before a cutoff but arrived too late to be used. This distinction is the central causal test: occurrence time alone is not eligibility." ] }, { "cell_type": "code", "execution_count": 3, "id": "signal-quest-07", "metadata": { "execution": { "iopub.execute_input": "2026-08-03T19:40:34.018213Z", "iopub.status.busy": "2026-08-03T19:40:34.018158Z", "iopub.status.idle": "2026-08-03T19:40:34.025517Z", "shell.execute_reply": "2026-08-03T19:40:34.025033Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
event_timereceive_timemidspreadimbalanceevent_id
02026-01-01 00:00:00+00:002026-01-01 00:03:01+00:0099983.5771561.086732-0.531457evt-0000
12026-01-01 00:01:00+00:002026-01-01 00:01:00+00:0099958.9951440.978767-0.589098evt-0001
22026-01-01 00:02:00+00:002026-01-01 00:02:03+00:0099965.6238051.235367-0.837308evt-0002
32026-01-01 00:03:00+00:002026-01-01 00:03:03+00:0099976.5131672.715420-0.673456evt-0003
42026-01-01 00:04:00+00:002026-01-01 00:04:02+00:0099977.1722960.8278410.606708evt-0004
\n", "
" ], "text/plain": [ " event_time receive_time mid spread \\\n", "0 2026-01-01 00:00:00+00:00 2026-01-01 00:03:01+00:00 99983.577156 1.086732 \n", "1 2026-01-01 00:01:00+00:00 2026-01-01 00:01:00+00:00 99958.995144 0.978767 \n", "2 2026-01-01 00:02:00+00:00 2026-01-01 00:02:03+00:00 99965.623805 1.235367 \n", "3 2026-01-01 00:03:00+00:00 2026-01-01 00:03:03+00:00 99976.513167 2.715420 \n", "4 2026-01-01 00:04:00+00:00 2026-01-01 00:04:02+00:00 99977.172296 0.827841 \n", "\n", " imbalance event_id \n", "0 -0.531457 evt-0000 \n", "1 -0.589098 evt-0001 \n", "2 -0.837308 evt-0002 \n", "3 -0.673456 evt-0003 \n", "4 0.606708 evt-0004 " ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def synthetic_events(n: int = 720) -> pd.DataFrame:\n", " event_time = pd.date_range(\"2026-01-01\", periods=n, freq=\"min\", tz=\"UTC\")\n", " mid = 100_000 + np.cumsum(rng.normal(0, 14, n))\n", " spread = np.maximum(0.5, rng.lognormal(mean=0.4, sigma=0.35, size=n))\n", " imbalance = np.tanh(rng.normal(0, 0.9, n))\n", " receive_delay_seconds = rng.integers(0, 4, n)\n", " receive_delay_seconds[::71] += 180 # deliberately late records\n", " return pd.DataFrame({\n", " \"event_time\": event_time,\n", " \"receive_time\": event_time + pd.to_timedelta(receive_delay_seconds, unit=\"s\"),\n", " \"mid\": mid,\n", " \"spread\": spread,\n", " \"imbalance\": imbalance,\n", " \"event_id\": [f\"evt-{i:04d}\" for i in range(n)],\n", " })\n", "\n", "events = synthetic_events()\n", "events.head()\n" ] }, { "cell_type": "code", "execution_count": 4, "id": "signal-quest-08", "metadata": { "execution": { "iopub.execute_input": "2026-08-03T19:40:34.026639Z", "iopub.status.busy": "2026-08-03T19:40:34.026582Z", "iopub.status.idle": "2026-08-03T19:40:34.030126Z", "shell.execute_reply": "2026-08-03T19:40:34.029605Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Cutoff: 2026-01-01 01:12:00+00:00; eligible rows: 71\n" ] } ], "source": [ "def eligible_events(events: pd.DataFrame, cutoff: pd.Timestamp) -> pd.DataFrame:\n", " required = {\"event_time\", \"receive_time\", \"mid\", \"spread\", \"imbalance\", \"event_id\"}\n", " missing = required.difference(events.columns)\n", " if missing:\n", " raise ValueError(f\"Missing required evidence fields: {sorted(missing)}\")\n", " return events.loc[events[\"receive_time\"] <= cutoff].copy()\n", "\n", "cutoff = events.loc[72, \"event_time\"] # evt-0071 occurred but is received later.\n", "eligible = eligible_events(events, cutoff)\n", "assert (eligible[\"receive_time\"] <= cutoff).all()\n", "assert len(eligible) < len(events.loc[events[\"event_time\"] <= cutoff])\n", "print(f\"Cutoff: {cutoff}; eligible rows: {len(eligible)}\")\n" ] }, { "cell_type": "markdown", "id": "signal-quest-09", "metadata": {}, "source": [ "## 3. Ask the LLM for a bounded code draft\n", "\n", "The LLM receives a small, explicit contract. It must return **only one pure Python function** and tests. The notebook stores its response, hashes it, parses it, and refuses to execute it automatically. Set `SIGNAL_QUEST_USE_LLM=1` and `OLLAMA_API_KEY` only when you are ready to make an API request. Otherwise, the cell emits a reviewable placeholder.\n", "\n", "The prompt is outcome-first: required inputs, allowed output, safety boundary, and completion test are explicit.\n", "\n" ] }, { "cell_type": "code", "execution_count": 5, "id": "signal-quest-10", "metadata": { "execution": { "iopub.execute_input": "2026-08-03T19:40:34.031191Z", "iopub.status.busy": "2026-08-03T19:40:34.031134Z", "iopub.status.idle": "2026-08-03T19:40:34.036074Z", "shell.execute_reply": "2026-08-03T19:40:34.035588Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "review_required a31c43d91a6436fa799ea9d740e80d7ac9841fcf86376d36620b0194e7aa771c\n" ] } ], "source": [ "OLLAMA_MODEL = os.getenv(\"OLLAMA_MODEL\", \"kimi-k3:cloud\")\n", "OLLAMA_HOST = os.getenv(\"OLLAMA_HOST\", \"https://ollama.com\").rstrip(\"/\")\n", "USE_LLM = os.getenv(\"SIGNAL_QUEST_USE_LLM\", \"0\") == \"1\"\n", "\n", "FEATURE_PROMPT = \"\"\"\n", "Mission: draft one pure Python function named build_features(rows).\n", "\n", "Input contract:\n", "- rows is a pandas DataFrame with mid, spread, imbalance, receive_time, and event_time.\n", "- rows contains only events eligible at a decision cutoff.\n", "\n", "Output contract:\n", "- Return a dict with float keys: last_mid, mean_spread, mean_imbalance, and return_3.\n", "- Raise ValueError for fewer than four rows or missing fields.\n", "\n", "Hard rules:\n", "- Use no imports, file access, network calls, randomness, globals, eval, exec, classes, or model fitting.\n", "- Do not mention market performance or trading.\n", "- Return code only, followed by exactly three assert-style tests in comments.\n", "\n", "Completion bar: a reviewer can read the function, parse it, and test it deterministically.\n", "\"\"\".strip()\n", "\n", "def request_llm_code(prompt: str) -> str:\n", " \"\"\"Request an untrusted teaching-code draft from Ollama.\"\"\"\n", " if not USE_LLM:\n", " return \"# LLM disabled. Set SIGNAL_QUEST_USE_LLM=1 after reviewing the prompt.\\n\"\n", " api_key = os.getenv(\"OLLAMA_API_KEY\")\n", " if not api_key:\n", " raise RuntimeError(\"LLM mode requested but OLLAMA_API_KEY is not set.\")\n", " import urllib.error\n", " import urllib.request\n", "\n", " payload = json.dumps({\n", " \"model\": OLLAMA_MODEL,\n", " \"messages\": [{\"role\": \"user\", \"content\": prompt}],\n", " \"stream\": False,\n", " \"options\": {\"temperature\": 0},\n", " }).encode(\"utf-8\")\n", " request = urllib.request.Request(\n", " f\"{OLLAMA_HOST}/api/chat\",\n", " data=payload,\n", " headers={\"Content-Type\": \"application/json\", \"Authorization\": f\"Bearer {api_key}\"},\n", " method=\"POST\",\n", " )\n", " try:\n", " with urllib.request.urlopen(request, timeout=120) as response:\n", " result = json.loads(response.read().decode(\"utf-8\"))\n", " except urllib.error.HTTPError as exc:\n", " raise RuntimeError(f\"Ollama request failed with HTTP {exc.code}.\") from exc\n", " except urllib.error.URLError as exc:\n", " raise RuntimeError(f\"Ollama is unreachable at {OLLAMA_HOST}.\") from exc\n", " content = result.get(\"message\", {}).get(\"content\")\n", " if not isinstance(content, str) or not content.strip():\n", " raise RuntimeError(\"Ollama returned no message content.\")\n", " return content\n", "\n", "llm_draft = request_llm_code(FEATURE_PROMPT)\n", "trace = {\n", " \"timestamp_utc\": datetime.now(timezone.utc).isoformat(),\n", " \"provider\": \"ollama\",\n", " \"host\": OLLAMA_HOST,\n", " \"model\": OLLAMA_MODEL if USE_LLM else \"disabled\",\n", " \"prompt\": FEATURE_PROMPT,\n", " \"response\": llm_draft,\n", " \"response_sha256\": hashlib.sha256(llm_draft.encode()).hexdigest(),\n", " \"status\": \"review_required\",\n", "}\n", "(WORKDIR / \"llm_feature_draft.json\").write_text(json.dumps(trace, indent=2))\n", "print(trace[\"status\"], trace[\"response_sha256\"])\n", "\n" ] }, { "cell_type": "code", "execution_count": 6, "id": "signal-quest-11", "metadata": { "execution": { "iopub.execute_input": "2026-08-03T19:40:34.037157Z", "iopub.status.busy": "2026-08-03T19:40:34.037092Z", "iopub.status.idle": "2026-08-03T19:40:34.040383Z", "shell.execute_reply": "2026-08-03T19:40:34.039889Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['LLM disabled: no candidate submitted']\n", "Human gate: copy an approved candidate into the next cell; do not exec model output.\n" ] } ], "source": [ "# The LLM's text is data. It is not executed. Validate its AST before human review.\n", "FORBIDDEN = (ast.Import, ast.ImportFrom, ast.Global, ast.Nonlocal, ast.ClassDef, ast.AsyncFunctionDef)\n", "FORBIDDEN_CALLS = {\"eval\", \"exec\", \"open\", \"compile\", \"__import__\", \"input\"}\n", "\n", "def static_code_review(candidate: str) -> list[str]:\n", " findings = []\n", " try:\n", " tree = ast.parse(candidate)\n", " except SyntaxError as exc:\n", " return [f\"syntax error: {exc}\"]\n", " functions = [n for n in tree.body if isinstance(n, ast.FunctionDef)]\n", " if len(functions) != 1 or functions[0].name != \"build_features\":\n", " findings.append(\"require exactly one function named build_features\")\n", " for node in ast.walk(tree):\n", " if isinstance(node, FORBIDDEN):\n", " findings.append(f\"forbidden syntax: {type(node).__name__}\")\n", " if isinstance(node, ast.Call) and isinstance(node.func, ast.Name) and node.func.id in FORBIDDEN_CALLS:\n", " findings.append(f\"forbidden call: {node.func.id}\")\n", " return sorted(set(findings))\n", "\n", "review_findings = static_code_review(llm_draft) if USE_LLM else [\"LLM disabled: no candidate submitted\"]\n", "print(review_findings)\n", "print(\"Human gate: copy an approved candidate into the next cell; do not exec model output.\")\n" ] }, { "cell_type": "code", "execution_count": 7, "id": "signal-quest-12", "metadata": { "execution": { "iopub.execute_input": "2026-08-03T19:40:34.041425Z", "iopub.status.busy": "2026-08-03T19:40:34.041370Z", "iopub.status.idle": "2026-08-03T19:40:34.044403Z", "shell.execute_reply": "2026-08-03T19:40:34.043914Z" } }, "outputs": [], "source": [ "# Student-controlled implementation after review. This is the only feature code used below.\n", "def build_features(rows: pd.DataFrame) -> dict[str, float]:\n", " required = {\"mid\", \"spread\", \"imbalance\"}\n", " if missing := required.difference(rows.columns):\n", " raise ValueError(f\"missing fields: {sorted(missing)}\")\n", " if len(rows) < 4:\n", " raise ValueError(\"need at least four rows\")\n", " mid = rows[\"mid\"].astype(float)\n", " return {\n", " \"last_mid\": float(mid.iloc[-1]),\n", " \"mean_spread\": float(rows[\"spread\"].mean()),\n", " \"mean_imbalance\": float(rows[\"imbalance\"].mean()),\n", " \"return_3\": float(mid.iloc[-1] / mid.iloc[-4] - 1.0),\n", " }\n", "\n", "assert set(build_features(eligible.tail(10))) == {\"last_mid\", \"mean_spread\", \"mean_imbalance\", \"return_3\"}\n", "try:\n", " build_features(eligible.head(3))\n", " raise AssertionError(\"short-window test did not fail\")\n", "except ValueError:\n", " pass\n" ] }, { "cell_type": "markdown", "id": "be561b6d", "metadata": {}, "source": [ "## 3A. On-demand Code Studio\n", "\n", "This is the notebook's LLM workbench for the entire book. Choose one topic, write a narrow learning request, and set `SIGNAL_QUEST_USE_LLM=1` only after you have read the generated prompt. The LLM may draft a small teaching artifact; it cannot access files, networks, credentials, live markets, or a trading account. Its draft is saved with a trace and is never executed automatically.\n" ] }, { "cell_type": "code", "execution_count": 8, "id": "fbd5067f", "metadata": { "execution": { "iopub.execute_input": "2026-08-03T19:40:34.045563Z", "iopub.status.busy": "2026-08-03T19:40:34.045500Z", "iopub.status.idle": "2026-08-03T19:40:34.048489Z", "shell.execute_reply": "2026-08-03T19:40:34.048071Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Available topics: contract, features, leakage, catboost, encoder, transformer, calibration, replay, guardian\n" ] } ], "source": [ "LESSON_CATALOG = {\n", " \"contract\": {\n", " \"chapter\": \"Observations, decision time, later settlement\",\n", " \"function\": \"define_decision_contract\",\n", " \"goal\": \"Create a dataclass that rejects a label horizon, tie rule, or evidence boundary that is missing.\",\n", " },\n", " \"features\": {\n", " \"chapter\": \"Causal features and the evidence boundary\",\n", " \"function\": \"build_causal_features\",\n", " \"goal\": \"Create pure rolling features from rows already eligible at a decision cutoff.\",\n", " },\n", " \"leakage\": {\n", " \"chapter\": \"Time travel and leakage tests\",\n", " \"function\": \"assert_no_future_events\",\n", " \"goal\": \"Create a test that fails when an event received after the cutoff is present.\",\n", " },\n", " \"catboost\": {\n", " \"chapter\": \"Tabular probability model\",\n", " \"function\": \"train_tabular_candidate\",\n", " \"goal\": \"Draft a chronological CatBoost training function that returns validation probabilities only.\",\n", " },\n", " \"encoder\": {\n", " \"chapter\": \"Self-supervised limit-order-book encoder\",\n", " \"function\": \"masked_window_loss\",\n", " \"goal\": \"Draft a small tensor-only masked reconstruction loss with shape checks.\",\n", " },\n", " \"transformer\": {\n", " \"chapter\": \"Causal limit-order-book Transformer\",\n", " \"function\": \"causal_attention_mask\",\n", " \"goal\": \"Create a strictly upper-triangular attention mask for an ordered sequence.\",\n", " },\n", " \"calibration\": {\n", " \"chapter\": \"Probabilities, calibration, and abstention\",\n", " \"function\": \"calibration_summary\",\n", " \"goal\": \"Calculate reliability-bin summaries without choosing an action.\",\n", " },\n", " \"replay\": {\n", " \"chapter\": \"Conservative replay and costs\",\n", " \"function\": \"paper_replay_row\",\n", " \"goal\": \"Score one paper-trade record with supplied price, conservative cost, and later outcome.\",\n", " },\n", " \"guardian\": {\n", " \"chapter\": \"Agentic monitoring and reversible containment\",\n", " \"function\": \"guardian_check\",\n", " \"goal\": \"Return an evidence-only incident record; never modify a model, policy, or account.\",\n", " },\n", "}\n", "\n", "print(\"Available topics:\", \", \".join(LESSON_CATALOG))\n" ] }, { "cell_type": "code", "execution_count": 9, "id": "3722513c", "metadata": { "execution": { "iopub.execute_input": "2026-08-03T19:40:34.049447Z", "iopub.status.busy": "2026-08-03T19:40:34.049394Z", "iopub.status.idle": "2026-08-03T19:40:34.052783Z", "shell.execute_reply": "2026-08-03T19:40:34.052424Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "review_required leakage a31c43d91a6436fa799ea9d740e80d7ac9841fcf86376d36620b0194e7aa771c\n" ] } ], "source": [ "ON_DEMAND_TOPIC = \"leakage\" # Change to any key in LESSON_CATALOG.\n", "ON_DEMAND_REQUEST = \"Explain the idea with one small, deterministic function and three assert-style tests.\"\n", "\n", "def on_demand_prompt(topic: str, learner_request: str) -> str:\n", " if topic not in LESSON_CATALOG:\n", " raise ValueError(f\"Unknown topic {topic!r}. Choose one of: {sorted(LESSON_CATALOG)}\")\n", " lesson = LESSON_CATALOG[topic]\n", " return f\"\"\"\n", "You are the Signal Quest teaching-code assistant. Produce one small, reviewable Python teaching artifact.\n", "\n", "Book chapter: {lesson['chapter']}\n", "Required function name: {lesson['function']}\n", "Teaching goal: {lesson['goal']}\n", "Student request: {learner_request}\n", "\n", "Hard safety and evidence rules:\n", "- Return code only, followed by exactly three assert-style tests in comments.\n", "- Define exactly one function. Use only parameters passed to that function.\n", "- No imports, files, network, environment variables, randomness, subprocesses, eval, exec, classes, model persistence, live data, or order placement.\n", "- Do not claim predictive performance, profitability, or deployment readiness.\n", "- For model topics, return probabilities or tensors only; do not make a trading decision.\n", "- For guardian topics, return observations and a human-review requirement only.\n", "\n", "The output will be statically reviewed by a human before any manual copy into an approved cell.\n", "\"\"\".strip()\n", "\n", "ON_DEMAND_PROMPT = on_demand_prompt(ON_DEMAND_TOPIC, ON_DEMAND_REQUEST)\n", "on_demand_draft = request_llm_code(ON_DEMAND_PROMPT)\n", "on_demand_trace = {\n", " \"topic\": ON_DEMAND_TOPIC,\n", " \"chapter\": LESSON_CATALOG[ON_DEMAND_TOPIC][\"chapter\"],\n", " \"provider\": \"ollama\",\n", " \"model\": OLLAMA_MODEL if USE_LLM else \"disabled\",\n", " \"prompt\": ON_DEMAND_PROMPT,\n", " \"response\": on_demand_draft,\n", " \"response_sha256\": hashlib.sha256(on_demand_draft.encode()).hexdigest(),\n", " \"status\": \"review_required\",\n", "}\n", "(WORKDIR / \"llm_on_demand_code_trace.json\").write_text(json.dumps(on_demand_trace, indent=2))\n", "print(on_demand_trace[\"status\"], on_demand_trace[\"topic\"], on_demand_trace[\"response_sha256\"])\n", "\n" ] }, { "cell_type": "code", "execution_count": 10, "id": "85ed9e02", "metadata": { "execution": { "iopub.execute_input": "2026-08-03T19:40:34.053841Z", "iopub.status.busy": "2026-08-03T19:40:34.053787Z", "iopub.status.idle": "2026-08-03T19:40:34.057310Z", "shell.execute_reply": "2026-08-03T19:40:34.056866Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['LLM disabled: set SIGNAL_QUEST_USE_LLM=1 and supply OLLAMA_API_KEY.']\n", "Human gate: inspect the trace, review the draft, then manually copy an approved function below.\n" ] } ], "source": [ "ON_DEMAND_FORBIDDEN = {\n", " \"eval\", \"exec\", \"open\", \"compile\", \"__import__\", \"input\", \"requests\", \"urlopen\",\n", " \"subprocess\", \"system\", \"popen\", \"fit\", \"predict\", \"save\", \"load\",\n", "}\n", "\n", "def review_on_demand_draft(candidate: str, required_function: str) -> list[str]:\n", " try:\n", " tree = ast.parse(candidate)\n", " except SyntaxError as exc:\n", " return [f\"syntax error: {exc}\"]\n", " findings = []\n", " functions = [node for node in tree.body if isinstance(node, ast.FunctionDef)]\n", " if len(functions) != 1 or functions[0].name != required_function:\n", " findings.append(f\"require exactly one function named {required_function}\")\n", " for node in ast.walk(tree):\n", " if isinstance(node, FORBIDDEN):\n", " findings.append(f\"forbidden syntax: {type(node).__name__}\")\n", " if isinstance(node, ast.Attribute) and node.attr.startswith(\"__\"):\n", " findings.append(\"forbidden dunder attribute\")\n", " if isinstance(node, ast.Call):\n", " name = node.func.id if isinstance(node.func, ast.Name) else getattr(node.func, \"attr\", \"\")\n", " if name in ON_DEMAND_FORBIDDEN:\n", " findings.append(f\"forbidden call: {name}\")\n", " return sorted(set(findings))\n", "\n", "if USE_LLM:\n", " on_demand_findings = review_on_demand_draft(\n", " on_demand_draft, LESSON_CATALOG[ON_DEMAND_TOPIC][\"function\"]\n", " )\n", "else:\n", " on_demand_findings = [\"LLM disabled: set SIGNAL_QUEST_USE_LLM=1 and supply OLLAMA_API_KEY.\"]\n", "print(on_demand_findings)\n", "print(\"Human gate: inspect the trace, review the draft, then manually copy an approved function below.\")\n", "\n" ] }, { "cell_type": "markdown", "id": "0c46d27c", "metadata": {}, "source": [ "### Human approval gate\n", "\n", "The LLM draft is not program behavior. Read the saved trace, check the static findings, test the artifact in isolation, and then manually copy only an approved version into a new cell. If the draft fails the evidence boundary or safety contract, reject it and improve the prompt—do not weaken the guardrail.\n" ] }, { "cell_type": "markdown", "id": "signal-quest-13", "metadata": {}, "source": [ "## 4. Freeze features, then define a later label\n", "\n", "Each row below is a decision record. Features are built from an eligible history at the cutoff. The label is resolved five minutes later. The notebook discards unresolved horizons instead of guessing." ] }, { "cell_type": "code", "execution_count": 11, "id": "signal-quest-14", "metadata": { "execution": { "iopub.execute_input": "2026-08-03T19:40:34.058370Z", "iopub.status.busy": "2026-08-03T19:40:34.058318Z", "iopub.status.idle": "2026-08-03T19:40:34.240939Z", "shell.execute_reply": "2026-08-03T19:40:34.240438Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
last_midmean_spreadmean_imbalancereturn_3cutofflabel
099960.5887251.5352990.0499900.0000862026-01-01 00:20:00+00:000
199976.5969721.6155990.0836960.0001992026-01-01 00:21:00+00:001
299965.5853311.6776180.0806410.0001442026-01-01 00:22:00+00:000
399978.6882971.6751010.1207640.0001812026-01-01 00:23:00+00:001
499954.3782321.5833420.110286-0.0002222026-01-01 00:24:00+00:001
\n", "
" ], "text/plain": [ " last_mid mean_spread mean_imbalance return_3 \\\n", "0 99960.588725 1.535299 0.049990 0.000086 \n", "1 99976.596972 1.615599 0.083696 0.000199 \n", "2 99965.585331 1.677618 0.080641 0.000144 \n", "3 99978.688297 1.675101 0.120764 0.000181 \n", "4 99954.378232 1.583342 0.110286 -0.000222 \n", "\n", " cutoff label \n", "0 2026-01-01 00:20:00+00:00 0 \n", "1 2026-01-01 00:21:00+00:00 1 \n", "2 2026-01-01 00:22:00+00:00 0 \n", "3 2026-01-01 00:23:00+00:00 1 \n", "4 2026-01-01 00:24:00+00:00 1 " ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def make_dataset(events: pd.DataFrame, lookback: int = 20, horizon: int = 5) -> pd.DataFrame:\n", " rows = []\n", " for i in range(lookback, len(events) - horizon):\n", " cutoff = events.loc[i, \"event_time\"]\n", " history = eligible_events(events.iloc[: i + 1], cutoff).tail(lookback)\n", " if len(history) < lookback:\n", " continue\n", " features = build_features(history)\n", " start_mid = float(events.loc[i, \"mid\"])\n", " end_mid = float(events.loc[i + horizon, \"mid\"])\n", " rows.append({**features, \"cutoff\": cutoff, \"label\": label_up(start_mid, end_mid)})\n", " return pd.DataFrame(rows)\n", "\n", "dataset = make_dataset(events)\n", "assert dataset[\"cutoff\"].is_monotonic_increasing\n", "assert dataset[\"label\"].isin([0, 1]).all()\n", "dataset.head()\n" ] }, { "cell_type": "markdown", "id": "signal-quest-15", "metadata": {}, "source": [ "## 5. Evaluate fairly: chronological split, simple baseline, and calibration\n", "\n", "Random splitting would let nearby observations leak across train and test. We use time order. The baseline is the training prevalence; a score only earns attention if it beats that reference under the same evidence contract." ] }, { "cell_type": "code", "execution_count": 12, "id": "signal-quest-16", "metadata": { "execution": { "iopub.execute_input": "2026-08-03T19:40:34.242140Z", "iopub.status.busy": "2026-08-03T19:40:34.242069Z", "iopub.status.idle": "2026-08-03T19:40:34.245279Z", "shell.execute_reply": "2026-08-03T19:40:34.244838Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'train': 417, 'valid': 139, 'test': 139, 'base_rate': 0.4460431654676259, 'baseline_brier': 0.2571813053154599}\n" ] } ], "source": [ "def chronological_split(frame: pd.DataFrame, train_fraction: float = 0.60, valid_fraction: float = 0.20):\n", " n = len(frame)\n", " a, b = int(n * train_fraction), int(n * (train_fraction + valid_fraction))\n", " return frame.iloc[:a].copy(), frame.iloc[a:b].copy(), frame.iloc[b:].copy()\n", "\n", "train, valid, test = chronological_split(dataset)\n", "base_probability = float(train[\"label\"].mean())\n", "\n", "def brier(y: pd.Series, p: np.ndarray) -> float:\n", " return float(np.mean((np.asarray(y) - np.asarray(p)) ** 2))\n", "\n", "baseline_brier = brier(test[\"label\"], np.full(len(test), base_probability))\n", "print({\"train\": len(train), \"valid\": len(valid), \"test\": len(test), \"base_rate\": base_probability, \"baseline_brier\": baseline_brier})\n" ] }, { "cell_type": "code", "execution_count": 13, "id": "signal-quest-17", "metadata": { "execution": { "iopub.execute_input": "2026-08-03T19:40:34.246322Z", "iopub.status.busy": "2026-08-03T19:40:34.246261Z", "iopub.status.idle": "2026-08-03T19:40:34.874105Z", "shell.execute_reply": "2026-08-03T19:40:34.873583Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Validation Brier: 0.25171696377318237\n", "Test Brier: 0.2710190132809431\n", "Test accuracy: 0.39568345323741005\n", "Confusion matrix:\n", " [[44 20]\n", " [64 11]]\n", "Interpretation: simulated result only. Accuracy alone is not a claim of usefulness.\n" ] } ], "source": [ "# A small transparent research baseline, not a production model.\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.metrics import accuracy_score, confusion_matrix\n", "\n", "FEATURES = [\"mean_spread\", \"mean_imbalance\", \"return_3\"]\n", "baseline_model = LogisticRegression(random_state=SEED, class_weight=\"balanced\")\n", "baseline_model.fit(train[FEATURES], train[\"label\"])\n", "valid_probability = baseline_model.predict_proba(valid[FEATURES])[:, 1]\n", "test_probability = baseline_model.predict_proba(test[FEATURES])[:, 1]\n", "\n", "print(\"Validation Brier:\", brier(valid[\"label\"], valid_probability))\n", "print(\"Test Brier:\", brier(test[\"label\"], test_probability))\n", "print(\"Test accuracy:\", accuracy_score(test[\"label\"], test_probability >= 0.5))\n", "print(\"Confusion matrix:\\n\", confusion_matrix(test[\"label\"], test_probability >= 0.5))\n", "print(\"Interpretation: simulated result only. Accuracy alone is not a claim of usefulness.\")\n" ] }, { "cell_type": "code", "execution_count": 14, "id": "signal-quest-18", "metadata": { "execution": { "iopub.execute_input": "2026-08-03T19:40:34.875426Z", "iopub.status.busy": "2026-08-03T19:40:34.875312Z", "iopub.status.idle": "2026-08-03T19:40:34.882743Z", "shell.execute_reply": "2026-08-03T19:40:34.882297Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
bincountpredictedobserved
0(-0.001, 0.2]0NaNNaN
1(0.2, 0.4]0NaNNaN
2(0.4, 0.6]1390.4909950.388489
3(0.6, 0.8]0NaNNaN
4(0.8, 1.0]0NaNNaN
\n", "
" ], "text/plain": [ " bin count predicted observed\n", "0 (-0.001, 0.2] 0 NaN NaN\n", "1 (0.2, 0.4] 0 NaN NaN\n", "2 (0.4, 0.6] 139 0.490995 0.388489\n", "3 (0.6, 0.8] 0 NaN NaN\n", "4 (0.8, 1.0] 0 NaN NaN" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def calibration_table(y: pd.Series, p: np.ndarray, bins: int = 5) -> pd.DataFrame:\n", " frame = pd.DataFrame({\"y\": np.asarray(y), \"p\": np.asarray(p)})\n", " frame[\"bin\"] = pd.cut(frame[\"p\"], bins=np.linspace(0, 1, bins + 1), include_lowest=True)\n", " return frame.groupby(\"bin\", observed=False).agg(count=(\"y\", \"size\"), predicted=(\"p\", \"mean\"), observed=(\"y\", \"mean\")).reset_index()\n", "\n", "calibration_table(valid[\"label\"], valid_probability)\n" ] }, { "cell_type": "markdown", "id": "signal-quest-19", "metadata": {}, "source": [ "## 6. The three-model ladder\n", "\n", "The first rung is a tabular candidate. The second learns a representation from permitted unlabeled sequence windows. The third applies a causal mask to ordered limit-order-book states. None proves an edge by existing. Each must survive the same temporal, calibration, replay, and safety gates.\n", "\n", "Use the LLM to draft one module at a time from the prompts below. Review the output exactly as you reviewed `build_features`; do not paste generated code directly into a training run." ] }, { "cell_type": "code", "execution_count": 15, "id": "signal-quest-20", "metadata": { "execution": { "iopub.execute_input": "2026-08-03T19:40:34.883967Z", "iopub.status.busy": "2026-08-03T19:40:34.883903Z", "iopub.status.idle": "2026-08-03T19:40:34.886232Z", "shell.execute_reply": "2026-08-03T19:40:34.885812Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "CATBOOST PROMPT\n", "Draft a single Python function train_tabular_candidate(X_train, y_train, X_valid, y_valid). Use\n", "CatBoostClassifier only; no file, network, or plotting access. Return model and validation\n", "probabilities. Include an early-stopping parameter and do not claim performance.\n", "\n", "ENCODER PROMPT\n", "Draft a PyTorch nn.Module named MaskedLOBEncoder. It accepts [batch, time, features], masks only\n", "supplied positions, and returns embeddings. Do not create data loaders, files, or training loops.\n", "Include shape assertions in comments.\n", "\n", "CAUSAL_TRANSFORMER PROMPT\n", "Draft a PyTorch nn.Module named CausalLOBTransformer. It accepts ordered [batch, time, features]\n", "data and applies a strictly upper-triangular causal attention mask. Return one probability logit per\n", "sequence. No file, network, or execution code.\n" ] } ], "source": [ "MODEL_PROMPTS = {\n", " \"catboost\": \"Draft a single Python function train_tabular_candidate(X_train, y_train, X_valid, y_valid). Use CatBoostClassifier only; no file, network, or plotting access. Return model and validation probabilities. Include an early-stopping parameter and do not claim performance.\",\n", " \"encoder\": \"Draft a PyTorch nn.Module named MaskedLOBEncoder. It accepts [batch, time, features], masks only supplied positions, and returns embeddings. Do not create data loaders, files, or training loops. Include shape assertions in comments.\",\n", " \"causal_transformer\": \"Draft a PyTorch nn.Module named CausalLOBTransformer. It accepts ordered [batch, time, features] data and applies a strictly upper-triangular causal attention mask. Return one probability logit per sequence. No file, network, or execution code.\",\n", "}\n", "for name, prompt in MODEL_PROMPTS.items():\n", " print(f\"\\n{name.upper()} PROMPT\\n{textwrap.fill(prompt, width=100)}\")\n" ] }, { "cell_type": "code", "execution_count": 16, "id": "signal-quest-21", "metadata": { "execution": { "iopub.execute_input": "2026-08-03T19:40:34.887295Z", "iopub.status.busy": "2026-08-03T19:40:34.887234Z", "iopub.status.idle": "2026-08-03T19:40:34.889318Z", "shell.execute_reply": "2026-08-03T19:40:34.888910Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ ] chronological train/validation/test split is recorded\n", "[ ] normalization fit uses training period only\n", "[ ] sequence window contains no record received after its cutoff\n", "[ ] causal mask blocks every future position\n", "[ ] probabilities are calibrated on validation data only\n", "[ ] comparison includes baseline and conservative replay\n", "[ ] artifact manifest records code, config, data, and model hashes\n" ] } ], "source": [ "# Design-target acceptance tests for every advanced candidate.\n", "# These tests are intentionally stated before implementation.\n", "ADVANCED_MODEL_GATES = [\n", " \"chronological train/validation/test split is recorded\",\n", " \"normalization fit uses training period only\",\n", " \"sequence window contains no record received after its cutoff\",\n", " \"causal mask blocks every future position\",\n", " \"probabilities are calibrated on validation data only\",\n", " \"comparison includes baseline and conservative replay\",\n", " \"artifact manifest records code, config, data, and model hashes\",\n", "]\n", "for gate in ADVANCED_MODEL_GATES:\n", " print(\"[ ]\", gate)\n" ] }, { "cell_type": "markdown", "id": "signal-quest-22", "metadata": {}, "source": [ "## 7. Probability is not permission\n", "\n", "A model probability becomes a possible action only after conservative cost, calibration, data-health, and risk gates. The policy below is deliberately fail-closed: missing evidence, a bad calibration result, or an unsafe cost assumption produces `NO_TRADE`. It does not place orders." ] }, { "cell_type": "code", "execution_count": 17, "id": "signal-quest-23", "metadata": { "execution": { "iopub.execute_input": "2026-08-03T19:40:34.890433Z", "iopub.status.busy": "2026-08-03T19:40:34.890378Z", "iopub.status.idle": "2026-08-03T19:40:34.893388Z", "shell.execute_reply": "2026-08-03T19:40:34.892983Z" } }, "outputs": [], "source": [ "@dataclass(frozen=True)\n", "class PolicyInput:\n", " probability_up: float\n", " executable_up_price: float\n", " conservative_cost: float\n", " calibration_ok: bool\n", " data_health_ok: bool\n", " risk_ok: bool\n", "\n", "def decide_paper_only(x: PolicyInput, buffer: float = 0.03) -> str:\n", " if not all([x.calibration_ok, x.data_health_ok, x.risk_ok]):\n", " return \"NO_TRADE\"\n", " conservative_probability = x.probability_up - buffer\n", " return \"PAPER_UP\" if conservative_probability > x.executable_up_price + x.conservative_cost else \"NO_TRADE\"\n", "\n", "assert decide_paper_only(PolicyInput(.80, .60, .02, True, True, True)) == \"PAPER_UP\"\n", "assert decide_paper_only(PolicyInput(.80, .60, .02, False, True, True)) == \"NO_TRADE\"\n" ] }, { "cell_type": "markdown", "id": "signal-quest-24", "metadata": {}, "source": [ "## 8. Replay the policy under a reconstructed clock\n", "\n", "This minimal replay uses synthetic executable prices and conservative costs. It does not model a live book and it does not predict a real outcome. Its purpose is to make assumptions inspectable and to demonstrate that a score may still lead to abstention." ] }, { "cell_type": "code", "execution_count": 18, "id": "signal-quest-25", "metadata": { "execution": { "iopub.execute_input": "2026-08-03T19:40:34.894491Z", "iopub.status.busy": "2026-08-03T19:40:34.894435Z", "iopub.status.idle": "2026-08-03T19:40:34.900995Z", "shell.execute_reply": "2026-08-03T19:40:34.900583Z" } }, "outputs": [ { "data": { "text/plain": [ "decision\n", "NO_TRADE 139\n", "Name: count, dtype: int64" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def paper_replay(frame: pd.DataFrame, probabilities: np.ndarray, cost: float = 0.02) -> pd.DataFrame:\n", " ledger = []\n", " for (_, row), probability in zip(frame.iterrows(), probabilities):\n", " synthetic_price = float(np.clip(0.50 + 4 * row[\"return_3\"], 0.05, 0.95))\n", " decision = decide_paper_only(PolicyInput(float(probability), synthetic_price, cost, True, True, True))\n", " ledger.append({\n", " \"cutoff\": row[\"cutoff\"],\n", " \"probability_up\": float(probability),\n", " \"synthetic_executable_price\": synthetic_price,\n", " \"cost\": cost,\n", " \"decision\": decision,\n", " \"later_label\": int(row[\"label\"]),\n", " \"epistemic_status\": \"simulated\",\n", " })\n", " return pd.DataFrame(ledger)\n", "\n", "ledger = paper_replay(test, test_probability)\n", "assert set(ledger[\"decision\"]) <= {\"PAPER_UP\", \"NO_TRADE\"}\n", "ledger[\"decision\"].value_counts(dropna=False)\n" ] }, { "cell_type": "markdown", "id": "signal-quest-26", "metadata": {}, "source": [ "## 9. Agentic monitoring: observe, contain, escalate\n", "\n", "The guardian may summarize evidence, quarantine a bad artifact, and request human review. It may not alter a model, policy, dataset, account, or external system. The LLM may draft a human-readable incident summary only from the provided facts." ] }, { "cell_type": "code", "execution_count": 19, "id": "signal-quest-27", "metadata": { "execution": { "iopub.execute_input": "2026-08-03T19:40:34.902043Z", "iopub.status.busy": "2026-08-03T19:40:34.901983Z", "iopub.status.idle": "2026-08-03T19:40:34.904721Z", "shell.execute_reply": "2026-08-03T19:40:34.904285Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"status\": \"HEALTHY\",\n", " \"issues\": [],\n", " \"allowed_actions\": [\n", " \"write_incident_record\",\n", " \"quarantine_research_artifact\",\n", " \"request_human_review\"\n", " ],\n", " \"forbidden_actions\": [\n", " \"place_order\",\n", " \"change_model\",\n", " \"change_policy\",\n", " \"delete_evidence\"\n", " ]\n", "}\n" ] } ], "source": [ "def guardian_check(frame: pd.DataFrame, calibration_ok: bool, manifest_ok: bool) -> dict:\n", " issues = []\n", " if not frame[\"cutoff\"].is_monotonic_increasing:\n", " issues.append(\"non_monotonic_cutoff\")\n", " if not calibration_ok:\n", " issues.append(\"calibration_gate_failed\")\n", " if not manifest_ok:\n", " issues.append(\"artifact_manifest_missing\")\n", " return {\n", " \"status\": \"HEALTHY\" if not issues else \"QUARANTINE_AND_ESCALATE\",\n", " \"issues\": issues,\n", " \"allowed_actions\": [\"write_incident_record\", \"quarantine_research_artifact\", \"request_human_review\"],\n", " \"forbidden_actions\": [\"place_order\", \"change_model\", \"change_policy\", \"delete_evidence\"],\n", " }\n", "\n", "guardian = guardian_check(ledger, calibration_ok=True, manifest_ok=True)\n", "print(json.dumps(guardian, indent=2))\n" ] }, { "cell_type": "code", "execution_count": 20, "id": "signal-quest-28", "metadata": { "execution": { "iopub.execute_input": "2026-08-03T19:40:34.905872Z", "iopub.status.busy": "2026-08-03T19:40:34.905815Z", "iopub.status.idle": "2026-08-03T19:40:34.908387Z", "shell.execute_reply": "2026-08-03T19:40:34.907951Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Saved bounded incident-summary prompt; no remediation is invoked.\n" ] } ], "source": [ "INCIDENT_PROMPT = \"\"\"\n", "Summarize this research incident for a human reviewer. Use only supplied facts.\n", "State: observed evidence, allowed reversible containment, required human decision, and what cannot be inferred.\n", "Do not recommend trading, policy changes, or automatic remediation.\n", "Facts: {facts}\n", "\"\"\".strip()\n", "\n", "incident_trace = {\n", " \"prompt\": INCIDENT_PROMPT.format(facts=json.dumps(guardian)),\n", " \"allowed_output\": \"human-readable incident summary\",\n", " \"requires_human_review\": True,\n", " \"epistemic_status\": \"design target\",\n", "}\n", "(WORKDIR / \"guardian_llm_prompt.json\").write_text(json.dumps(incident_trace, indent=2))\n", "print(\"Saved bounded incident-summary prompt; no remediation is invoked.\")\n" ] }, { "cell_type": "markdown", "id": "signal-quest-29", "metadata": {}, "source": [ "## 10. Reproducibility and doctoral defense\n", "\n", "A result without a data boundary, code trace, configuration, and limitations is not a result that another student can audit. The final manifest records this lab's synthetic inputs and prompts. It is not a benchmark report." ] }, { "cell_type": "code", "execution_count": 21, "id": "signal-quest-30", "metadata": { "execution": { "iopub.execute_input": "2026-08-03T19:40:34.909430Z", "iopub.status.busy": "2026-08-03T19:40:34.909373Z", "iopub.status.idle": "2026-08-03T19:40:34.912389Z", "shell.execute_reply": "2026-08-03T19:40:34.911944Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"title\": \"Signal Quest LLM-guided capstone\",\n", " \"epistemic_status\": \"simulated teaching lab\",\n", " \"decision_contract\": {\n", " \"horizon_minutes\": 5,\n", " \"tie_rule\": \"UP\",\n", " \"receive_time_required\": true,\n", " \"research_only\": true\n", " },\n", " \"seed\": 8414,\n", " \"rows\": {\n", " \"events\": 720,\n", " \"dataset\": 695,\n", " \"test\": 139\n", " },\n", " \"feature_draft\": \"llm_feature_draft.json\",\n", " \"feature_draft_sha256\": \"b74624f36136a6c2a34ab5df574afcaceae1e4f09471393cf1bab87181b2bcf3\",\n", " \"policy\": \"paper-only, fail-closed\",\n", " \"guardian\": {\n", " \"status\": \"HEALTHY\",\n", " \"issues\": [],\n", " \"allowed_actions\": [\n", " \"write_incident_record\",\n", " \"quarantine_research_artifact\",\n", " \"request_human_review\"\n", " ],\n", " \"forbidden_actions\": [\n", " \"place_order\",\n", " \"change_model\",\n", " \"change_policy\",\n", " \"delete_evidence\"\n", " ]\n", " },\n", " \"limitations\": [\n", " \"synthetic data\",\n", " \"no executable venue data\",\n", " \"no live performance claim\",\n", " \"no order authority\",\n", " \"advanced-model cells are design-target scaffolds until independently implemented and tested\"\n", " ]\n", "}\n" ] } ], "source": [ "def sha256_path(path: Path) -> str:\n", " return hashlib.sha256(path.read_bytes()).hexdigest()\n", "\n", "manifest = {\n", " \"title\": \"Signal Quest LLM-guided capstone\",\n", " \"epistemic_status\": \"simulated teaching lab\",\n", " \"decision_contract\": asdict(CONTRACT),\n", " \"seed\": SEED,\n", " \"rows\": {\"events\": len(events), \"dataset\": len(dataset), \"test\": len(test)},\n", " \"feature_draft\": \"llm_feature_draft.json\",\n", " \"feature_draft_sha256\": sha256_path(WORKDIR / \"llm_feature_draft.json\"),\n", " \"policy\": \"paper-only, fail-closed\",\n", " \"guardian\": guardian,\n", " \"limitations\": [\n", " \"synthetic data\",\n", " \"no executable venue data\",\n", " \"no live performance claim\",\n", " \"no order authority\",\n", " \"advanced-model cells are design-target scaffolds until independently implemented and tested\",\n", " ],\n", "}\n", "(WORKDIR / \"experiment_manifest.json\").write_text(json.dumps(manifest, indent=2))\n", "print(json.dumps(manifest, indent=2))\n" ] }, { "cell_type": "markdown", "id": "signal-quest-31", "metadata": {}, "source": [ "## Capstone defense prompts\n", "\n", "1. Show one event that occurred before the cutoff but arrived too late. Why is it forbidden?\n", "2. Read the LLM prompt and trace. What exact code behavior was requested, and what behavior was prohibited?\n", "3. Why is the student-controlled feature function the only function used in the experiment?\n", "4. Compare the baseline Brier score to the candidate score. What does the comparison fail to prove?\n", "5. Which advanced model gate would catch a future-looking sequence implementation?\n", "6. Find a case where the policy abstains despite a high model probability.\n", "7. What can the guardian do automatically, and what is it structurally unable to do?\n", "8. State the strongest defensible conclusion from this notebook without using the words “profitable,” “edge,” or “prediction success.”" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.15" }, "signal_quest": { "author": "Dr. Mallarapu", "epistemic_status": "simulated teaching lab", "safety_boundary": "research and paper-trading only; no execution authority" } }, "nbformat": 4, "nbformat_minor": 5 }