feat: integrate ruflo + real YouTube/Reddit/HN/web reading

RUFLO (ruvnet/ruflo) — agent meta-harness: swarms, 314+ MCP tools, vector memory,
federation, 30+ plugins. The runtime ships on npm, so `aios ruflo` delegates to
`npx ruflo@latest` and the vendored tree is trimmed 92MB -> 10MB (skills + docs
only, 136 plugin skills preserved). New: `aios ruflo plugins` catalogues them,
`aios ruflo mcp-register` registers its MCP server with Claude Code, the
ruflo-orchestration skill, and ruvnet/ruflo added to supervised updates.

WEB READING (aios_web.py) — the agents can now actually read a link instead of
guessing from the URL slug:
  youtube(url)  full transcript (flat / timestamped / segments), title, channel,
                duration, views, description, chapters, available languages
  reddit(t)     a thread + comment tree, a subreddit listing, or a search
  hackernews()  story search, or a story with its comment thread
  fetch(url)    any page reduced to readable text (auto-delegates YT/Reddit URLs)
  search(q)     DuckDuckGo, no key, sponsored slots filtered

Exposed as POST /api/web, `aios web <op>`, chat commands /yt /reddit /hn /web, and
the web-research skill; the Brain's prompt now tells it to use these and never
claim it cannot open a link. Every call is audited.

Both sources fought back, and the fixes are the substance here:
- YouTube's timedtext endpoint now returns HTTP 200 with an EMPTY body — verified
  across five InnerTube clients and a full cookie session. Pure scraping is dead.
  yt-dlp is the only thing that keeps up, so it's the primary path (metadata +
  captions) with the stdlib scrape as fallback, and `aios setup` installs it. When
  it's absent you still get metadata plus a note naming the exact fix.
- Reddit 403s its JSON API from this network regardless of User-Agent (tested four,
  plus old.reddit, api.reddit and a redlib mirror). RSS still works, so that's the
  fallback: responses carry source="rss" and scores are null — the skill tells
  agents not to report vote counts they didn't get. Added a request throttle,
  429 backoff, and caching of the JSON-blocked state (retrying it burned two
  requests per call and pushed us into rate limits before RSS even ran).

Verified live: pulled the real transcript of the first YouTube video ever
("Me at the zoo", 6 segments + 3 chapters), a real r/programming thread with 8
comments via RSS, HN search, DuckDuckGo results, and all of it through the hub
endpoint and the /yt + /reddit chat commands.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: ZD Studios

README.md

@@ -2,7 +2,7 @@
 
 # 🧠 The AI OS
 
-### Eight open-source AI projects. One operating system. One Control Room.
+### Nine open-source AI projects. One operating system. One Control Room.
 
 <p align="center">
   <a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-8B5CF6?style=for-the-badge" alt="MIT License"></a>
@@ -23,6 +23,7 @@
   <img src="https://img.shields.io/badge/+_Fabric-255_patterns-0EA5E9?style=flat-square" alt="Fabric">
   <img src="https://img.shields.io/badge/+_Caveman-say_less-A16207?style=flat-square" alt="Caveman">
   <img src="https://img.shields.io/badge/+_Ponytail-build_less-DB2777?style=flat-square" alt="Ponytail">
+  <img src="https://img.shields.io/badge/+_Ruflo-314_MCP_tools-7C3AED?style=flat-square" alt="Ruflo">
 </p>
 
 **[Install](#-install-one-line)** · **[Control Room](#-the-control-room)** · **[Architecture](#️-architecture)** · **[Commands](#️-command-reference)** · **[Full docs](README.aios.md)** · **[Website](https://zdstudios.github.io/AIOS/)**
@@ -31,7 +32,7 @@
 
 ---
 
-**The AI OS** takes eight independent open-source AI projects and makes them work as **one auto-configured
+**The AI OS** takes nine independent open-source AI projects and makes them work as **one auto-configured
 system** you drive from a single command — **`aios`** — and a single web **Control Room** where you can
 **talk to everything and have the agents talk to each other**.
 
@@ -41,7 +42,7 @@ together, mounts your shared skills, and adds **OpenUI** generative-UI context t
 
 > You can't fuse three runtimes and multiple package managers into a single file — that would just break
 > everything. So "one file" here means **one control surface** (`aios`) + **one dashboard** (the Hub) over
-> eight real, unmodified projects. It installs, configures, wires, runs, tests, and debugs all of them.
+> nine real, unmodified projects. It installs, configures, wires, runs, tests, and debugs all of them.
 
 <div align="center">
 
@@ -240,6 +241,54 @@ That is deliberately the "ambient authority" posture that made **OpenClaw's CVE-
 
 **Active Memory:** a memory sub-agent runs on **every** turn — recall is a free FTS5 query, fact-extraction is one small async call — so the agents actually learn your workflow over time instead of only reading memory at session start (`memory.active`).
 
+## 🌐 It can actually read the web
+
+Paste a link and the agents read it — no API keys, no "I can't open URLs".
+
+| Source | What you get |
+|---|---|
+| **YouTube** | Full **transcript** (flat, timestamped, or segment objects), title, channel, duration, views, description, **chapters**, available languages |
+| **Reddit** | A thread with its **comment tree**, a subreddit listing, or a search |
+| **Hacker News** | Story search, or a story with its full comment thread |
+| **Any page** | Reduced to clean readable text |
+| **Search** | DuckDuckGo, no key |
+
+```bash
+aios web youtube "https://youtu.be/VIDEO_ID"      # transcript in your terminal
+aios web reddit  "r/LocalLLaMA"                   # or a thread URL, or a search
+aios web fetch   "https://example.com/article"
+```
+
+In chat: `/yt <url>`, `/reddit <target>`, `/hn <query>`, `/web <url>` — or just paste a
+link and ask; the Brain calls `POST /api/web` itself. The bundled **`web-research`**
+skill teaches every agent the same API.
+
+Two honest caveats, both surfaced in the response rather than hidden:
+- **YouTube transcripts need `yt-dlp`** (`aios setup` installs it). YouTube now serves
+  its caption endpoint an empty body unless the request carries a proof-of-origin
+  token, so scraping it directly no longer works — yt-dlp is what keeps up with that.
+  Without it you still get metadata, and the `note` tells you what to install.
+- **Reddit blocks its JSON API from many networks**, so AIOS falls back to RSS
+  automatically. When that happens the response says `"source":"rss"` and **scores are
+  `null`** — the agents are told not to report vote counts they didn't actually get.
+  Reddit also rate-limits hard, so calls are throttled and retried.
+
+## 🤖 Ruflo — heavyweight orchestration on demand
+
+[ruvnet/ruflo](https://github.com/ruvnet/ruflo) is an agent meta-harness: multi-agent
+**swarms**, **314+ MCP tools**, vector memory, federation, and 30+ plugins. AIOS ships
+its skills and plugin catalogue and drives the runtime from npm, so it's always current.
+
+```bash
+aios ruflo plugins         # the 30+ plugins and their skill counts
+aios ruflo init wizard     # set it up inside a project
+aios ruflo mcp-register    # register its MCP server with Claude Code
+aios ruflo <anything>      # passed through to `npx ruflo@latest`
+```
+
+AIOS is the always-on system (services, channels, dashboard, Task Brain); ruflo is a
+per-project harness you invoke for a heavy job. They compose.
+
 ## 🎨 Agents can redesign the dashboard (safely)
 
 Ask in chat — *"make the accent blue and the text bigger"*, *"hide Automations"*, *"put a

aios.py

@@ -269,6 +269,7 @@ def resolve_root(*candidates) -> Path | None:
     "fabric": resolve_root("fabric-main/fabric-main", "fabric-main", "fabric"),
     "caveman": resolve_root("caveman-main/caveman-main", "caveman-main", "caveman"),
     "ponytail": resolve_root("ponytail-main/ponytail-main", "ponytail-main", "ponytail"),
+    "ruflo": resolve_root("ruflo-main/ruflo-main", "ruflo-main", "ruflo"),
 }
 
 
@@ -810,6 +811,7 @@ def cmd_setup(args):
     if cfg_get(cfg, "skills.mount", True):
         mount_aios_skills(cfg)
         mount_agent_modes(cfg)  # caveman + ponytail skills → every agent
+        mount_ruflo(cfg)        # ruflo orchestration skill → every agent
 
     # 7) wire openclaw-os plugin (best-effort; needs openclaw runnable)
     if cfg_get(cfg, "services.openclaw_os.enabled", True) and not args.skip_wire:
@@ -955,8 +957,31 @@ def _ensure_tools(args):
                 err(f"{t} still not found — open a new shell or install manually: {cmdline}")
 
 
+def install_web_deps():
+    """yt-dlp powers YouTube transcripts. YouTube now serves its caption endpoint an
+    empty body without a proof-of-origin token, and yt-dlp is what keeps up with
+    that — without it AIOS can read video metadata but not what was said."""
+    try:
+        import yt_dlp  # noqa: F401,PLC0415
+        ok("yt-dlp present (YouTube transcripts enabled)")
+        return True
+    except Exception:
+        pass
+    print(f"{C.GRY}$ {sys.executable} -m pip install yt-dlp{C.R}")
+    r = subprocess.run([sys.executable, "-m", "pip", "install", "--quiet",
+                        "--disable-pip-version-check", "yt-dlp"],
+                       capture_output=True, text=True)
+    if r.returncode == 0:
+        ok("installed yt-dlp (YouTube transcripts enabled)")
+        return True
+    warn("could not install yt-dlp — YouTube metadata will work, transcripts won't. "
+         "Install it manually: pip install yt-dlp")
+    return False
+
+
 def _install_all(cfg):
     head("Installing dependencies")
+    install_web_deps()
     # opencode: --ignore-scripts skips native postinstalls (tree-sitter-powershell,
     # node-pty) that need VS Build Tools on Windows and are unused by `serve`.
     jobs = [
@@ -1822,6 +1847,108 @@ def cmd_wire(args):
     wire_full_control(load_config())
 
 
+def mount_ruflo(cfg: dict):
+    """Mount ruflo's entry-point skill so agents know the `npx ruflo` surface and its
+    ~314 MCP tools. Its 136 plugin skills stay on disk and are listed by
+    `aios ruflo plugins` — copying them all into every agent would drown context."""
+    src = PROJECTS.get("ruflo") or (ROOT / "ruflo-main")
+    sk = src / "SKILL.md"
+    if not sk.exists():
+        return
+    head("Mounting ruflo skill")
+    for t in (Path.home() / ".openclaw" / "skills" / "ruflo",
+              Path.home() / ".hermes" / "skills" / "ruflo",
+              STATE / "skills" / "ruflo"):
+        try:
+            t.mkdir(parents=True, exist_ok=True)
+            shutil.copy(sk, t / "SKILL.md")
+        except Exception as e:
+            warn(f"could not mount ruflo → {t}: {e}")
+    n = len(list((src / "plugins").glob("*/skills/*/SKILL.md"))) if (src / "plugins").exists() else 0
+    ok(f"ruflo skill mounted ({n} plugin skills available via `aios ruflo plugins`)")
+
+
+def cmd_ruflo(args):
+    """Run ruflo (agent meta-harness). Delegates to the npm package, which is the
+    supported delivery channel — the vendored copy here is only skills + docs."""
+    src = PROJECTS.get("ruflo") or (ROOT / "ruflo-main")
+    sub = list(getattr(args, "args", []) or [])
+
+    if sub and sub[0] == "plugins":
+        pdir = src / "plugins"
+        if not pdir.exists():
+            die("ruflo plugins not found — re-run `aios setup`")
+        head("Ruflo plugins")
+        for d in sorted(p for p in pdir.iterdir() if p.is_dir()):
+            skills = list(d.glob("skills/*/SKILL.md"))
+            say(f"  {C.CYN}{d.name:<26}{C.R}{C.GRY}{len(skills)} skill(s){C.R}")
+        say(f"\n{C.GRY}Enable one inside a project:{C.R} npx ruflo@latest plugin add <name>")
+        return
+
+    if sub and sub[0] == "mcp-register":
+        claude = find_tool("claude")
+        if not claude:
+            die("the `claude` CLI is not installed — see `aios claude-login`")
+        cmd = [claude, "mcp", "add", "ruflo", "--", "npx", "ruflo@latest", "mcp", "start"]
+        say(f"{C.GRY}$ {' '.join(cmd)}{C.R}")
+        rc = subprocess.run(cmd, cwd=str(ROOT)).returncode
+        (ok if rc == 0 else warn)("ruflo MCP registered with Claude Code" if rc == 0
+                                  else f"registration exited {rc}")
+        return
+
+    npx = find_tool("npx")
+    if not npx:
+        die("npx not found — install Node >= 20 (see `aios doctor`)")
+    cmd = [npx, "ruflo@latest"] + (sub or ["--help"])
+    say(f"{C.GRY}$ {' '.join(cmd)}{C.R}")
+    try:
+        sys.exit(subprocess.run(cmd, cwd=str(ROOT), env=child_env({})).returncode)
+    except KeyboardInterrupt:
+        pass
+
+
+def cmd_web(args):
+    """Read YouTube / Reddit / Hacker News / any page from the terminal."""
+    sys.path.insert(0, str(ROOT))
+    import aios_web as W
+    op, arg = args.op, " ".join(args.arg or [])
+    if not arg:
+        die(f"usage: aios web {op} <url-or-query>")
+    fn = {"youtube": W.youtube, "yt": W.youtube, "reddit": W.reddit,
+          "hn": W.hackernews, "search": W.search, "fetch": W.fetch}[op]
+    r = fn(arg)
+    if not r.get("ok"):
+        warn(r.get("error", "failed"))
+        sys.exit(1)
+    if op in ("youtube", "yt"):
+        head(r["title"])
+        say(f"  {C.GRY}{r['channel']} · {r['duration']} · {r['views']:,} views{C.R}")
+        if r.get("chapters"):
+            say(f"\n{C.B}Chapters{C.R}")
+            for c in r["chapters"][:12]:
+                say(f"  {C.GRY}{c['ts']:>7}{C.R}  {c['title']}")
+        if r.get("transcript"):
+            auto = " (auto-generated)" if r.get("transcript_auto") else ""
+            say(f"\n{C.B}Transcript{C.R} {C.GRY}{r['words']} words{auto}{C.R}\n")
+            say(r["transcript_timestamped"] if getattr(args, "timestamps", False)
+                else r["transcript_text"])
+        else:
+            warn(r.get("note", "no transcript"))
+    elif op == "reddit" and r.get("kind") == "post":
+        head(r["title"])
+        if r.get("selftext"):
+            say(r["selftext"][:2000])
+        say(f"\n{C.B}{r.get('comments_returned', 0)} comments{C.R}")
+        for c in r.get("comments", [])[:25]:
+            sc = f"[{c['score']}] " if c.get("score") is not None else ""
+            body = c["body"][:600].replace("\n", "\n  ")
+            say(f"\n  {C.CYN}{c['author']}{C.R} {C.GRY}{sc}{C.R}\n  {body}")
+        if r.get("note"):
+            say(f"\n{C.GRY}{r['note']}{C.R}")
+    else:
+        print(json.dumps(r, indent=2)[:12000])
+
+
 def cmd_updates(args):
     """Supervised updates: what moved upstream, what the agents think, apply/rollback."""
     sys.path.insert(0, str(ROOT))
@@ -2056,11 +2183,11 @@ def _print_urls(cfg):
 # unreadable once you have 25 subcommands.
 HELP_GROUPS = [
     ("Get started", ["setup", "start", "stop", "restart", "status", "url"]),
-    ("Everyday", ["logs", "doctor", "exec", "channels", "profile", "token"]),
-    ("Agents & updates", ["updates", "update", "claude-login", "attach", "migrate", "wire"]),
+    ("Everyday", ["logs", "doctor", "exec", "web", "channels", "profile", "token"]),
+    ("Agents & updates", ["updates", "update", "claude-login", "ruflo", "attach", "migrate", "wire"]),
     ("Advanced", ["test", "debug", "autostart", "install-cli", "bootstrap"]),
 ]
-TAGLINE = "Eight open-source AI projects. One operating system."
+TAGLINE = "Nine open-source AI projects. One operating system."
 
 
 class AiosParser(argparse.ArgumentParser):
@@ -2195,6 +2322,17 @@ def build_parser():
     s.add_argument("--rollback", metavar="PROJECT", help="restore a project's previous version")
     s.set_defaults(func=cmd_updates)
 
+    s = sub.add_parser("web", help="read a YouTube video, Reddit thread, HN or any page")
+    s.add_argument("op", choices=["youtube", "yt", "reddit", "hn", "search", "fetch"])
+    s.add_argument("arg", nargs=argparse.REMAINDER)
+    s.add_argument("--timestamps", action="store_true", help="keep [mm:ss] on transcripts")
+    s.set_defaults(func=cmd_web)
+
+    s = sub.add_parser("ruflo", help="ruflo meta-harness: 314+ MCP tools, swarms, plugins")
+    s.add_argument("args", nargs=argparse.REMAINDER,
+                   help="passed to `npx ruflo` — or: plugins | mcp-register")
+    s.set_defaults(func=cmd_ruflo)
+
     s = sub.add_parser("token", help="print the hub token (needed for non-loopback access)")
     s.add_argument("--rotate", action="store_true", help="generate a fresh token")
     s.set_defaults(func=cmd_token)

aios_hub.py

@@ -34,6 +34,7 @@
 import aios_sec as sec        # noqa: E402  the gate in front of full-control mode
 import aios_tools as tools    # noqa: E402  shell + channel/skill catalogs
 import aios_updates as updates  # noqa: E402  agent-supervised dependency updates
+import aios_web as web          # noqa: E402  YouTube / Reddit / HN / page readers
 
 PORT = int(os.environ.get("AIOS_HUB_PORT", "8787"))
 ROOT = Path(os.environ.get("AIOS_ROOT", Path(__file__).resolve().parent))
@@ -463,6 +464,15 @@ def brain_system_prompt(message: str) -> str:
             "Ops: set (vars/css/scale/density) · nav (hidden/order) · panel (add/del/clear, "
             "slot top|chat|sidebar, self-contained HTML styled with var(--accent) etc.) · "
             "reset (what: all|vars|css|panels|nav). Full reference: the `dashboard-designer` skill.")
+        sysp += (
+            "\n\nREADING THE WEB: you can read YouTube videos (full transcripts), Reddit "
+            "threads, Hacker News and any web page through the hub — use these instead of "
+            "writing your own scraper, and never claim you cannot open a link:\n"
+            f"  RUN: curl -s -X POST http://127.0.0.1:{PORT}/api/web -H 'Content-Type: application/json' "
+            "-d '{\"op\":\"youtube\",\"url\":\"<link>\"}'\n"
+            "ops: youtube (transcript_timestamped + title/channel/chapters) · reddit (post+comments, "
+            "r/sub, or a search phrase) · hn · search · fetch (any URL -> readable text). "
+            "Add \"|python -c ...\" or pipe through jq to trim big payloads before reading them.")
     sysp += active_recall(message)      # Active Memory: relevant context, every turn
     sysp += modes_prompt_suffix()       # caveman / ponytail overlays when toggled on
     return sysp
@@ -1478,6 +1488,26 @@ def do_POST(self):
                              f"**{st[hit]['level']}** — {meta['blurb']}. `/{hit} off` to stop.")
                 self._send(200, {"target": target, "reply": reply, "ran": []})
                 return
+            # Inline web reads: /yt <url> · /reddit <target> · /web <url>
+            wm = re.match(r"^/(yt|youtube|reddit|web|hn)\s+(.+)$", message.strip(), re.I)
+            if wm:
+                kind, arg = wm.group(1).lower(), wm.group(2).strip()
+                fn = {"yt": web.youtube, "youtube": web.youtube, "reddit": web.reddit,
+                      "hn": web.hackernews}.get(kind, web.fetch)
+                r = fn(arg)
+                if not r.get("ok"):
+                    md = f"⚠️ {r.get('error', 'failed')}"
+                elif kind in ("yt", "youtube"):
+                    md = (f"### {r['title']}\n**{r['channel']}** · {r['duration']} · "
+                          f"{r['views']:,} views\n\n" +
+                          (f"**Transcript** ({r.get('words', 0)} words"
+                           f"{', auto-generated' if r.get('transcript_auto') else ''}):\n\n"
+                           f"{r.get('transcript_timestamped', '')[:6000]}"
+                           if r.get("transcript") else f"_{r.get('note', 'No transcript.')}_"))
+                else:
+                    md = "```json\n" + json.dumps(r, indent=1)[:6000] + "\n```"
+                self._send(200, {"target": "web", "reply": md, "ran": []})
+                return
             # Inline fabric: /p <pattern> <text>  or  /pattern <name> <text>
             if cmd.startswith(("/p ", "/pattern ")):
                 rest = message.split(None, 1)[1] if len(message.split()) > 1 else ""
@@ -1596,6 +1626,30 @@ def do_POST(self):
             st = set_mode(payload.get("mode", ""), payload.get("enabled", False),
                           payload.get("level", "full"))
             self._send(200, {"state": st})
+        elif self.path == "/api/web":
+            # Real reads of YouTube / Reddit / HN / any page. One op per source so
+            # an agent never has to hand-roll scraping.
+            op = (payload.get("op") or "fetch").lower()
+            q = payload.get("url") or payload.get("query") or payload.get("target") or ""
+            try:
+                if op in ("youtube", "yt"):
+                    r = web.youtube(q, want_transcript=payload.get("transcript", True),
+                                    lang=payload.get("lang", "en"))
+                elif op == "reddit":
+                    r = web.reddit(q, limit=int(payload.get("limit", 60)),
+                                   sort=payload.get("sort", "top"))
+                elif op in ("hn", "hackernews"):
+                    r = web.hackernews(q, story_id=payload.get("story_id", ""),
+                                       limit=int(payload.get("limit", 20)))
+                elif op == "search":
+                    r = web.search(q, limit=int(payload.get("limit", 10)))
+                else:
+                    r = web.fetch(q, max_chars=int(payload.get("max_chars", 20000)))
+            except Exception as e:
+                r = {"ok": False, "error": f"{op} failed: {e}"}
+            brain.audit(payload.get("by", "agent"), "web." + op,
+                        f"{q[:120]} -> ok={r.get('ok')}")
+            self._send(200, r)
         elif self.path == "/api/dashboard":
             self._send(200, dashboard_update(payload))
         elif self.path == "/api/updates":

aios_updates.py

@@ -52,6 +52,7 @@
     "fabric":     {"repo": "danielmiessler/fabric",     "dir": "fabric-main",          "tier": "content"},
     "caveman":    {"repo": "JuliusBrussee/caveman",     "dir": "caveman-main",         "tier": "content"},
     "ponytail":   {"repo": "DietrichGebert/ponytail",   "dir": "ponytail-main",        "tier": "content"},
+    "ruflo":      {"repo": "ruvnet/ruflo",              "dir": "ruflo-main",           "tier": "content"},
 }
 
 # What each project is used for — given to the agent so its risk call is grounded
@@ -66,6 +67,8 @@
     "fabric": "AIOS reads data/patterns/*/system.md only (255 prompt files). No code from it executes.",
     "caveman": "AIOS reads skills/caveman/SKILL.md as a system-prompt overlay and mounts its skills.",
     "ponytail": "AIOS reads skills/ponytail/SKILL.md as a system-prompt overlay and mounts its skills.",
+    "ruflo": ("AIOS mounts its SKILL.md and lists plugins/*/skills. The runtime comes from "
+              "npm (`npx ruflo@latest`), so upstream code changes here are docs/skills only."),
 }