2026-07-05

🦾 Weekly Full Audit (cron 05:15)

• CLI updates: mcporter 0.12.2→0.12.3 · openclaw 2026.6.10→2026.6.11 · codex 0.142.3→0.142.5 · kilo 7.3.54→7.4.1 · Hermes updated (v0.18.0) · gh apt update available (2.92.0→2.96.0).
• Python: critical pkgs auto-updated (openai/anthropic/google-genai/groq/elevenlabs/yfinance...); 293 pkgs still outdated globally (non-critical). pypfopt install failed (wrong pkg name — actual: PyPortfolioOpt).
• New models: Anthropic claude-sonnet-5 (already in registry) · Google gemini-3.1-flash-lite-image, gemini-omni-flash-preview, gemini-2.5-flash-native-audio-latest · OpenRouter poolside/laguna-xs-2.1(:free) · Mistral labs-leanstral-1-5(-1). All watchlist/experimental — primary opus-4-8 unchanged. Updated MODEL_REGISTRY.md header.
• APIs: 15 reachable / 0 failed. Nexos 401 (yamnual only). HeyGen video endpoint timed out (transient).
• Drift guard: NO CRITICAL DRIFT. Optional missing: lms/blender/libreoffice/whisper CLI (cloud fallbacks exist).
• Skills: 141 workspace + 18 Hermes. Plugins 29/76 enabled.

Portal fixes (05:47-05:55 Kuwait) — 3 tasks from Dr. Wael voice note

1. Cost dashboard totals — added tfoot total row to provider table + model table + grand-total banner (calls+USD, per-window) in projects/cost-dashboard/app.py. Restarted :8915 (PID rotated). Verified: grandtot markers served, API 30d window_total=$6204.61 / 5 providers / 9 models. 2. Invest not daily — ROOT CAUSE: cron 3719e746 was 30 9 1,4 (Mon+Thu), last run July 2. Changed to DAILY 30 9 (Asia/Kuwait) + updated name/description/payload wording + run_update.sh header. Ran catch-up now: 5/5 pages deployed @ 05:54, index as-of 2026-07-05. 3. Hermes web dashboard broken (OTP then "reconnecting session") — ROOT CAUSE: during the incomplete July-4 cloudflared-tunnel migration, the Traefik router /docker/traefik/dynamic/hermes.yml was disabled (renamed .disabled-20260704191508) but DNS still points to Traefik (A record). So after CF Access OTP, all dashboard API/WS calls hit a dead route → reconnect loop. FIX: restored hermes.yml from the .disabled file (byte-identical to the July-4 Playwright-verified config with hermes-host Host+Origin rewrite needed for Hermes DNS-rebinding WS defense). Verified differential: origin Host=hermes 404→403 (hermes-cfonly active = route loaded), no Traefik errors. Orphaned cloudflared tunnel (PID 2040102, id 52e99306) left running — harmless (DNS doesn't point to it); cleanup optional. My own Hermes path = CLI v0.18.0 + local :9120 (HTTP 200), independent of the web tunnel, never affected.

Hermes — CORRECTED complete diagnosis (screenshots + code proof)

Screenshots showed HTTP worked (sessions/model loaded) but WS = "code 1006 / events feed disconnected". So HTTP reached Hermes even with Traefik router disabled → the LIVE path was the cloudflared tunnel, NOT Traefik (verified: hermes DNS was CNAME→52e99306.cfargotunnel.com via CF API; public dns-json hides it because proxied). ROOT CAUSE: cloudflared tunnel ingress rewrites Host→127.0.0.1 (httpHostHeader) but NOT Origin. Hermes web_server.py _ws_host_origin_reason rejects any WS whose Origin netloc isn't loopback → close 4403 → browser sees 1006. HTTP worked because only Host is checked for HTTP (middleware) and tunnel rewrites Host. FIX (root, proven): flipped hermes DNS CNAME→cfargotunnel back to A→76.13.41.228 (proxied, mirrors panel/grafana) via CLOUDFLARE_ZONE_TOKEN (has DNS:Edit). Now path = CF edge (Access OTP) → Traefik → hermes router (restored) → hermes-host middleware rewrites BOTH Host+Origin→127.0.0.1 → forwarder → dashboard. PROOF (direct WS handshake to 127.0.0.1:9120 with real __HERMES_SESSION_TOKEN__): Origin=http://127.0.0.1 → 101 Switching Protocols ✅ ; Origin=https://hermes... → 403 ❌ ; no token → 403. Public endpoint still 302→CF Access (OTP intact). DNS confirmed A/proxied. Layers preserved: CF Access (email OTP) + Traefik hermes-cfonly (CF-IP-only) + basicauth (admin, same as wiki/files) + Hermes Host/Origin hardening. Orphaned: cloudflared tunnel proc 2040102 (id 52e99306) still running but DNS no longer points to it — harmless; optional cleanup later (its keepalive guard, if any, only restarts the process, not DNS). The July-4 tunnel migration was the thing that broke WS; reverting to Traefik = correct (all sibling services use Traefik+CF anyway; "zero exposed ports" not worth a broken dashboard). My own Hermes usage = CLI v0.18.0 direct + local :9120 — never touched the web path, always worked.

Hermes verification (Dr. Wael asked to confirm) + Omni Flash + Python fix

• Hermes DIRECT (my path): live smoke test hermes chat -q "1723 + capital of Australia" → "391; Canberra" in 7s, exit 0. CLI v0.18.0 fully functional & ready. (Warnings: unknown toolsets messaging/moa + tirith scanner unavailable — cosmetic, non-blocking.)
• Hermes WEB screenshot problem = WS code 1006: tunnel rewrote Host but not Origin → Hermes CORS/DNS-rebind guard rejected WS. Proven: dashboard accepts WS with Origin=http://127.0.0.1 → 101; with Origin=https://hermes... → 403. FIX = DNS flipped CNAME(tunnel)→A 76.13.41.228 (Traefik, hermes-host rewrites Host+Origin→127.0.0.1). Verified: DNS=A/proxied, route 404→403, public 302→CF Access. Only unverifiable-by-me step = Dr. Wael's email-OTP login click.
• Python pypfopt→PyPortfolioOpt fixed in weekly_full_audit.sh + weekly_software_update.sh (grep-match proven).
• Gemini Omni Flash: video-only (no image). #1 T2V (+158 vs Veo), tied #1 I2V (+77), ~$0.10/s native audio. Weak faces/physics. Updated DOMAIN_MODEL_RANKINGS.md T2V PRIMARY=Omni Flash (Veo fallback) with unverified-Elo caveat. Report: reports/gemini_omni_flash_eval_2026-07-05.md. PENDING Dr. Wael go: change actual video_generate T2V default + I2V co-default (keep Grok mandatory).
• PENDING Dr. Wael go (self-audit): add meta-guard heartbeat on host cron + extend banner_truth_guard to versions/prices/tool-existence.

Gemini Omni Flash — CORRECTION (hallucination caught by live test)

Subagent report claimed it's #1 T2V video gen (+158 vs Veo). FALSE. Live API: "This model only supports Interactions API", supportedGenerationMethods=[generateContent], no predictLongRunning. = realtime/omni interaction model (GPT-Realtime competitor), NOT video/image gen, not in video_generate tool. Reverted DOMAIN_MODEL_RANKINGS.md (Veo 3.1 stays T2V PRIMARY, Grok I2V). Logged ERR-20260705-001. NO video default change. Dr. Wael's insistence on real testing prevented a wrong config change.

Self-audit stability improvement — IMPLEMENTED + TESTED (Dr. Wael approved)

Multi-level critique refined the design: instead of a NEW meta-guard cron (new failure surface, M-062 lesson), used ZERO new process: 1. daemons_keepalive.sh writes memory/keepalive_heartbeat (epoch) at end of each cycle (last line, || true). Tested: file created, keepalive exit 0. 2. Host guard /docker/openclaw-4x4t/openclaw_host_recovery.sh (already independent, every 2min, has alert()) now checks that heartbeat: if container UP but heartbeat >20min stale → best-effort one-shot recovery (keepalive is singleton-flock safe) + throttled Telegram alert (30min). Closes the M-026/M-062 "silent guard death" gap. bak saved. Tested: runs clean when fresh (no false alert), stale-branch logic correct at 25min. DROPPED after critique: extending banner_truth_guard to free-text version/price/tool claims — unreliable on free text = false positives = instability (contradicts "zero downsides"). The real win = behavioral rule ERR-20260705-001 (verify subagent/model claims via live API). Final double-check ALL GREEN: verify_agreements exit 0 (ALL VERIFIED), cost :8915 ok, hermes :9120 HTTP 200, keepalive heartbeat fresh, gateway_error_guard exit 0. No crash, stability intact.

Batch 2 (Dr. Wael follow-ups) — Fable 5 + Hermes tunnel

• Fable 5 CONFIRMED wired: model claude-fable-5 is REAL + reachable (live API returned OK, echoed claude-fable-5). skills/fable-orchestrator: architect(ph3-4)/review(ph11,diff-only)/unblock(circuit-breaker after 2 fails), token caps, input guards, ledger, never-stop fallback Fable→Opus4.8→liberate_code. architect() calls Fable directly. NUANCE: benchmark_gate_passed=false (informational, not enforced in call path) — we use it on trust; ROI benchmark not yet run. It's a methodology/CLI tool (fable_core.py), invoked at design step — not an auto-hook into Kilo/Claude Code.
• Hermes orphan cloudflared tunnel (52e99306) REMOVED: DNS is A→Traefik so tunnel was orphaned + churning. Disabled keepalive guard (line 330 if false, reversible + commented) + killed proc. Verified: 0 info-run procs, other 3 quick-tunnels untouched, local :9120=200, forwarder :9121=200, my CLI hermes chat -q → "READY". ZERO impact on CLI or portal (both use Traefik/local, not tunnel). LESSON re-hit: pgrep -f self-match trap (M-076) — my kill matched own exec shell → aborted; use bracket-grep ps -eo args | grep '[c]loudflared...'.

Tech bulletin follow-up (Claude Code + Claude Science)

• Claude Code: installed 2.1.201 == npm latest 2.1.201 → already up to date on server (no action). Dr. Wael updating his laptop separately.
• Claude Science: already studied 2026-07-01 + logged in MODEL_WATCHLIST 2026-07-03. It's a claude.ai PRODUCT (not API), launched 2026-06-30, science workbench macOS/Linux beta, Pro/Max/Team/Enterprise, bio/pharma-tilted (not our civil-eng domain). DECISION: watch, don't integrate platform; adopt architecture ideas free. What we BUILT instead: skills/science-arsenal/ (science_workspace.py route/research/verify + science_router.py) — unifies lit_search (OpenAlex/Crossref/arXiv/EuropePMC) + consensus_check + cite_check + medical-arsenal + RAG, engineering branch → Opus 4.8. Live-tested today: route "concrete beam shear" → branch=engineering, correct tools/sources. Alert triggers if: Claude-for-Engineering ships / Science gets API / native Windows. Link: anthropic.com/news/claude-science-ai-workbench.

Final confirmation sweep (Dr. Wael) — images/video leadership + Omni + STT + guards

• IMAGES: GPT-image-2 = #1 GLOBALLY (AA Image Arena Elo 1339, clean sweep ALL image leaderboards +242 gap over Nano Banana 2/FLUX.2 max/Seedream 4.0; also #1 image EDITING Elo 1254). We HAVE it (face #1 per TOOLS.md). = LEADING.
• VIDEO: current T2V leader = Seedance 2.0 (~1216 Elo, native audio+camera) — we HAVE it via fal (bytedance/seedance-2.0/). Also Veo 3.1, Grok Imagine (I2V PRIMARY, mandatory rule intact), Sora 2, Kling 3.0, Hailuo, Runway, Wan. = LEADING/covered. Old "Seedance 2.0 no API" note was stale.
• Google Omni Flash: NOT image/video gen (realtime Interactions model, proven). No integration needed for gen. Marginal for voice (our stack = Gemini 3.1 Flash TTS + gpt-realtime-mini already set). Verdict: watch, no action.
• STT: #2 global Scribe v2 (AA-WER 2.2%, we have via ElevenLabs creator key, live-tested perfect Arabic). #1 Fun-Realtime (1.7%, no API for us = watch). Diacritics: NONE output by any STT → pipeline STT+CATT. Recommendation: Scribe v2 for critical, Groq whisper (free) daily, gpt-4o-transcribe co-option.
• Guards: keepalive_heartbeat fresh (247s), host guard staleness-check present (count=1). Stable.

Final sweep pre-/new (Dr. Wael) — all verified live + music fix

• Fable 5: model claude-fable-5 reachable, 3 roles (architect/review/unblock), never-stop net. ✅
• Guard: keepalive_heartbeat fresh (167s) + host staleness-check present + host cron /2min. ✅
• STT: doc updated (Scribe v2 #2 AA-WER, diacritics=none→CATT pipeline). ✅
• MUSIC FIX: table was missing Suno V5 as vocal primary. Fixed: "أغنية بكلمات/غناء" row = Suno V5 (primary) → ElevenMusic → MiniMax Music 2.6 (fallback/backup). Matches Dr. Wael's request + memory. Lyria = cinematic/instrumental.
• Video/image rules intact: I2V=Grok mandatory, T2V PRIMARY=Veo (Omni reverted). We have Seedance 2.0/Runway Gen-4.5/Veo/Sora/Kling/Hailuo/Wan — lead/covered. Images: GPT-image-2 #1 global.
• verify_agreements final: see /tmp/va_final.log.

Batch 3 — Hermes single-auth + Portal completeness audit

• Hermes login SIMPLIFIED: removed redundant Traefik basicauth (hermes-auth) from router. Now single sign-on = Cloudflare Access email OTP ONLY. Security preserved: hermes-cfonly (CF-IP-only, direct origin=403) + hermes-host (Host+Origin rewrite) + Hermes hardening intact. Verified: public=302→CF Access (no 401 basicauth), origin=403. bak saved.
• PORTAL AUDIT: 33 cards, 32 healthy (200/302/401/403). Only render=404 = Blender on-demand BY DESIGN (exited clean 12d ago, blender_ondemand.sh, saves 3.7GB RAM; agent starts on need). Annotated card "Blender (عند الطلب)". Non-portal live services (bot/chat/private/www) = internal (bot-proxy/local-ai/landing), not gaps. All our additions (cost/invest/ask-RAG/hermes/canvas/code/hub/obsidian/sync/twin/cv/eng/go/etc) present + healthy on portal.

Hermes Arabic font fix (Dr. Wael: Arabic glyphs unclear in dashboard)

• ROOT CAUSE: Hermes dashboard CSS uses Latin-only fonts (Mondwest, Rules Compressed/Expanded, JetBrains Mono, Courier New) with NO Arabic glyphs → Arabic falls to poor browser fallback, esp. in mono/code areas.
• FIX: scripts/daemons/hermes_arabic_font_fix.sh — idempotent inject of Noto Naskh Arabic (Google Fonts) into web_dist/index.html <head>, with the font unicode-range-scoped to Arabic ONLY (via Google's own @font-face). Latin/icons/code KEEP original fonts; only Arabic glyphs use Noto → clear bilingual output. Redefines --default/theme font vars + targets text/message/code containers.
• SAFE: touches only served static HTML head. WS untouched (verified 101 post-fix), Traefik/forwarder untouched, zero impact on other services.
• PERSISTENCE: Hermes updates overwrite web_dist/index.html → wired idempotent re-inject into daemons_keepalive.sh (grep-guarded, || true). Verified: served HTML now has Noto Naskh link + hermes-ar-font-fix style; WS=101.
• Dr. Wael action: hard-refresh (Ctrl+F5) the Hermes tab to load the new font.

TTS 6-min delay — ROOT CAUSE FIXED (ERR-20260705-002)

Why 6 min + text fallback: NOT engine failure. tts_unified.py concat_mp3 used ffmpeg -c copy to put MP3 chunks into .ogg container → fails ("incorrect codec parameters") → engine-agnostic so cascaded through all 6 engines regenerating audio each time (~6 min) → text fallback. FIX: container-aware _deliver() (merge to temp mp3, then transcode to requested container: ogg/opus→libopus, mp3→copy). Verified: .ogg single-chunk 11s, 3000-char 2-chunk 82s, valid Opus, 1 engine, zero cascade. Fallback chain + ranking were always correct (Charon #1); bug was purely container mismatch. media_say.sh unchanged (calls tts_unified).

🔴 INCIDENT — Full death/paralysis ~18:16–18:34 Kuwait (2026-07-05) — root cause + fix

What Dr. Wael experienced: complete death + total paralysis; had to go to Hostinger and use Cody via Terminal to run multiple restarts to revive me. Furious (rightly).

Verified evidence (live):

• openclaw PID 262 uptime = ~10 min at 17:44 CEST → process restarted ~15:34 UTC (18:34 Kuwait).
• openclaw log: repeated failed respawns 15:16→15:29 UTC every ~3 min, all: "Gateway failed to start: gateway already running (pid 2191685); lock timeout after 5000ms" + "Port 18789 is already in use." → deadlock loop.

ROOT CAUSE (self-inflicted regression — my fault, not Hostinger/Anthropic):

• A hung zombie gateway (pid 2191685) held port 18789 + the gateway lock but stopped serving /health.
• watchdog_unified.sh recovery used pkill -x openclaw = SIGTERM only. A hung/zombie proc ignores SIGTERM → never died → kept holding the port → every respawn died on "Port already in use / lock timeout." Full paralysis until Dr. Wael manually killed the zombie via Cody.
• Aggravated by today's M-070 "radical false-kill fix" (deployed 2026-07-05) which made the watchdog reluctant to kill → the zombie was never force-removed.

FIX deployed (this incident): 1. Escalating kill in hung branch: SIGTERM → wait → SIGKILL (-9) if still alive. 2. New free_gateway_port() helper: fuser -k -9 18789/tcp + ss PID fallback → force-frees the port by PID before respawn. 3. Pre-respawn guard: if gateway still down, free_gateway_port first so bind cannot fail on stale lock/port.

• bash -n syntax OK. Watchdog restarted fresh: old PID 862 (SIGKILL) → new PID 4789 with fix loaded.

TODO (next, low-priority follow-up): add M-NNN to MISTAKES_LEDGER + verify_agreements guard that scans openclaw log for "lock timeout"/"Port ... already in use" loops and asserts free_gateway_port is present in watchdog.

🔴 INCIDENT PART 2 — why AUTO-recovery failed (host-layer root cause) — fixed 2026-07-05 ~20:05 Kuwait

Dr. Wael's core complaint (rightly furious): 12+ deaths in one day; even after Hostinger/VPS restarts I stayed dead; he had to open laptop -> Hostinger -> Cody -> manual restart. The REAL failure = the system could not self-heal.

ROOT CAUSE of the self-heal failure (host-level, proven from /var/log/openclaw_host_recovery.log):

• Host recovery guard (/docker/openclaw-4x4t/openclaw_host_recovery.sh, cron every 2min) decided liveness via gw_http_alive() = curl http://ip:8899/ -> ANY HTTP code != 000 = "ALIVE -> NEVER restart", and RESET the fail counter every cycle.
• The hung zombie (pid 2191685) kept HOLDING port 18789 and answering TCP/HTTP superficially while the engine processed nothing. So the guard logged status probe failed but gateway HTTP alive (busy, not dead) -- restart SUPPRESSED and suppressed recovery FOREVER. Same "alive-but-dead" class; the guard could not tell busy from zombie.
• Aggravated by M-070 (deployed earlier today) which made ALL guards reluctant to act. So neither the in-container watchdog NOR the host guard killed the zombie -> only manual Cody restart worked.

FIX (host guard busy-vs-zombie by DURATION):

• Added BUSY_STATE counter + BUSY_MAX=5. On status fail + HTTP alive: increment; suppress only up to ~10min (protects any legitimate heavy turn/compaction); beyond that = ZOMBIE CONFIRMED -> force reactive recovery (doctor --fix + docker restart, which frees the port by respawning clean). Counter resets only on a real status success.
• Backup saved on host: openclaw_host_recovery.sh.bak.20260705T170705Z. bash -n OK both sides. cron runs the new version live (verified fires 17:04/17:06/17:08). Counters read fails=0 busyfails=0 (healthy).
• Now TWO layers break a zombie: in-container watchdog free_gateway_port (~85s-3min) + host guard (~10min ultimate safety net). Worst-case paralysis ~3min (was ~30min + manual).

THREE root-cause fixes deployed today (all live-verified): (1) watchdog free_gateway_port deadlock-breaker (in-container). (2) host guard busy-vs-zombie duration gate (host-level). (3) FACTS_FROZEN root-ownership self-heal in keepalive (meta-guard). Plus Layer-C session-poison self-heal ARMED (compact RPC).

STILL TODO (durable guards, must not be forgotten): M-NNN ledger entries + verify_agreements checks: (a) assert free_gateway_port present in watchdog, (b) assert BUSY_MAX/zombie-gate present in host_recovery, (c) scan for repeated restart SUPPRESSED with unchanged gateway pid = zombie-suppression regression.

🎬 Entertainment Hub — Genre/Category Filter Chips (2026-07-05 ~21:30)

طلب د. وائل: بحث الأفلام/المسلسلات يبقى كما هو + شريط تصنيفات في الأعلى (الكل + أنواع) لتسهيل الاختيار.

• بُني: genres.py (مصنّف bilingual: يحوّل أسماء تصنيفات Xtream → 35 وسم موحّد؛ +18 حصري ونظيف) · build_cats.py (جدول cats = (sub,cat_id)→name+tags، 5833 صف) · تعديل catalog_search.py (LEFT JOIN cats → يرجع genres لكل نتيجة) · indexer.py (يبني cats داخل DB المؤقت فيبقى بعد swap) · templates/index.html (شريط .genre-bar + applyGenreFilter فلترة client-side فورية، الكل أولاً + أنواع لها نتائج فقط + عدد لكل نوع).
• تغطية: vod 57% / series 68% موسومة؛ الباقي يظهر تحت «الكل»/«أخرى».
• E2E (Playwright/browser): batman → 1004 عنوان، شريط 28 chip، النقر على «أبطال خارقون» فلتر لـ103 (مطابق للعداد)، «الكل» يرجّع 1004. healthz 200.
• أُعيد تشغيل hub (PID قديم 1469 → جديد) على 8899 + tunnel.

🎬 Hub genre filter v2 — توصيات 1-5 (2026-07-05 ~22:14)

نفّذ 1-4 + الـ5 بشكل خاص: 1. فلترة مزدوجة (AND): اختيار متعدد للأنواع (أجنبي+أكشن=42 عنوان، متحقق كلها تحمل الوسمين). الكل يمسح، +18/أخرى حصريان. 2. «يعرض الآن» (nowair): وسم جديد في genres.py (40 تصنيف) + cats أُعيد بناؤه. chip أول بعد «الكل» عند وجوده. 3. ترتيب: dropdown «الأنسب/الأحدث سنة/الأعلى تقييماً» (السنة تُستخرج من الاسم regex، التقييم من rating). newest متحقق تنازلياً. 4. تذكّر: localStorage (hubGenreSel+hubSortMode) — يُستعاد فقط إن الوسم موجود في نتائج البحث الجديد وإلا يرجع «الكل». متحقق بعد reload. 5. +18 حصيف: يبقى ضمن «الكل»، لكن chip الفلترة = دائرة صغيرة خافتة (●) بلا نص؛ الضغط يكشف «كبار +18» + العدد (حصري). متحقق.

• E2E Playwright: صفر أخطاء console. hub مستقر PID ثابت healthz 200 (الـ PID churn الأولي = سباق restart يدوي مع keepalive، استقر).

🎬 Hub genre filter v3 — 4 تحسينات إضافية (2026-07-05 ~23:26)

1. تصنيف عربي فرعي: وسوم ar_egypt/ar_gulf/ar_sham/ar_maghreb في genres.py (675 تصنيف). صفّ «المنطقة» يظهر تحت الشريط فقط عند تفعيل «عربي». إصلاح false-positive: «مغربي» كان يلتقط western (غربي substring) → regex boundary. 2. تلوين الأزرار: GENRE_COLORS لكل نوع (رعب أحمر داكن #7f1d1d، أطفال أخضر…)، tint خفيف idle + solid عند التفعيل. 3. زر «✕ مسح الفلاتر»: يظهر فقط عند وجود اختيار. 4. مفضّلات الأنواع: long-press/right-click على أي زر يثبّته (📌 hubGenreFav localStorage)، تظهر أولاً بعد «الكل»، بلا تكرار، فقط إن كانت ضمن نتائج البحث.

• E2E Playwright متحقق: عربي+مصري=21 (كلها valid)، ألوان inline، clear يظهر/يختفي، fav يتصدّر مع 📌، صفر أخطاء console. hub مستقر healthz 200.