🔧 Portal services outage + fix (M-062) — 2026-06-18 ~02:15 GMT+3

Report: Dr. Wael reported portal subdomains down after the last OpenClaw update: local(no models), obsidian, hub, wiki, apps, panel, status; render assumed intentional.

Diagnosis (live):

• hub/apps/obsidian = 502 (nginx proxy up, upstream 172.16.1.2:PORT refused). Those Flask services run INSIDE the openclaw container (8899 hub, 8903 apps, 8908 obsidian, 8901 wiki, 8902 RAG). They died on the container update/restart ~6h prior and were never auto-restarted.
• wiki = 401 (basic-auth gate; backend 8901 was actually dead → would 502 after login).
• local = 200 but open-webui couldn't list models: Ollama bound to 127.0.0.1 only → open-webui (separate container) at OLLAMA_BASE_URL=http://172.16.1.2:11434 got connection refused. Model qwen2.5:3b present in /data/.ollama.
• panel/status = actually 200/healthy all along — Dr. Wael saw them "broken" because he clicked them FROM the apps portal which was 502.
• render = no container = intentionally off/on-demand (confirmed). 404 expected.

ROOT CAUSE (the big one): scripts/daemons/daemons_keepalive.sh had set -Eeuo pipefail. The script is full of var=$(curl/pgrep/cat ...) assignments that legitimately fail as control flow. When the trycloudflare hub tunnel URL expired, code=$(curl ... dead-url) exited 6 (can't resolve host) and set -e aborted the ENTIRE keepalive at ~line 104 — BEFORE reaching the wiki/apps/ollama/hub-flask/obsidian guards (lines 118-547). So once that tunnel died, the keepalive silently stopped restarting all those services. Every restart since left them dead.

Fixes applied: 1. keepalive: set -Eeuo pipefail → set -uo pipefail (a keepalive must never abort on one failed probe). Hardened the 2 trycloudflare curl lines with || echo 000. 2. Ollama: restarted bound 0.0.0.0:11434; fixed scripts/ollama_autostart.sh (was 127.0.0.1 + empty OLLAMA_MODELS=/data/ollama → now 0.0.0.0, default model path /data/.ollama). open-webui now reaches it; qwen2.5:3b visible. 3. Stale lock: the old keepalive.lock fd leaked into the gateway process (37020) during my manual diagnostic runs via exec tool → cron saw "already running — skip" forever. Rotated lock path to keepalive.run.lock. (Old stale fd harmless; clears on next gateway restart.)

Verified: keepalive now runs to EXIT 0, reaches all guards. Public HTTPS: local/obsidian/hub/apps/panel/status=200, wiki=401(auth gate, backend live), render=404(intentional). All local ports 8899/8903/8908/8901/8902/11434=200.

Lesson: never use set -e in a keepalive/watchdog full of probe-style command substitutions. One expired tunnel silently disabled the whole self-healer for ~all services.

M-063 — replay_invalid / thinking-signature transient error + live alert guard (2026-06-18 03:1x)

الحدث: الساعة 02:41-02:42 ظهر للمستخدم LLM request failed: provider rejected the request schema or tool payload. الجذر الحقيقي (من اللوج الخام): messages.3.content.25: Invalid signature in thinking block · providerRuntimeFailureKind: replay_invalid. حصل reset للجلسة (sessionId 23e78802 -> 93e1f3eb) بعد turn ثقيل فيه side-effects، فصار توقيع كتلة تفكير قديمة لاغياً عند الـ replay. failoverReason=null صحيح (خطأ 400 schema، ليس انقطاع مزوّد — fallback ما كان ينفع). انحلّ ذاتياً بالـ reset لمحادثة نظيفة. صفر تأثير على البيانات/الخدمات. التوصية #2 (تجريد thinking عند replay) مرفوضة: كود الإنجن shouldPreserveThinkingBlocks() يعيد TRUE لـ opus-4/sonnet-4/haiku-4/claude-5..9 عمداً (حذفها يكسر prompt caching = شبح $678/يوم + يدهور الدقة). الإعداد غير مكشوف في config schema أصلاً. الإنجن يطبّق السياسة المثلى لـ Opus 4.8. good-zephyr + gentle-prairie (Process failed): أوامر فحص خلفية شغّلتها ثم قتلتها/يتّمتها — ضجيج تجميلي صفر تأثير. القاعدة السلوكية M-063 (دائمة): (1) أوامر الفحص foreground مضبوطة تنتهي ضمن النافذة (2) لو تحتاج وقتاً = poll حتى الإكمال، لا kill (3) لا exec خلفي يعيش بعد الـ turn. الإصلاح المنفّذ (بند 1): كتلة replay_error_alert في scripts/daemons/daemons_keepalive.sh (read-only، debounced 30min، grep -cF على نمط "providerRuntimeFailureKind":"replay_invalid" المحصّن ضد تلوّث أصداء الأوامر، صفر LLM، set -u safe، كل op بـ || true). state بذرة memory/.replay_error_state. اختُبر A/B/C + end-to-end keepalive EXIT=0 + verify_agreements EXIT=0 + كل الخدمات 200/401(gate)/404(off). backup: daemons_keepalive.sh.bak-replayalert-*. المتبقي (بموافقة لاحقة): بند 3 — إزالة Nexos من التدقيق الأسبوعي + فحص cron EPG (c0ff96fb).