}-jhXdZddlmZddlZddlZddlZddlZddlZddlZddl Z ddl m Z ddl m Z mZmZmZejeZd5d Zd6d Zd6dZd7dZ d8d9dZddd:dZd;dd%Z d?d@d,ZdAd.Zd8d>d/Zd8d>d0Z dBdCd2Z!dDd3Z"ed4kre#e"dS)Eug Session recovery from .bak snapshots — last line of defense against data-loss bugs like #1558. ``Session.save()`` writes a ``.json.bak`` snapshot of the previous state whenever an incoming save would shrink the messages array. This module reads those snapshots back and restores any session whose live file has fewer messages than its backup, or whose live file is missing while a valid backup remains. Three integration points: 1. ``recover_all_sessions_on_startup()`` — called from server.py at boot, scans the session dir, restores any session whose JSON has fewer messages than its .bak, and recreates a missing ``.json`` from an orphaned ``.json.bak`` when the canonical state DB still has that session. Idempotent: a clean run is a no-op. 2. ``recover_session(sid)`` — single-session helper backing the ``POST /api/session/recover`` endpoint, so users can re-run recovery manually if their session was open through a server restart. 3. ``inspect_session_recovery_status(sid)`` — read-only audit returning message counts for the live JSON, the .bak, and a recommendation. ) annotationsN)Path)derive_turn_journal_statesis_terminal_turn_eventiter_turn_journal_session_idsread_turn_journalprreturnintc@ tj|d}n##ttjt f$rYdSwxYwt |tsdS|d}t |trt|ndS)u Return the number of messages in a session JSON file, or -1 on read/parse error. Returns -1 for any non-session-shape file: - File can't be read (OSError) - Top-level isn't valid JSON or is invalid (JSONDecodeError, ValueError) - Top-level isn't a dict (AttributeError on .get) — e.g. ``_index.json`` which is a top-level list of session metadata, not a session itself. The startup recovery scanner globs ``*.json`` and would otherwise crash on the first non-dict file it encounters. utf-8encodingmessages) jsonloads read_textOSErrorJSONDecodeError ValueError isinstancedictgetlistlen)r datamsgss */root/hermes-webui/api/session_recovery.py _msg_countr /sz!++w+7788 T): 6rr dD ! !r 88J  D"4.. 63t999B6s(+A  A  session_pathrc|d}t|}|s |j|dddSt|}||kr |j||ddS|j||ddS)a Return a status dict describing whether recovery is recommended. { "session_id": "...", "live_messages": int, # -1 if live file unreadable "bak_messages": int, # -1 if no .bak or unreadable "recommend": "restore" | "no_action" | "no_backup", } .json.bakr no_backup) session_id live_messages bak_messages recommendrestore no_action) with_suffixr existsstem)r!bak_path live_count bak_counts rinspect_session_recovery_statusr1Ds'' 44HL))J ??   &+'$    8$$I:&+'%"    #'#!   cBt|}|ddkri|ddiS|d}|d} tj||||nq#t $rd}t d|| |d n#t $rYnwxYwi|dt|d cYd }~Sd }~wwxYwt d |j |d |di|ddiS)zRestore session_path from its .bak when the bak has more messages. Returns a status dict identical to ``inspect_session_recovery_status`` plus a "restored" boolean. r(r)restoredFr#z.json.recover.tmpz'recover_session: copy failed for %s: %sT missing_ok)r4errorNu|recover_session: restored %s from .bak (live=%d → bak=%d messages). See #1558 for the data-loss class this guards against.r&r') r1r+shutilcopyfilereplacerloggerwarningunlinkstrname)r!statusr.tmp_pathexcs rrecover_sessionrCgs{ -\ : :F ki'',&,*e,,,'' 44H''(;<4J (f 'j$ ' ''sB*A:: C(C#!B87C#8 CC#CC#C(#C(r%r> state_db_path Path | Noneboolc||sdS tjd|dd5}|d}| ddddS|d|f}|ducdddS#1swxYwYdS#t $r'}t d||Yd}~dSd}~wwxYw) a!Return whether state.db still knows this session. The check is deliberately fail-open: recovery must not be prevented by a locked, absent, or older-schema state DB. When a DB is readable and has no row, treat the orphan backup as a tombstoned/deleted session and skip it. NTfile:?mode=rourizBselect 1 from sqlite_master where type='table' and name='sessions'z+select 1 from sessions where id = ? limit 1z2state_db session tombstone check failed for %s: %s)r,sqlite3connectexecutefetchone Exceptionr;debug)r%rDconncurrBs r_state_db_has_sessionrTsfM$8$8$:$:t  _<]<<<$ G G G .4,,TC||~~%  . . . . . . . . ,,Lzm\\C<<>>- . . . . . . . . . . . . . . . . . .  I:WZ[[[tttttsFB2+B% B2,,B% B2%B))B2,B)-B22 C#<CC# session_dir list[Path]cg}t|dD]}|d}|jds|rFt |dkrZ|j}t||s!t d|j| ||S)aHReturn live ``.json`` paths whose ``.json.bak`` exists. ``Path.glob('*.json')`` does not see orphan backups because their suffix is ``.bak``. Existing startup recovery only handled shrunken live files; this helper covers the crash shape where the live sidecar is gone but the rescue copy remains. *.json.bak_rz[recover_all_sessions_on_startup: skipped orphan backup %s; state.db has no live session row) sortedglobr+r? startswithr,r r-rTr;infoappend)rUrDpathsr. live_pathr%s r_orphaned_backup_live_pathsrbsE;++L99::  ((,, > $ $S ) ) Y-=-=-?-?   h  ! # # ^ $Z??  KK3      Y Lr2F include_emptyrd list[dict]c\||sgS tjd|dd5}tj|_d|dD}d|d D}d d h|sgcdddStd |}td |}td|d}td|} td|d} td|} td|} td|} td|}td|}g}|d|d|d|d| d| d| d| d| d|d|dD]%}t|}t| d pd }|r||dz reg}hd|rd|vrd |vrdnd }d|vrdnd!}|d"|d#||fD]=}|d$|d%pdd&}|d |d|d<| |>|s|s||d'<| |d(<| |'|cdddS#1swxYwYdS#t$r(}td)||gcYd}~Sd}~wwxYw)*z@Return WebUI-origin state.db rows whose JSON sidecar is missing.NrHrITrJch|] }|d S.0rows r z6_read_state_db_missing_sidecar_rows..eeesCFeeer2zPRAGMA table_info(sessions)ch|] }|d Srhrjrks rrnz6_read_state_db_missing_sidecar_rows..ror2zPRAGMA table_info(messages)idsourcetitlemodel started_at0parent_session_id message_count workspace worktree_pathworktree_branchworktree_repo_rootworktree_created_atz$ SELECT id, source, z, z, z FROM sessions WHERE source = 'webui' ORDER BY COALESCE(started_at, 0) DESC rY.json>rolecontentr% timestampz timestamp, idrowidzNULL AS timestampzSELECT role, content, z- FROM messages WHERE session_id = ? ORDER BY rr)rrr_state_db_empty_messagesz6state_db sidecar reconciliation scan failed for %s: %s)r,rLrMRow row_factoryrNfetchallissubset_sql_optional_colrr>rstripr_rPr;rQ)rUrDrdrR session_cols message_cols title_expr model_expr started_expr parent_exprmsg_count_exprworkspace_exprworktree_path_exprworktree_branch_exprworktree_repo_root_exprworktree_created_at_exprrowsrmrsid message_rowsorderts_exprmsgmessagerBs r#_read_state_db_missing_sidecar_rowsrsM$8$8$:$: 8 _<]<<<$ G G G4 4&{D eedll;X.Y.Y.b.b.d.deeeLeedll;X.Y.Y.b.b.d.deeeL(#,,\::  4 4 4 4 4 4 4 4  +7LAAJ*7LAAJ,\<MML+,?NNK. cRRN.{LIIN!2?L!Q!Q #45F #U#U &78Ll&[&[ #'89NP\']'] $D||$.2<@L#'59G+/C0 4L   hjj" "" "Cyy$((4...B//5577{]]]:BBDD+- 444==lKK 5/:l/J/JtWcOcOcOOipE-8L-H-HkkNaG#||nnnglnn  hjj 5 5 %(K'*9~';##{+736{3CGK0$++G4444#M#/Z 7C3C/0 D!!!!i4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 j  M}^abbb sNK9B K, K9 HK, K9,K00K93K04K99 L+L& L+&L+NULLr?columnsset[str]fallbackc||vr|n|d|S)Nz AS rj)r?rrs rrrs"7??448(?(?(?(??r2rmcr ddlm}n#t$rd}YnwxYwt|dpd}|r ||n|pd|pd|r|ndd}|dpd}t|dtr|dng}|r6t|dtr|dd n|}|d pd}id |d d |d pdd t|tr|nddt|dtr|dnt|d|dpdd|dpdd|dpdd|dpdd|dpdddd|d|p|ddddddddddiddd dd!dd"dd#dd$gd%dd&dd'dd(dd)dd*dd+dd,dd-gd.dd/|d/d|pdd0|di|gd1d2S)3Nr)normalize_agent_session_sourcerrrY) raw_sourcesession_source source_labelrurrrryr%rqrszRecovered WebUI Sessionrxrzr{r|r}rtunknownmodel_provider created_at updated_atpinnedFarchived project_idprofile input_tokens output_tokensestimated_cost personalityactive_stream_idpending_user_messagepending_attachmentspending_started_atcompression_anchor_visible_idxcompression_anchor_message_keycompression_anchor_summarycontext_lengththreshold_tokenslast_prompt_tokensgateway_routinggateway_routing_historyllm_title_generatedrw)is_cli_session source_tagT)enabled_toolsetscomposer_draftr tool_calls_recovered_from_state_db) api.agent_sessionsrrPr>rrlowerrsrrrr r)rmrrr source_metarurlast_tsworkspace_values r_state_db_row_to_sidecarrs".EEEEEEE ...)-&&&. ""(b ) ) / / 1 1 7 7 9 9F%>+   ?C(H(HP__b+  Z@X@XZ]5^5^q111dghpdqdq +  119T +  377#455= +  cgg&:;;Ct+  sww'<==E+  !!.Y+  $+  j+  g++  %+  E+  d+  4!+ " #+ + $ %+ & $'+ ( t)+ * D++ , -+ . r/+ 0 d1+ 2 )$3+ 4 )$5+ 6 %d7+ 8 $9+ : D;+ < d=+ > 4?+ @ "2A+ B uC+ D SWW%899E+ + F nI+ + + J K+ L!$(U+ + + +s  c nt||}d}g}|dd|D]t}t|dpd}|s<||dz }|rYt |}dtjdtj j } | | } d } | tj|d d d nf#t $rY} | dn#t $rYnwxYw||d t| dYd} ~ :d} ~ wwxYwd } tjt| t|d} nJ#t($rYn>t $r2} ||d t| dd} Yd} ~ nd} ~ wwxYw | dn<#t $rYn0wxYw# | dw#t $rYwwxYwxYw| rB|dz }||dt+|dpgdY| s||d ddvt+|||dS)zEMaterialize missing WebUI JSON sidecars from canonical state.db rows.rT)parentsexist_okrqrYr~z.json.reconcile.tmp..F) ensure_asciiindentr rr5)r% materializedr7Nrir)r%rr!sidecar_appeared_during_reconcile)r%rskipped)scannedrdetails)rmkdirr>rrr,rosgetpid threadingcurrent_threadidentr+ write_textrdumpsrr=r_linkFileExistsErrorr)rUrDrrrrmrtargetpayload tmp_suffixtmpdetail_recordedrBmaterialized_nows r&recover_missing_sidecars_from_state_dbr@sI .{M J JDLGdT222.w.w#''$--%2&&,,..  #}}}, ==??  *3//]BIKK\\):R:T:T:Z\\   ,,  NN4:gE!LLLW^N _ _ _ _     d ++++     NN#usSVxxXX Y Y Y HHHH  !  GCHHc&kk * * *#      D # # # NN#usSVxxXX Y Y Y"OOOOOO #  d ++++      d ++++      w A L NN#tQTU\U`U`akUlUlUrprQsQstt u u u u  w NN#uQtuu v v v4yy,7 S SSs,D  E-D,+E(, D96E(8D99)E((E-31F%$H% G,/H1 G,:(G'"H'G,,H0H HHIH0/I0 H= :I<H= =Irkindcategoryrecommendationr&r'c B||||||d}|||S)N)r%rrrr&r')update)r%rrrr&r'extraitems r_new_audit_itemrxs;!(&$   D KK Kr2 index_pathc tj|d}n0#ttjt f$rt cYSwxYwt|tst St }|D]Z}t|trCt| dtr| |d[|S)Nr rr%) rrrrrrsetrrrrr>add)rridsentrys r_read_index_session_idsrsz*...@@AA T): 6uu  dD ! !uu EEC)) eT " " )z%))L2I2I3'O'O ) GGE,' ( ( ( Js(+*AAcn |s dddddgdSg}td|dD}d|D}|D]}t|}|dd krU|t |d d d d |dd|ddt|dD]}|d}|s|j drFt|}|j } |dkr(|t | dddd|t| |r(|t | dd d d||t | dddd||dz } | rt| } t| |z D]'} |t | dd d(t|| z D]<} |t | dd dt|| dz d=t||dD]} t| d pd} | d!r(|t | d"ddddc|t | d#d d$ddt!|D]} t#| |%}t%|d&pg\}}|| dz }t|}t'} t)j|d'(}t/|t0r|d)pgD]y}t/|t0rb|d*d+krI|t|d,pdzn"#t6t(jt:f$rYnwxYwt|D]\}}t?|rt|d,pd}|r||vrR|t | d-d d.|d|t|d/pd0tA|ddd}|D]+}|d1}||vr||xxd2z cc<,|drd3}n |d rd4}nd}|||dS)5a3Read-only audit of session recovery state. The audit intentionally does not mutate files. It classifies only the safe recovery primitives this module knows how to perform: backup restores and derived index rebuilds. Call ``recover_all_sessions_on_startup`` separately for safe repairs. okr)r repairableunsafe_to_repair)r@summaryitemsc3NK|] }|jd|V!dS)rZNr?r]rlr s r z)audit_session_recovery..s8\\aQVEVEVWZE[E[\\\\\\\r2*.jsonch|] }|j Srj)r-rs rrnz)audit_session_recovery..s+++1+++r2r(r)r% shrunken_liverrestore_from_bakr&rr'rXrYrZmalformed_orphan_backupr manual_review orphan_backuporphan_backup_without_state_rowz _index.jsonindex_missing_file rebuild_indexindex_missing_entryr~Trcrqrstate_db_orphan_webui_rowstate_db_missing_sidecarmaterialize_from_state_db)rUeventsr rrruserrturn_journal_pending_turnaudit_only_pending_turn_journalevent)turn_idrrrineeds_manual_reviewwarn)!r,r[r\r1rr_rr+r?r]r r-rTrrr>rrrrrrrrrrrrrrrrr)rUrDr live_pathslive_idsrar@r.r'r%r index_idsrmrjournalstatesrZr&existing_user_messagesrrrrrrrroveralls raudit_session_recoveryr s      qaHH   E\\;#3#3H#=#=\\\\\J++ +++H   0;; ::k " "i / / LL|$" ?B// >2..    ;++L99::((,,      !:!:3!?!?  !(++ ^ !   LL57I?\^`l    #:} = =  LLO\;MrS_     LL1"     },J +J77  X!566  J LL0,    !I!566  J LL1<;J)=)=)==>>     3; ]abbb  #''$--%2&& 77- . .  LL+"      _  &  '         4K@@ #JKHHH.w{{8/D/D/JKK Z"6"6"66 "9-- +.55 j!4!4g!4!F!FGGG'4(( ^&{{:66<"^^G!'400^W[[5H5HF5R5R.223w{{97M7M7SQS3T3T3Z3Z3\3\]]]-z:    D $V\\^^44  NGU%e,, %))I..4"55;;==G g)??? LL+1%))G,,233         "__A1MMG##88J'' w   H    "   !"'  'E B BBs%CQ44RRct||}t|d|}t||}|drF ddlm}|dn2#t $r%}td |Yd}~nd}~wwxYwt||}t|d pid pd}t|d pid pd} |dko| dk} | | t|d pdt|dpdz||||dS)aRun safe, deterministic session recovery repairs. This mutates only repairable classes already handled by startup recovery: shrunken live sidecars and orphan backups that are not tombstoned by a readable state.db. Unsafe audit findings remain for manual review. rDT)r rDrr_write_session_indexNupdateszTrepair_safe_session_recovery: index rebuild after state.db reconciliation failed: %srrrr4)cleanrrepairedbefore backup_repairsidecar_repairafter) r recover_all_sessions_on_startuprr api.modelsr$rPr;r<r ) rUrDr)r*r+r$rBr,unsafe_remainingrepairable_remainingr's rrepair_safe_session_recoveryr1!s$K} M M MF3#M j>jj&(  s A B 'BB r c |sdddgdSd}d}g}dt|dD}t||}g||D]}|dz } t |} nK#t $r>} t d|jt| j | Yd} ~ [d} ~ wwxYw| d r|dz }| | |rdt d |||rF dd l m} | d n2#t $r%} t d | Yd} ~ nd} ~ wwxYw||t||dS)zScan session_dir for shrunken/orphaned sessions and restore from .bak. Returns {"scanned": N, "restored": M, "orphaned_backups": K, "details": [...]}. r)rr4orphaned_backupsrcFg|]}|jd|S)rZr)rlpaths r z3recover_all_sessions_on_startup..Rs.ggg4TYMaMabeMfMfg$gggr2rr"riz9recover_all_sessions_on_startup: skipped %s due to %s: %sNr4urecover_all_sessions_on_startup: restored %d/%d sessions from .bak. If you weren't expecting this, check the session list for missing messages — see #1558.r#r%z9recover_all_sessions_on_startup: index rebuild failed: %s)r,r[r\rbrCrPr;r<r?type__name__rr_r.r$r) rUr rDrr4rr orphan_pathsr5resultrBr$s rr-r-Ds     S!rRRRGHGgg6+*:*:8*D*D#E#EgggJ.{-XXXL,*,|,## 1  $T**FF    NNK 499-s    HHHH  ::j ! ! # MH NN6 " " " a &'/    a a;;;;;;$$T22222 a a aZ\_```````` a --   s0-A== C4CCD,, E6EEc,tjd}|ddd|dtdd |d tdd |ddd|}|jrt |j|j}n8|j rt|j|j}n| dttj|ddS)Nz)Audit Hermes WebUI session recovery state) descriptionz--audit store_truezrun a read-only recovery audit)actionhelpz --session-dirTz path to WebUI sessions directory)r7requiredr?z --state-dbzoptional Hermes state.db path)r7defaultr?z --repair-safez-run safe deterministic repairs after auditingr"zchoose --audit or --repair-safe) sort_keysr)argparseArgumentParser add_argumentr parse_args repair_safer1rUstate_dbauditr r7printrr)parserargsreports r_mainrN|s  $1\ ] ] ]F  ,=]^^^ dTHjkkk  4Dcddd  Crsss     D 8-d.>dm\\\ 8'(8 VVV 6777 $*Vt , , ,--- 1r2__main__)r rr r )r!rr r)r%r>rDrEr rF)N)rUrrDrEr rV)rUrrDrErdrFr re)r)r?r>rrrr>r r>)rmrr r)rUrrDrEr r)rr)r%r>rr>rr>rr>r&r r'r r r)rrr r)FN)rUrr rFrDrEr r)r r )$__doc__ __future__rrCrloggingrr8rLrpathlibrapi.turn_journalrrrr getLoggerr8r;r r1rCrTrbrrrrrrr r1r-rN SystemExitrjr2rrWs2#"""""    8 $ $7777*    F((((>2"&D AAAAAAH@@@@@::::z5T5T5T5Tz *    CCCCCCCCCCL     J !%55555p    " z *UUWW  r2