rancher-ai-ui PR 346, round three: token usage after the second fixes

Our Request changes review (2026-11-17, at 2f157f3, 4 comments) → torchiaf rebased onto main eaf81a0, answered three threads and pushed four commits: 357913b unpin the agent, b41c5a7 keep the last count on errors, 66899a1 new e2e counts, 9d17835 RcIconTooltip. A git range-diff shows the reviewed commits unchanged (2f157f3 = 2dc6e2a), so only these four are new. Two passes again: Pass A with Cody's my-pr-review-new-commits (ai-skills c04c11a), Pass B with our process in a fresh agent.

13.62 still runs an agent without the usage route and an llm-config without an active LLM. The "can't start" spinner and "Token usage not available" were seen for real. Everything else was driven by stubs in the browser only. Nothing was written to his environment.

Where it stands

Three of your four comments are done: the warning icons now work from the keyboard and with a screen reader, a failed refresh keeps the count, and the log carries the agent's error. The optional spinner comment got no reply and no change.

Two new functional problems, both from these commits:

One more, outside the new commits: after you close and reopen the panel, the count stops updating. It's round-one code that both rounds missed. Cody's round-two rule keeps it out, and our "PR-introduced" rule lets it in. Your call.

Nothing look-only is in the review. My suggestion: Request changes again, with the e2e point required, the spinner required or optional (your call; I lean required, since it's on the first screen of a default setup), and the reopen freeze only if you want it.

Your four comments

Your commenttorchiaf's answerChecked at 9d17835State
Optional: a 500 after 1.8K dropped the footer to "Tokens used: 0" (useAIAgentApiComposable.ts:426)"Fixed, thanks"Same steps as round two: 1.8K shown, then a 500 on the refresh after an answer. It stays "Tokens used: 1.8K in total" and the button stays enabled. A non-JSON 502 also keeps it (Pass B).Addressed
Optional: every failure logged "body stream already read" (:412)"Not an issue - we don't want to parse the error here"The line changed in b41c5a7 anyway. The log now reads "Failed to fetch chat usage: Error: Internal server error", or "Chat not found". A non-JSON body logs a SyntaxError, which is log-only, so not reopened.Addressed in code
Optional: the spinner never stops when the chat can't start (TokenUsage.vue:110)No replyUnchanged. On 13.62 as it is ("active LLM not specified", "Connection closed"), the footer still spins after 30 s.Not addressed
Required: the warning icons were mouse-only (TokenUsage.vue:178)"Fixed, thanks" + screenshotAll three warnings are RcIconTooltip now. Tab from the count lands on button "More Info", focus opens the tooltip, and the warning is the button's description in the accessibility tree. axe is clean. Checked in the partial, none and real no-route states.Addressed
Footer still reads Tokens used 1.8K in total after a failed refresh
Comment 1: after a 500 on the refresh (stub), the count stays.
Warning icon focused from the keyboard with its tooltip open
Comment 4: reached with Tab; the tooltip opens on focus.
Footer reads Tokens used with a spinner under a configuration error
Comment 3: still spinning after 30 s on 13.62 as it is.

How the two passes compare

Pass A (Cody's new-commits process)Pass B (our process)Lands as
Thread statesAddressed · Addressed in code · Not addressed · AddressedSameBody line; resolve 3 threads
e2e counts fail on the headMissed: e2e was still running when Pass A finishedYes, from the CI log, with the llm-mock token formulaInline 1 (I re-checked the log)
A failed first load spinsYes: new chat with "Chat not found", 500 on first load; agent source readYes, same cases plus the UI-tools ConfigMap answered 404Inline 2
Count freezes after a reopenYes, round-one code: kept out per the new-commits ruleNot looked at (outside the new commits)Optional inline 3, your call
:deep restyle of RcIconTooltipSeen, look-onlyHeld as a look-only nitNot posted
multi-agent/chat.spec red on the headChecked: a 409 flake; the earlier unpinned run passed itSeen, cause not establishedNot posted
CostIn session, about 40 min including the preview rebuildFresh agent, 18 min, 75 tool calls, ~210k tokens

New in these commits

1

The new e2e token counts fail on this head's own CI

Pass B
Functional · test

66899a1 sets the third step to Input 41,948, Total 42,295. The e2e run on 9d17835 fails this test on all three attempts with Input 41,938 · Output 347 · Total 42,285 · supervisor 27,329 · Rancher 14,956 · agent 25,463 · ui-tools 16,822. Output and the earlier steps match; only the input side of this step is off, by 3 to 10 tokens.

llm-mock counts input tokens from the size of the request the agent sends, and this step's requests carry the listKubernetesResources result from the test cluster's cattle-ai-agent-system namespace. So the exact numbers follow that environment: the run before (34301df) never reached them because it failed on "42K" vs "42.3K", so 41,948 most likely came from a local run. 357913b also unpins the agent, so the counts now move with whatever image the agent's chart points to (alpha.8 today).

The other red spec on that run, multi-agent/chat.spec, starts with a 409 Conflict on an AIAgentConfig PUT. It passed on the earlier unpinned run, so it's a flake and not filed. Main is red too (multi-agent/message.spec).

2
Functional · UX

b41c5a7 returns null for every failure, and fetchUsage keeps the old value when it gets null. On a first load the old value is the null the reset just set, and the footer draws null as loading. So a failed first load never leaves the spinner until an answer refreshes it.

The common case: the welcome message only calls the LLM when the UI-tools definition is set up and enabled (toolsSelector). Without it, the new chat has no messages and no usage, and agent main answers 404 {"detail":"Chat not found"} (fetch_chat is None, 0 LLM calls). The footer shows "Tokens used:" spinning until the user sends something; at 2f157f3 the same answer read "Tokens used: 0". A 500 on a new chat's first load spins for 30 s and longer.

Same cause as your open spinner thread: null now means loading, failed and nothing yet. The new unit tests cover a failed refresh but not a failure right after a reset.

Footer reads Tokens used 0
Before (round two, same "Chat not found" answer): "Tokens used: 0".
Footer reads Tokens used with a spinner
Now: a new chat, 10 s after opening, still spinning.

Outside the new commits

3

The count stops updating after closing and reopening the panel

Pass A
Functional · your call

Closing the panel keeps the WebSocket open once the chat has started (Chat.vue unmount()), and the store's open returns early while a socket exists. So the socket's onmessage still belongs to the first panel. After a reopen, each answer runs the first panel's fetchUsage, which refreshes its chatUsage, not the one on screen.

Seen with stubs: 1.8K, then 2.6K after an answer. After a reopen, the next answer's usage request returned 3.5K for the same chat, and the footer still read 2.6K. Without a reopen, the refresh works. This is round-one code that both earlier rounds missed. It's functional and introduced by the PR, but not by these commits.

Chat panel with two answers and the footer reading 2.6K in total
After a reopen: two answers, the second one's request returned 3.5K, and the footer stays at 2.6K.

Checked, and not filed

Not checked: a real agent with the usage route (13.62 has none), Cypress locally (policy), and a narrow panel and dark theme for the new icons.

What I'd post

Request changes: a short body and two inline comments, plus a third only if you want the reopen point. No replies on the old threads. You can resolve comments 1, 2 and 4.

Review body

Thanks, the warning icons work from the keyboard now and a failed refresh keeps the count. Two points inline.

Inline 1, required · cypress/e2e/tests/features/chat/chat-token-usage.spec.ts:190

This run's own e2e gets 41,938 here on all three attempts (42,285 in total), so the new numbers don't hold: the input side includes what `listKubernetesResources` returns from the test cluster, and with the agent unpinned, the prompts of whatever version its chart points to.

Inline 2, required or optional · pkg/rancher-ai-ui/composables/useChatMessageComposable.ts:916

A failed first load now leaves the spinner on: a new chat whose welcome makes no LLM call (no UI tools set up) gets the agent's 404 "Chat not found", and the footer shows "Tokens used:" spinning until the first answer, where it read 0 before. A 500 on the first load does the same, so it's the earlier spinner thread on a common path.

[before / after screenshots]

Inline 3, only if you want it · pkg/rancher-ai-ui/composables/useChatMessageComposable.ts:91

Optional: after closing and reopening the panel, the count stops updating. The socket stays open, so each answer still runs the first panel's handler and refreshes its `chatUsage`, not the one on screen: here 2.6K stayed while the next answer's request returned 3.5K.

[panel screenshot]