An always-on-top Windows 11 HUD that fact-checks what is said, as it is said. It transcribes your microphone and the call locally with Whisper. Each sentence that states a fact is checked by gpt-4.1-mini on your Azure AI Foundry resource, using Wikipedia articles as evidence. The sentence then turns green (correct) or red (incorrect), with the reason underneath. With a code repository connected, talk about the code is looked up in it: a question gets an answer and a statement a verdict, listed on the Code facts page.
Instead of Azure, claims can be checked by a model on your own PC (Ollama, Foundry Local, LM Studio), or by Azure with that local model taking over whenever Azure is down. On a call, the voices on the other side are told apart (Them, Them 2…), and when a meeting ends its summary can be posted to Slack, Teams or Discord, or emailed.
Audio never leaves the machine. Only the text of claims goes to your Foundry endpoint (or nowhere, with a local model only), and their key words go to Wikipedia's and Wikidata's search, and for a claim about the present to GDELT's news search (switch all three off with Look claims up in Settings). With a code repository connected, a sentence about the code goes to the endpoint with the list of the repository's file paths and up to three excerpts of its files (6,000 characters each). A meeting summary sends that day's transcript to the endpoint, only when you ask for one or mute after a conversation. With Watch the screen on, a picture of the monitor you are working on goes to your endpoint whenever it changes (at most every 3 s); it is never saved.
- First run: the window opens on Settings, and the microphone stays off until an endpoint and key pass the connection check (see below). The same happens at a later start if the key is refused or the deployment is gone.
- The window is an ordinary one: drag it by the headline, resize, snap, minimize, maximize or close it. Closing quits, so nothing listens after the window is gone. It stays on top of other windows so it can sit beside a call.
- The header is the title bar: the microphone with a level meter (its bars breathe while a claim is being checked), the latest verdict or status, a spinner while a claim is being checked, a small line of the share found correct after each recent verdict (red when most are wrong), the compact button and ⚙. Click the microphone, or use Mute in the tray menu, to mute. There is no global hotkey, so no shortcut of Teams or another app is taken. Muting drops anything not yet transcribed. The bar along the header's bottom edge is this session's verdicts as shares of green, red and amber. The tooltip of either gives the counts and how the model has done: calls, average time, tokens, retries and failures. A claim found incorrect washes the header red for a moment.
- Compact (the button left of ⚙): the window shrinks to its header, a strip to keep beside a call; the same button brings the pages back.
- Five pages, switched at the top of the window (in a narrow window the tabs drop their icons). All keep a readable width when the window is wide.
- Live: every sentence as it is heard, one per line, with the clock time; the newest slides in. A claim reads Checking… while the model works, then is highlighted Correct or Incorrect with the reason. A specific claim the model cannot settle gets a quiet Can't tell note. Questions, requests, opinions, predictions, remarks about your own company or team, and garbled speech are not claims: they stay plain and dimmed. The page follows the newest line unless you scroll up. With Tell voices apart on, each voice on the call has its own name: the first heard is Them, the next Them 2, and so on. Names renames You and Them for this call ("Alex"), on every page at once; for several voices, type a list in the order they spoke ("Alex, Sam"). The history keeps the plain sides and the voice numbers.
- Facts: only the claims found correct or incorrect, newest first. Each is restated to stand on its own ("It was finished in 1889" becomes "The Eiffel Tower was finished in 1889"), with its reason, its source, when it was said, and a thin bar for how sure the model was. A claim said twice is one row marked ×2, showing when it was said last. Filter the list by words or a speaker's name, and by kind (correct, incorrect, answers, said by you or by them). Copy puts the list as filtered on the clipboard.
- Code (Code facts, with a code repository connected): a sentence is about the code when it says a repository's name ("the Punkouter software website"), names a file, class or entity in it ("the invoice service"; a one-word name only for a code file, and short words must match exactly, so "we" is never web.config), or uses words like our code, the codebase, the repo, our app, our website, the backend, the API or the database. A sentence right after one of those counts too when it points back or asks ("Tell me about our repo." then "How many pages are in it?"); if it turns out to be about something else, it gets the ordinary check. Then, questions and requests ("tell me about…") included, it goes to the code rather than to Wikipedia: the model gets the list of every file path, the files the sentence names and the files whose text best matches its words. A question ("How many pages are on our website?") gets an answer and a statement ("The invoice service retries three times") a verdict; talk the files say nothing about is dropped. The Live line shows it, and Code facts lists answers, correct and incorrect (a Can't tell stays a quiet note on its line, as on Facts), newest first, with the files it rests on: each opens on GitHub or Azure DevOps (or on disk for a folder), and VS Code opens it at the line it is about (for a URL, that is the app's copy in
repos\, which the next refresh overwrites); each file has an icon for its kind (code, a page, a document, settings, data). With several repositories connected, the sentence goes to the one whose name it says, else the one with the files and classes it names, else the one whose text matches it best, and the row says which. The page filters and copies as Facts does. - Screen (Screen facts, with Watch the screen on): at the top, what the screen shows now in the model's words and since when (select it to copy, hover for all of it). Below, the claims judged against the screen, correct and incorrect, newest first, in the same rows as Facts and with the same filter and Copy. These are not on Facts.
- History: any day from the session log, searchable. Summary sums the day up with the model (main points, decisions, corrections, open questions, and what was asked or decided about the code) and keeps it beside the day's history; muting after a conversation (a dozen lines or more since the last summary) does the same in the background. Export saves the day with its summary as
Documents\PoVoiceHud\Meeting {date}.mdand opens it. Share posts the day's summary to the chat webhook set in Settings, or without one opens it as an email.
- When a meeting ends (you mute after it, or with Listen only during calls the call app lets go of the microphone), the day is summed up as above. With Offer to share when a meeting ends on, that meeting alone (from its first line) is summed up too, and a bar at the top offers Post to Slack (or Teams, Discord: whatever the webhook is), or Email it without a webhook. Nothing is sent until you press it.
- When checking fails (the service is down or slow, or the key stops working), the header says so once, Fact checks paused: …, and the lines stay plain. It says Fact checks are working again when they are. With Azure, and a local model when Azure is down, a failed call is answered by the local model instead, and every call goes there for the next 2 minutes before Azure is tried again; the header says Azure is not answering: checking with … on this PC, and Azure is answering again when it is. A sentence Azure's content filter refuses is judged by the local model too. A sentence that Azure's content filter refuses gets its own quiet Can't tell.
- Sources: a link to an article the app looked up itself is marked (looked up). A URL the model wrote is marked (cited by the model), because it may not exist.
- Tray icon: left click hides or shows the HUD. The menu has Mute/Unmute, Open the history folder, Copy diagnostics, Check for updates and Quit.
| Setting | |
|---|---|
| Where claims are checked | Azure AI Foundry; Azure, and a local model when Azure is down; or A local model only (nothing leaves the PC but what Look claims up sends). The page shows the fields the choice needs. |
| Azure AI Foundry endpoint | https://<resource>.services.ai.azure.com. The resource needs a deployment named gpt-4.1-mini. |
| Key | Kept in Windows Credential Manager (PoVoiceHud/FoundryKey), never in a file. |
| Local model server and Model | Any OpenAI-compatible server: Ollama (http://localhost:11434/v1), Foundry Local, LM Studio (http://localhost:1234/v1), and the model's name there (qwen2.5:7b). It must answer in a JSON schema (structured outputs); 7 billion parameters or more judge claims well. It gets three times Azure's time limit (at least 30 s), since the first call loads it. Apply asks it once; a model the server does not have says how to pull it. Its knowledge ends where its training did, so claims about the present need Look claims up. |
| Code repositories (optional) | One per line: a GitHub URL (https://github.com/org/repo), an Azure DevOps URL (https://dev.azure.com/org/project/_git/repo, as Clone copies it) or a folder on this PC that holds a git repository. A URL is cloned into repos\ and refreshed at each start (if that fails, last time's copy is used and the header says so). A folder is watched: a saved file is read again on its own, a commit or a new file rebuilds the index. Their names and file names also go into Whisper's vocabulary. Apply reads each new repository first and says what it found. See Code above for what they are used for. |
| Repository access token | Only for private repositories (one token for all): a GitHub personal access token with read access to contents, or an Azure DevOps one with Code (Read). Kept in Credential Manager (PoVoiceHud/RepoToken). |
| Advanced: Transcription | Which Whisper model: Fast base.en (148 MB), More accurate small.en (264 MB, 8-bit) or Most accurate large-v3 turbo (574 MB, 5-bit; multilingual, best with a graphics card; the default). Downloaded once when applied, checked against a pinned SHA-256. |
| Cue volume | How loud the cues are. |
| Advanced: Chat webhook for meeting summaries | Optional: a Slack incoming webhook, a Teams Workflows "when a Teams webhook request is received" URL (an Adaptive Card is posted), a Discord webhook (cut to 2,000 characters), or any https URL that takes JSON ({"title", "text"}). Kept in Credential Manager (PoVoiceHud/ShareWebhook). |
The rows marked Advanced are folded under Advanced: transcription, sharing at the bottom of the page.
The page has no on/off switches. These behave one way, and only settings.json changes them (Apply keeps whatever the file says):
| Behaviour | Key | ||
|---|---|---|---|
| Look claims up | always on | LookUpClaims |
Up to three Wikipedia article intros, Wikidata's key facts (dates, heights, populations, who leads what) for the thing the claim names, and for a claim about the present ("current", "still"…) up to three recent news headlines from GDELT go to the model as evidence, all at once (2.5 s limit each; a slow or failed one is left out). Off: nothing goes to any of them. Set it to false and claims are judged from the model's own knowledge, with nothing sent to any of them. |
| Start with Windows | always off | StartWithWindows |
Adds PoVoiceHud to the Windows startup list (HKCU\...\Run). |
| Listen only during calls | always on | CallsOnly |
Hear nothing until a call app (Teams, Zoom, Slack, Webex, Discord, Skype, a browser call…) has a microphone open, found from Windows' audio sessions every 3 s, and stop when it closes it. The header says Waiting for a call meanwhile; the mute button stays as you left it. |
| Sounds and animations | always on | Sounds |
A short cue per verdict, a bell for an answer from the code, and one for mute (each under 240 ms, so call capture never transcribes one); the highlight fading in, the newest line sliding in, the level bars breathing while a claim is checked, and the red wash for an incorrect claim. Windows' Animation effects switch also turns the animations off. Cue volume sets how loud the cues are. |
| Watch the screen | always on | WatchScreen |
Every 3 s the monitor holding the active window is captured; when it has visibly changed, gpt-4.1-mini describes it in a few lines (the document or form, and the names, dates, numbers and field values it shows). Each check then gets that description, so "the start date is October 1st" is Incorrect — On screen: the form says 09/28/2026. and is listed on Screen facts. The pictures stay in memory; the descriptions go to the history. While it is on, the HUD is hidden from captures, so it never reads its own verdicts back, and from screen sharing too. Paused while muted. With a local model, that model must accept pictures (such as qwen2.5vl or llava). |
| Use the graphics card | always on | UseGpu |
Whisper runs through Vulkan on any NVIDIA, AMD or Intel graphics card, and on the CPU when there is none or its driver fails. It takes effect at the next start (the runtime loads once). On integrated graphics base.en is no faster (measured: 466 ms on Vulkan, 435 ms on the CPU for the test sentence); the bigger models gain most. |
| Tell voices apart | always on | SeparateSpeakers |
Every utterance from the call gets a voice print from WeSpeaker's ResNet34 (26 MB, run with ONNX Runtime on this PC), and prints are grouped into voices as they come: Them, Them 2… A voice is new only after 1.5 s of speech; under 1 s a line keeps the plain Them. Ten quiet minutes start the numbering again. |
| Offer to share when a meeting ends | always off | OfferToShare |
Set it to true and, when a meeting ends, a bar offers to post that meeting's summary to the webhook (or email it); nothing is sent until its button is pressed. History's Share posts a day's summary either way. |
Settings is a page in the window, so switching to the Azure portal to copy a key leaves it as it is. Apply saves and uses the changes without a restart. When the endpoint or key changed, it first sends one message to the model; if that fails, it stays on the page and says why (key refused, no such deployment, unreachable). Cancel (or Esc, or ⚙ again) puts back what was applied. Applying only replaces the fact checker: Whisper and the audio keep running. The page's corner shows when the copy was built (Build YYMMDD) and its version. %LOCALAPPDATA%\PoVoiceHud\settings.json holds the above plus four keys for the file only: WhisperThreads, WhisperLanguage (en, another language code, or auto; only large-v3-turbo hears more than English), Vocabulary (comma-separated names Whisper should expect) and UpdateFeed. Endpoints, key, Whisper model, voices and the screen switch apply without a restart; only the graphics card switch waits for the next start.
%LOCALAPPDATA%\PoVoiceHud\ |
|
|---|---|
| Settings | settings.json |
| History | sessions\{date}.jsonl: every line heard (with its voice number on a call whose voices are told apart), every verdict and each new screen description, on this PC only; sessions\{date}-summary.md: that day's meeting summary, when there is one |
| Exports | Documents\PoVoiceHud\Meeting {date}.md, written by Export on the History page |
| Logs | logs\{date}.log (14 days) and crash.log. Copy diagnostics zips these with settings.json and a machine description, but never the history or the key. |
| Repositories | repos\: the clone of a repository URL, one folder each |
| Models | models\: the chosen Whisper model (ggml-large-v3-turbo-q5_0.bin, ~574 MB, by default), silero_vad.onnx and wespeaker_resnet34.onnx (26 MB, for Tell voices apart), downloaded when first needed and checked against a pinned SHA-256 |
dotnet build PoVoiceHud.sln
dotnet run --project src/PoVoiceHud.App # the real app: opens the microphone
dotnet run --project src/PoVoiceHud.App -- --demo # a sample conversation, no audio (Debug builds)
dotnet run --project src/PoVoiceHud.App -- --demo --facts # the same, on the Facts page
dotnet run --project src/PoVoiceHud.App -- --demo --code # the same, on the Code facts page
dotnet run --project src/PoVoiceHud.App -- --demo --history # the History page (reads the real session log)
dotnet run --project src/PoVoiceHud.App -- --demo --compact # the header alone
dotnet run --project src/PoVoiceHud.App -- --demo --scripted # sentences through the real routing, checker and code answerer, with a canned model and this repo
dotnet run --project src/PoVoiceHud.App -- --demo --settings # the Settings page
dotnet run --project src/PoVoiceHud.App -- --demo --setup # the first-run page
dotnet run --project src/PoVoiceHud.App -- --demo --share # the offer to post a meeting's summary
dotnet test --solution PoVoiceHud.sln # 105 tests: 81 unit + 24 integration (models, Credential Manager, git)
pwsh build/pack.ps1 # Setup.exe + update packages in artifacts/releases (Velopack)Requirements: Windows 11, .NET 10 SDK, a microphone. No GPU is needed; one with a Vulkan driver is used when there is one. Build output goes to artifacts/.
Install and update: Setup.exe installs per user (no admin) to %LOCALAPPDATA%\PoVoiceHud.App. It is not code-signed, so SmartScreen warns the first time (More info → Run anyway). Raise <Version> in src/PoVoiceHud.App/PoVoiceHud.App.csproj before packing, and keep artifacts/releases between versions for delta updates. With UpdateFeed set to an https URL or a folder (publish with pwsh build/pack.ps1 -Feed <folder>), an installed copy downloads a newer version in the background and installs it when it closes.
src/PoVoiceHud.Core has everything testable and never references WinUI. src/PoVoiceHud.App is the WinUI 3 window, tray icon and Velopack entry point. tests/PoVoiceHud.Core.Tests is xUnit v3.
WasapiDualCapture: 16 kHz mono frames from the mic and the speakers' loopback, mixed. There is no switch for this: the other side of a call is always heard and checked too, and each line is labelled You or Them by which input was louder (? when neither was). Anything else playing on the default communications device is heard as well. Use a headset: on open speakers the microphone hears the call a second time.SileroVad+UtteranceSegmenter: one utterance per sentence. Anything under 300 ms is dropped, which is why the cues stay shorter.WhisperTranscriber: the selected model (Settings defaults tolarge-v3-turbo) on Vulkan or the CPU. Beside it, for an utterance from the call,CallVoicesmakes a voice print (Fbank: Kaldi's 80-bin log-mel filterbank in C#, as WeSpeaker computes it; thenVoicePrinter, ResNet34 on ONNX Runtime) andVoiceClustersassigns it to a voice by cosine to each voice's average (0.4 or more is the same voice).HudPipeline.Sentencessplits what it heard into sentences (not at initials or titles), and each one is a line on the Live page, stamped with the clock time it started.FactChecker.IsCandidate: a generous word-list check. It sends on every sentence that has at least three words (numbers count), is not only fillers, and is not a plain question. "…, right?" after a statement still goes; the model'snot_a_claimsettles requests and chatter.- A sentence about the code repository goes to
CodeAnswererinstead (step 4 does not apply, so questions go too): the file list, the files named (SymbolMatcheronCodeRepository's index) and a text search, toFoundryClientwith its own reply schema (question or statement, evidence, answer or reason, then the verdict, files). The verdict comes after the evidence so the model looks before it decides, and a statement keeps Correct or Incorrect only when its evidence is a quote of at least 12 characters from an excerpt it was sent, or a path it was sent; otherwise it is Can't tell. Whisper's common mishearing of "our" as "are" ("which model does are app use") still counts as code words, and a question Whisper ends with a full stop is still a question. With several repositories,CodeAnswerer.Pickchooses one per sentence. Any other candidate goes toFactChecker: evidence from Wikipedia, Wikidata and (about the present) GDELT news, all at once (EvidenceSources), thenFoundryClient(gpt-4.1-mini, or a local model on an OpenAI-compatible server, orFallbackModelwith both; verdict JSON schema with the claim, reason and source before the verdict and a 0–1 confidence after it, 8 s per attempt, retried on 429/5xx; each call's time, tokens, retries and failures counted inModelStats). The sentence travels as JSON data with today's date and the two sentences heard before it as context, so speech cannot pass as instructions. A sentence that says "it" or "they" is looked up together with the one before it. The model restates the claim to stand on its own. The prompt counts opinions, predictions, anecdotes, remarks about the speaker's own company or team, vague remarks and garbled speech asnot_a_claim, and those lines stay plain. Verdicts are cached per claim, and per context when the claim leans on it. ScreenWatcher(with Watch the screen):ScreenCapturetakes the monitor every 3 s with GDI, scaled to 1600 px;ScreenShot.Differscompares 64×36 grey thumbnails, and only a change in over 3 % of the cells (more than a call's video tiles) sends a JPEG toFoundryClient(20 s limit,{"screen": …}schema).Currentis the latest description; a failed read clears it rather than leave a stale one. Each check in step 5 carries it asscreen: a statement about what is shown is judged against it (the model gives"screen"as source, and the reason reads On screen: …;Insight.OnScreensends it to Screen facts), even about the speaker's own project, and the rule that the present needs a looked-up article does not apply.HudViewModel: the line takes its verdict, and correct and incorrect claims go to Facts (or Screen facts); the header's trend line and cue follow each verdict. Everything heard and checked is appended to the session log, which the History page reads back.
Transcription and checking are separate stages, and up to three claims are checked at once, so a slow model never delays the next line or the next claim. Muting bumps a generation counter, and anything still in flight is discarded. A checker that cannot be reached throws FactCheckUnavailableException, which the pipeline reports once as a status.
Design notes
- Fast CPU option:
base.enwith a 10 s audio context takes about 300–450 ms per sentence on 8 CPU threads.tiny.enwas faster but wrong. The bigger models are there for accuracy, and Vulkan for a graphics card that can carry them; the CPU stays the fallback, so a broken driver costs speed, not transcription. A vocabulary prompt adds about 80–100 ms, so it is kept to 120 characters. - A fixed cloud model:
gpt-4.1-minianswers in about 1–1.5 s and honours the JSON schema.Llama-3.3-70B-Instructwas as fast on average but now and then ran past the 8 s limit. - Voice prints on ONNX Runtime, not sherpa-onnx: sherpa-onnx's C# package would tell voices apart in a few lines, but it ships its own
onnxruntime.dll, which clashes with the one Silero already loads. So its WeSpeaker model runs on the existing ONNX Runtime, with the filterbank (about 100 lines) written here. Two synthetic voices score 0.85–0.91 against themselves and about 0.2 against each other; two halves of one short real sentence score 0.49, hence the 0.4 threshold, compared with a voice's running average rather than a single print. - A local model as an OpenAI-compatible server: Ollama, Foundry Local and LM Studio all speak
/v1/chat/completionswith JSON-schema replies, so one client covers them, and the app never manages a model itself. The fallback is a circuit shared by every client: one failure moves them all for 2 minutes, so an outage costs one slow call rather than one per claim. - Webhooks rather than Graph for sharing: an incoming webhook needs no sign-in, no app registration and no permission to read anyone's mail or calendar, and Slack, Teams (Workflows) and Discord all take one. Only the meeting just held is posted, never the whole day, and only when the button is pressed.
- Wikipedia as evidence: a model judging alone invents sources. With the article intros it cites a page the app fetched itself, which the Facts page can mark (looked up). Wikidata adds the exact numbers and dates a model's memory slips on. For the present, GDELT's news search is open data meant for this; Google's and Bing's news feeds answer faster but their terms allow only showing them in a personal feed reader.
- A deterministic pre-filter, not an LLM router: it costs nothing and is easy to test. It drops only what cannot be a claim: a missed claim costs more than a model call, and
not_a_claimcatches the rest. - Reason, then decide: both reply schemas put the verdict after the reasoning (and the evidence, for the code), since a model writes its properties in schema order. With the verdict first it decided, then explained: "the app checks facts with Llama" came back correct because gpt-4.1-mini is "a language model similar to Llama".
- JSON without reflection: settings, the session log and the window placement go through a source-generated
CoreJsoncontext, and prompts are built from JSON nodes, soPoVoiceHud.Corepasses the trimming and Native AOT analyzers (IsAotCompatible).Settingskeeps plain setters: the generated reader fills everyinitproperty, so a key missing from an older file would lose its default. A full Native AOT publish of the app also needs Visual Studio's C++ build tools for its linker. - The present needs a source:
gpt-4.1-mini's knowledge ends in mid-2024. Even when told the date, it judged "the current president is…" from its training. So a claim about now ("current", "still", "latest"…) keeps Correct or Incorrect only when it rests on an article looked up for it; otherwise it is Can't tell. - The screen as text, on change only: describing a picture once and handing the text to every check costs one image call per new screen, not one per sentence. A thumbnail diff decides what is new, so a still form with a call's video tiles beside it is described once.
- Settings check before they apply: an endpoint or key is saved only once the model has answered with it, and the app does not listen until one has. It never runs in a state where every claim fails.
Working rules for coding agents are in AGENTS.md; docs/manual-test.md is the checklist for what the tests cannot reach.