Repository navigation
Closed
Labels
Awaiting PaymentAuto-added when associated PR is deployed to productionAuto-added when associated PR is deployed to productionDailyKSv2KSv2
Description
Summary
Coming from this internal discussion, we want to automate our PR quality review process for HelpDot articles. Right now, we manually copy/paste file contents into a GPT for QA. This should be replaced with a GitHub Action that automatically runs on relevant PRs.
The Action should evaluate all modified .md files and score each file for:
- Readability (score 1–10)
- AI Readiness (score 1–10)
- Expensify Style Compliance (score 1–10)
The goal is to ensure every HelpDot contribution is AI-ready, easy to understand, and matches our formatting and voice/tone guidelines.
Trigger Conditions
This check should only run when PRs include changes to .md files within the articles/docs directory (including subfolders).
Ignore any changes outside that path.
Prompt to Use (for each file)
Click to expand
## Support Doc Optimizer – QA Prompt for GitHub Action
This prompt will be used in a GitHub Action to automatically evaluate Expensify HelpDot article pull requests. It mirrors the internal Support Doc Optimizer GPT and must return structured scores and improvement suggestions.
---
### Prompt (for each changed file)
You are **Support Doc Optimizer** — an AI trained to evaluate HelpDot articles written for Expensify.
Your goal is to analyze the file below and score it in three categories using the criteria provided. Then, offer actionable, markdown-formatted feedback for the author.
---
## 1. Readability (score 1–10)
Score based on:
- Sentence clarity, grammar, and scannability
- Logical flow and ordering of sections
- Reading level is 8th grade or below
- Avoids unnecessary jargon or filler
- Uses numbered steps and bullet points correctly
---
## 2. AI Readiness (score 1–10)
Score based on:
- Full feature names are used in **every heading**
- Headings are descriptive (e.g., “Where to find Statement Matching” not “Where to find it”)
- No vague references like “this,” “that,” or “it” without clear context
- Includes YAML metadata block at the top:
```yaml
---
title: [Full article title]
description: [Concise, benefit-focused summary]
keywords: [feature name, related terms, navigation path, etc.]
---
- Includes one of the required wrappers:
- ‹div id="new-expensify" markdown="1"> for New Expensify
- ‹div id="expensify-classic" markdown="1"> for Classic
- Contains a breadcrumb path below the H1, formatted like:
- Settings > Workspaces > Workspace Name > People
- Account > Settings > Payments
- All headings use only # or ##. No ### or deeper.
## 3. Expensify Style Compliance (score 1-10)
Score based on:
- Voice and tone match Expensify guidelines:
- Casual yet professional
- Clear and concise
- Contractions allowed (e.g., it's, won't)
- Avoids excessive exclamation marks (max 1 per 400 words)
- Uses Expensify-approved terminology:
- "Workspace," not "policy"
- "Member," not "user"
• "Workspace Admin," "Domain Owner," etc.
- Button labels and actions follow Ul terms:
- Use "Next," not "Continue"
- Use "Confirm," not "Save" at end of flows
- Markdown formatting rules are followed:
- Bullet list = no period if a phrase
- Bullet list = has period if a full sentence
- FAQ section:
- Uses # FAQ as the heading
- All questions are #*# subheadings
- Answers are plain text (not collapsive)
## Output Format (markdown)
Return your review using this exact markdown structure:
## QA Review for: [FILENAME]
### Scores
- Readability: X/10
- AI Readiness: X/10
- Style Compliance: X/10
### Summary
Explain each score clearly. What did the article do well? Where did it fall short?
### Suggestions for Improvement
- List clear, actionable changes
- Use bullet points (e.g., “Add breadcrumb below H1”, “Update heading to use full feature name”, “Use correct button label”)
## Constraints
- Evaluate the text exactly as written — do not hallucinate or infer missing intent.
- Do not assume audience unless specified (default = Workspace Admin).
- Do not suggest changes outside the defined Expensify style and formatting rules.
- If required structure is missing (metadata, wrapper, breadcrumbs), call it out.
## Notes for Engineers
- This prompt will be used with OpenAI's GPT-4o or ConciergeGPT.
- It should be run against .md, .txt, or .mdx files modified in a PR.
- If any score is below 7, you can:
- Fail the GitHub check (exit 1)
- Or post a comment on the PR with feedback
</details>
Issue Owner
Current Issue Owner: @stephanieelliottMetadata
Metadata
Labels
Awaiting PaymentAuto-added when associated PR is deployed to productionAuto-added when associated PR is deployed to productionDailyKSv2KSv2
Activity
stephanieelliott commentedon Aug 18, 2025
Currently we run this check manually using a GPT, so it's probably the lowest priority out of the 3 QoL update issues assigned.
@rushatgabhane in case it helps, I included the prompt we use in the issue body
stephanieelliott commentedon Aug 21, 2025
Discussing here: #65828 (comment)
melvin-bot commentedon Aug 26, 2025
@rushatgabhane, @stephanieelliott Eep! 4 days overdue now. Issues have feelings too...
stephanieelliott commentedon Aug 27, 2025
Hey @rushatgabhane any update on this one?
23 remaining items
stephanieelliott commentedon Sep 15, 2025
This works!!!! And it's awesome - fab, thank you @rushatgabhane!!!
[-][Due for payment 2025-09-19] Automated PR Quality Check for HelpDot Articles[/-][+][Due for payment 2025-09-23] [Due for payment 2025-09-19] Automated PR Quality Check for HelpDot Articles[/+]melvin-bot commentedon Sep 16, 2025
The solution for this issue has been 🚀 deployed to production 🚀 in version 9.2.14-7 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:
If no regressions arise, payment will be issued on 2025-09-23. 🎊
For reference, here are some details about the assignees on this issue:
[-][Due for payment 2025-09-23] [Due for payment 2025-09-19] Automated PR Quality Check for HelpDot Articles[/-][+][$500] [Due for payment 2025-09-23] [Due for payment 2025-09-19] Automated PR Quality Check for HelpDot Articles[/+]melvin-bot commentedon Sep 23, 2025
stephanieelliott commentedon Sep 23, 2025
As discussed, upping this one to $500 since it required some experimentation and lots of tweaking to get running.
stephanieelliott commentedon Sep 23, 2025
Payment Summary
Contributor: @rushatgabhane due $500 via NedDot
Contributor+: N/A internal
garrettmknight commentedon Sep 24, 2025
$500 approved for @rushatgabhane