Skip to content

[$500] [Due for payment 2025-09-23] [Due for payment 2025-09-19] Automated PR Quality Check for HelpDot Articles #68503

Closed

Description

@stephanieelliott

Summary

Coming from this internal discussion, we want to automate our PR quality review process for HelpDot articles. Right now, we manually copy/paste file contents into a GPT for QA. This should be replaced with a GitHub Action that automatically runs on relevant PRs.

The Action should evaluate all modified .md files and score each file for:

  1. Readability (score 1–10)
  2. AI Readiness (score 1–10)
  3. Expensify Style Compliance (score 1–10)

The goal is to ensure every HelpDot contribution is AI-ready, easy to understand, and matches our formatting and voice/tone guidelines.

Trigger Conditions

This check should only run when PRs include changes to .md files within the articles/docs directory (including subfolders).
Ignore any changes outside that path.


Prompt to Use (for each file)

Click to expand
## Support Doc Optimizer – QA Prompt for GitHub Action

This prompt will be used in a GitHub Action to automatically evaluate Expensify HelpDot article pull requests. It mirrors the internal Support Doc Optimizer GPT and must return structured scores and improvement suggestions.

---

### Prompt (for each changed file)

You are **Support Doc Optimizer** — an AI trained to evaluate HelpDot articles written for Expensify.

Your goal is to analyze the file below and score it in three categories using the criteria provided. Then, offer actionable, markdown-formatted feedback for the author.

---

## 1. Readability (score 1–10)

Score based on:
- Sentence clarity, grammar, and scannability
- Logical flow and ordering of sections
- Reading level is 8th grade or below
- Avoids unnecessary jargon or filler
- Uses numbered steps and bullet points correctly

---

## 2. AI Readiness (score 1–10)

Score based on:
- Full feature names are used in **every heading**
- Headings are descriptive (e.g., “Where to find Statement Matching” not “Where to find it”)
- No vague references like “this,” “that,” or “it” without clear context
- Includes YAML metadata block at the top:

```yaml
---
title: [Full article title]
description: [Concise, benefit-focused summary]
keywords: [feature name, related terms, navigation path, etc.]
---

-  Includes one of the required wrappers:
  - ‹div id="new-expensify" markdown="1"> for New Expensify
  - ‹div id="expensify-classic" markdown="1"> for Classic
-  Contains a breadcrumb path below the H1, formatted like:
  - Settings > Workspaces > Workspace Name > People
  - Account > Settings > Payments
-  All headings use only # or ##. No ### or deeper.

## 3. Expensify Style Compliance (score 1-10)
Score based on:
 - Voice and tone match Expensify guidelines:
  - Casual yet professional
  - Clear and concise
  - Contractions allowed (e.g., it's, won't)
  - Avoids excessive exclamation marks (max 1 per 400 words)
 - Uses Expensify-approved terminology:
  - "Workspace," not "policy"
  - "Member," not "user"
• "Workspace Admin," "Domain Owner," etc.
 - Button labels and actions follow Ul terms:
  - Use "Next," not "Continue"
  - Use "Confirm," not "Save" at end of flows
 - Markdown formatting rules are followed:
  - Bullet list = no period if a phrase
  - Bullet list = has period if a full sentence
 - FAQ section:
  - Uses # FAQ as the heading
  - All questions are #*# subheadings
  - Answers are plain text (not collapsive)

##  Output Format (markdown)

Return your review using this exact markdown structure:

## QA Review for: [FILENAME]

### Scores
- Readability: X/10
- AI Readiness: X/10
- Style Compliance: X/10

### Summary
Explain each score clearly. What did the article do well? Where did it fall short?

### Suggestions for Improvement
- List clear, actionable changes
- Use bullet points (e.g., “Add breadcrumb below H1”, “Update heading to use full feature name”, “Use correct button label”)

##  Constraints
 - Evaluate the text exactly as written — do not hallucinate or infer missing intent.
 - Do not assume audience unless specified (default = Workspace Admin).
 - Do not suggest changes outside the defined Expensify style and formatting rules.
 - If required structure is missing (metadata, wrapper, breadcrumbs), call it out.

## Notes for Engineers
 - This prompt will be used with OpenAI's GPT-4o or ConciergeGPT.
 - It should be run against .md, .txt, or .mdx files modified in a PR.
 - If any score is below 7, you can:
 - Fail the GitHub check (exit 1)
 - Or post a comment on the PR with feedback

</details> 
Issue OwnerCurrent Issue Owner: @stephanieelliott

Activity

  1. stephanieelliott commented on Aug 18, 2025

    @stephanieelliott
    ContributorAuthor

    Currently we run this check manually using a GPT, so it's probably the lowest priority out of the 3 QoL update issues assigned.

    @rushatgabhane in case it helps, I included the prompt we use in the issue body

  2. stephanieelliott commented on Aug 21, 2025

    @stephanieelliott
    ContributorAuthor

    Discussing here: #65828 (comment)

  3. melvin-bot commented on Aug 26, 2025

    @melvin-bot

    @rushatgabhane, @stephanieelliott Eep! 4 days overdue now. Issues have feelings too...

  4. stephanieelliott commented on Aug 27, 2025

    @stephanieelliott
    ContributorAuthor

    Hey @rushatgabhane any update on this one?

  5. 23 remaining items

  6. stephanieelliott commented on Sep 15, 2025

    @stephanieelliott
    ContributorAuthor

    This works!!!! And it's awesome - fab, thank you @rushatgabhane!!!

    Image
  7. changed the title [-][Due for payment 2025-09-19] Automated PR Quality Check for HelpDot Articles[/-] [+][Due for payment 2025-09-23] [Due for payment 2025-09-19] Automated PR Quality Check for HelpDot Articles[/+] on Sep 16, 2025
  8. melvin-bot commented on Sep 16, 2025

    @melvin-bot

    The solution for this issue has been 🚀 deployed to production 🚀 in version 9.2.14-7 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

    If no regressions arise, payment will be issued on 2025-09-23. 🎊

    For reference, here are some details about the assignees on this issue:

  9. changed the title [-][Due for payment 2025-09-23] [Due for payment 2025-09-19] Automated PR Quality Check for HelpDot Articles[/-] [+][$500] [Due for payment 2025-09-23] [Due for payment 2025-09-19] Automated PR Quality Check for HelpDot Articles[/+] on Sep 23, 2025
  10. melvin-bot commented on Sep 23, 2025

    @melvin-bot

    ⚠️ Could not update price automatically because there is no linked Upwork Job ID. The BZ team member will need to update the price manually in Upwork.

  11. stephanieelliott commented on Sep 23, 2025

    @stephanieelliott
    ContributorAuthor

    As discussed, upping this one to $500 since it required some experimentation and lots of tweaking to get running.

  12. stephanieelliott commented on Sep 23, 2025

    @stephanieelliott
    ContributorAuthor

    Payment Summary

    Contributor: @rushatgabhane due $500 via NedDot
    Contributor+: N/A internal

  13. garrettmknight commented on Sep 24, 2025

    @garrettmknight
    Contributor

    $500 approved for @rushatgabhane

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Awaiting PaymentAuto-added when associated PR is deployed to productionDailyKSv2

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions