[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-07-03

[github-actions[bot]]

Summary

• Date of test: 2026-07-03
• Pages visited:
  • https://github.github.com/gh-aw/
  • https://github.github.com/gh-aw/setup/quick-start/
  • https://github.github.com/gh-aw/setup/cli/
• Overall impression: As a brand-new user, the home page is polished and clearly communicates the value proposition. The Quick Start guide is detailed but has a few spots that stumble a new user — particularly around why a separate GitHub token is needed, and what frontmatter means before it's explained.

---

🔴 Critical Issues Found

1. Missing setup notes for OpenAI (Codex) and Gemini

The Quick Start lists four AI engine options in the Prerequisites but provides inline setup notes for only two (Copilot and Anthropic). OpenAI and Gemini users are directed to a generic Authentication reference link with no inline quick-reference.

Impact: A new user picking Codex or Gemini at the add-wizard prompt gets stuck — no quick-copy token secret name or setup steps in the guide.

Recommendation: Add collapsed [!NOTE] blocks for OPENAI_API_KEY and GEMINI_API_KEY mirroring the existing Anthropic block.

---

2. gh aw add-wizard skips gh aw init

The CLI reference lists gh aw init as "Set up your repository for agentic workflows" — but the Quick Start never mentions it. A beginner reading the CLI commands page will wonder: "Should I run init first? Does add-wizard include that? What does init actually set up that add-wizard doesn't?"

Impact: Confusion about the correct entry path — init vs add-wizard.

Recommendation: Either mention in the Quick Start that add-wizard handles repository setup automatically (making init optional for the basic flow), or add a brief note clarifying when init is needed.

---

🟡 Confusing Areas

3. COPILOT_GITHUB_TOKEN vs GITHUB_TOKEN — confusing for beginners

The phrase "a separate GitHub token with Copilot access — distinct from the default GITHUB_TOKEN" appears mid-sentence in a dense paragraph. New users aren't yet familiar with GITHUB_TOKEN (the built-in Actions token), so distinguishing it from a personal PAT with Copilot access requires re-reading multiple times.

Where: Quick Start → Step 2 → Step 3 bullet

Recommendation: Pull this distinction into the inline NOTE block with a one-line callout: "⚠️ This is NOT the built-in Actions token — it is a Personal Access Token (PAT) you create manually."

---

4. "Frontmatter" jargon introduced before definition

Step 4 (Customize) says "If you changed the frontmatter (the YAML configuration block...)". The word "frontmatter" appears in parentheses with a definition — but the same word is also used as a link to a reference page. A complete beginner may not know to click the link to learn more.

Where: Quick Start → Step 4 → item 3

Recommendation: Good job including the inline definition! Consider making the parenthetical more prominent — e.g., using a > [!TIP] callout the first time frontmatter is mentioned.

---

5. .lock.yml file mystery — no visual example

The guide tells you two files get added (.md and .lock.yml) and instructs "never edit it by hand." But there's no example of what the file looks like or a quick explanation of why it exists. The NOTE just says to read a reference page.

Where: Quick Start → Step 2 → NOTE block

Recommendation: Add one sentence: "Think of it as the compiled version — like a .pyc from a .py file." This gives beginners an analogy to anchor the concept.

---

6. CLI page — sidebar label ordering confusion

The CLI page has sidebar: order: 200, placing it after Creating Workflows in the sidebar. But it's linked from the home nav as "CLI Commands" and from Quick Start as a natural next step. New users may not find it by browsing the sidebar.

Where: CLI Commands page sidebar position

Recommendation: Lower the sidebar order or add a direct "Next: CLI Commands" link at the bottom of the Quick Start page.

---

🟢 What Worked Well

• Home page is excellent: the tagline immediately explains what this is, the hero CTA buttons ("Quick Start with CLI" and "Creating Workflows") are prominent and labeled clearly. Zero ambiguity on where to start. ✅
• Video at the top of Quick Start is a great addition — seeing the CLI in action before reading steps sets the right mental model. ✅
• Estimated time (⏱️ 10 minutes) at the top of Quick Start is reassuring for new users. ✅
• "If you already have GitHub Copilot, start there" callout in Prerequisites is very helpful — removes decision fatigue. ✅
• CLI Commands table at the top of the CLI page is an excellent quick reference — the 10-row table gives a new user a complete mental map of the tool's capabilities instantly. ✅
• Interactive add-wizard concept described well — listing all 5 sub-steps gives beginners confidence they won't miss anything. ✅
• Public Preview notice is clearly placed and honestly worded. ✅

---

Recommendations

Quick wins (low effort, high impact):

1. Add [!NOTE] blocks for OPENAI_API_KEY and GEMINI_API_KEY setup in Quick Start (mirrors the existing Anthropic block).
2. Add one sentence clarifying COPILOT_GITHUB_TOKEN ≠ GITHUB_TOKEN in the relevant NOTE block.
3. Add an analogy for .lock.yml (e.g., "compiled output, like a .pyc file").
4. Clarify init vs add-wizard relationship in the Quick Start intro.

Longer-term improvements:
5. Add a "Glossary" page or sidebar section covering: frontmatter, lock file, safe outputs, agentic workflow, engine — terms that appear on page 1 but are defined elsewhere.
6. Consider a "Which AI engine should I pick?" decision tree on the Prerequisites section.
7. Lower CLI Commands sidebar order so it naturally follows Quick Start.

---

Screenshots

📎 [home.png] — Home page (full page)
https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/3f4eb3464faccb9c02614a12097408687af5e9a9187526a140da64d67779f76e.png?raw=true

📎 [quick-start.png] — Quick Start guide (full page)
https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/155c1b8a2a8174d9b68dc805d28d0b24108f2c69850e337815323b83ce3d44b4.png?raw=true

📎 [quick-start-top.png] — Quick Start hero/video viewport
https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/4b40137efde21f9ab511369b1e146ca63d2e4778a89f2295ff83ff170a0aa8d8.png?raw=true

📎 [cli-commands.png] — CLI Commands page (full page)
https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/e80c76ab79fd861116395a27c85a1f66a1ed9a7a117a94f0dad02650cc12ee42.png?raw=true

📎 [cli-top.png] — CLI Commands table viewport
https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/52dab7acc61a935a1d3f9daddaa6be4e6a64f8ecf89949f43cbdb05d65e297ce.png?raw=true

---

References: §28639371921

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · 88.7 AIC · ⌖ 20.1 AIC · ⊞ 6.7K · ◷

• expires on Jul 3, 2026, 9:08 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-07-04T05:08:08.455Z.

Closed by Workflow