Skip to content

docs(cli): clarify that --output json requires extraction strategy - #2335

Open
ege-arhan wants to merge 2 commits into
unclecode:developfrom
ege-arhan:fix/cli-json-output-help
Open

ege-arhan wants to merge 2 commits into
unclecode:developfrom
ege-arhan:fix/cli-json-output-help

Conversation

@ege-arhan

Copy link
Copy Markdown

Closes #2334

Clarifies in click option help that -o json expects an extraction strategy or schema to be set for structured extractions, preventing user confusion when raw markdown/HTML is returned.

@SohamKukreti

Copy link
Copy Markdown
Collaborator

Thanks for raising this and opening a fix!

A few changes before we can merge:

1. Base branch
This targets main. Please rebase onto develop. All changes go there first.

2. Help text wording
The real behavior is worse than the help text suggests. Without -j or -e, -o json crashes:

$ crwl https://example.com -o json
None
Error: the JSON object must be str, bytes or bytearray, not NoneType

We're fixing the crash in a separate maintainer PR: -o json without an extraction strategy will exit early with a clear error. So the help text just needs to say what json is for. Suggested wording:

json = extracted data only (needs -j or -e). Use 'all' (default) for the full crawl result.

@ege-arhan
ege-arhan force-pushed the fix/cli-json-output-help branch from 5184ba2 to 2229adc Compare October 12, 2026 11:08
@ege-arhan
ege-arhan changed the base branch from main to develop October 12, 2026 11:08
@ege-arhan

Copy link
Copy Markdown
Author

Updated help text to the suggested wording and rebased onto develop. Base branch updated as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs(cli): clarify --output json behavior when no extraction strategy is provided

2 participants