How our pull requests get an automatic first review

Our pull requests get an automatic first review as soon as they're opened. That review comes from a dedicated bot account and is a real GitHub review, either an approval or a request for changes, not just a comment. The goal is a consistent first pass that catches bugs, leaked secrets, injection risks and missing error handling before a person looks at the code. The review is also posted as a notification in the correct Slack channel for pull requests, so we can see what changed and what the bot thought of it without opening GitHub.

Behind it sits one Cloudflare Worker that receives GitHub webhooks. It accepts only POST requests and checks the X-Hub-Signature-256 header by computing an HMAC-SHA256 of the raw body with the Web Crypto API, then compares the two hex strings in constant time. It ignores anything that is not a pull_request event with the action opened, reopened or synchronize, and it skips pull requests opened by the bot itself to avoid loops. After that it replies 202 and hands the rest of the work to ctx.waitUntil, so GitHub gets an answer right away.

Next, the background task fetches the pull request as a raw diff from the GitHub API and trims it to 40,000 characters before sending it to Claude. The prompt asks for strict JSON containing a verdict and a review body. Real bugs, security problems or missing error handling should mean request-changes, while style nits should only be noted and approved. The worker submits the verdict to the reviews endpoint as APPROVE or REQUEST_CHANGES. It then converts the Markdown to Slack's mrkdwn format and posts it, as a reply under that day's brief message when it can find one.

Coordination turned out to be the tricky part. A separate description webhook also fires when a pull request is opened and posts its own announcement to Slack. The two workers run in parallel and share no state, so whichever model call finished first got posted first. The fix: on opened and reopened events, the review worker checks the pull request body once a second, for up to ten tries, until a marker comment from the description worker shows up. If the marker never appears, it posts anyway, so the review itself is never held up. Model output gets the same cautious treatment. Anything that won't parse or has an unknown verdict defaults to request-changes.