Requirements
- An endpoint that accepts
POSTrequests over HTTPS and is reachable from the public internet. - Membership of the workspace. Any member can set up the webhook and reveal its signing secret.
- Run events webhooks turned on for your workspace. Ask your QA Wolf representative to turn them on. Until then, Workspace settings → Integrations has no Run events webhook section.
Set up the webhook
1
Open Workspace settings → Integrations and find the Run events webhook section.
2
Select Reveal signing secret, then Copy signing secret. Store the secret where your endpoint can read it. Your endpoint uses it to verify each request.
3
Enter your endpoint’s address in Webhook URL and select Save. The URL must use HTTPS, must not include a username or password, and must resolve only to public internet addresses.
4
Select Send test event. QA Wolf sends a signed sample event and shows your endpoint’s response. A test event is sent once: QA Wolf doesn’t retry it, and it never turns the webhook off.
The request
QA Wolf sends each request as aPOST to your URL, with these headers:
The body is a JSON object with an
events array:
- A request carries between 1 and 100 events, up to about 1 MB in all. Make sure your endpoint, and any proxy in front of it, accepts bodies of at least 1 MB.
- Events in a request are ordered by the timestamp of their latest status, oldest first.
- Apart from test events, QA Wolf sends one request at a time for your workspace.
2xx status within 10 seconds. QA Wolf ignores the response body. Any other answer, or none, is a failed delivery, which QA Wolf retries (see Retries and turning off). To stay within the 10 seconds, store the events and answer before you process them.
QA Wolf never follows a redirect. A 3xx answer is a failed delivery.
Example:
Verify the signature
QA Wolf signs every request following the Standard Webhooks specification, so any Standard Webhooks library can verify it. Verify each request before you act on it, against the raw body exactly as it arrived. Parsing the JSON and serializing it again changes the bytes, and the signature no longer matches. The example below uses thestandardwebhooks npm package in an Express app. verify checks the signature, and that webhook-timestamp is within 5 minutes of now, then returns the parsed body. It throws when either check fails.
Example:
storeEvents with your own code that saves the events for processing. Set QAWOLF_WEBHOOK_SECRET to the signing secret, including its whsec_ prefix.
To verify without a library, join webhook-id, webhook-timestamp and the raw body with a . between each. Compute an HMAC-SHA256 of the result, keyed with the base64-decoded part of the secret after whsec_. Compare its base64 form, in constant time, with each space-separated v1, entry in webhook-signature. Reject the request when webhook-timestamp is more than 5 minutes from your current time, which stops an old request from being replayed. QA Wolf signs a request again each time it retries it after a failed delivery, so a retried request carries a recent timestamp.
Example:
Events
Every event has atype, a version and a data object. version is 1. Ignore fields you don’t recognize. Skip any event whose version or type you don’t recognize, and keep processing the rest of the request. Don’t reject the whole request because of one such event, and if you validate events against a schema, validate each event on its own. A rejected request is a failed delivery, so QA Wolf retries it, and after 8 hours the webhook turns off.
run.status-changed— a run’s status changed.flow.status-changed— the status of one flow in a run changed.
data in both event types:
Fields of
data in flow.status-changed only:
Names don’t change when you rename an environment, trigger or flow later. Match on
environmentId, triggerId and flowId instead.
Statuses
A flow’sstatuses use these values:
queued— waiting to start.running— running. QA Wolf retries a failing flow automatically within the run, and those retries add no statuses.passed— passed, or was marked as passing.failed— failed its last automatic attempt, or was marked failed because of a flow it depends on or must run with (see Run Rules).canceled— stopped before it finished.
statuses use the same values, plus:
superseded— a newer run replaced this one before it finished. Its unfinished flows endcanceled.
passed when every flow that counts passed, failed when any of them failed, and canceled when none count or the whole run was canceled. Canceled flows don’t count, and neither do flows marked failed without running because a flow they depend on was canceled.
A run whose flows had all finished before a newer run replaced it keeps the passed or failed it already sent. It doesn’t change to superseded.
These statuses report execution. The public API’s run.get reports QA Wolf’s verdict instead, which takes investigations and bugs into account, so the two can differ for the same run.
A status history can continue after a final status:
- A run’s
passedorfailedcan be followed byrunningwhen you rerun flows or mark a flow as passing, and then by a new final status. - A flow marked as passing adds
passedafter itsfailed. - A rerun flow keeps its
runIdandflowId. Thestatusesof its next event start again atqueued, with timestamps after the last status of the earlier execution.
Repeated and out-of-order events
Events of different runs and flows can arrive in any order relative to each other. The same event can also arrive more than once, for example when your endpoint answers after the 10 seconds and QA Wolf sends the request again. To handle both, remember the newest event you’ve processed for each run and each flow, and ignore any event that isn’t newer. Identify a run byrunId, and a flow by runId and flowId together. Compare two events for the same run or flow by, in order:
- The timestamp of their first status. A rerun flow’s history starts again, later than the earlier execution’s.
- The timestamp of their latest status.
- Their number of statuses, because several statuses can share a timestamp.
alertOnCall with your own code. The example records an event as processed only after acting on it, so an event whose alert fails is processed again on the next try. Keep what you’ve processed somewhere that survives a restart, such as your database, and process stored events one batch at a time.
Events carry no ID of their own, and
webhook-id can’t stand in for one. It is a SHA-256 hash of the run and flow IDs in the request and their statuses. A resend of exactly the same events carries the same webhook-id, but the same event can also arrive in a different request, with a different webhook-id.Retries and turning off
After a failed delivery, QA Wolf tries again after 5 seconds, then 5 minutes, then 30 minutes, then every hour. How QA Wolf handles each answer from your endpoint:2xx— delivered.410— QA Wolf turns the webhook off at once.413— QA Wolf halves the number of events per request and tries again. A single event your endpoint still refuses as too large is a failed delivery.429, or503with aRetry-Afterheader — a failed delivery. QA Wolf waits as long asRetry-Afterasks, between 5 seconds and 1 hour, instead of following the schedule.- Anything else — a failed delivery. This covers every other status, no answer within 10 seconds, and a connection, DNS or TLS failure. QA Wolf retries a connection, DNS or TLS failure up to twice within a few seconds, with the same signature, before it counts as a failed delivery.