Payout settlement failing on retry

Investigating

Opened by the settlement monitor.

Summary

Four consecutive runs of settle-payouts have returned a gateway timeout after roughly four minutes. The retry budget for the schedule is spent, so no further attempt will start on its own. Settlement for 1,180 accounts is queued behind it.

What happened

  1. Retry budget exhausted

    Run 4812 failed after 4m 11s. No automatic attempt remains.

  2. Assigned to Payments

    Routed by the on-call rota after the second failure.

  3. Gateway latency alert cleared

  4. First failure recorded

    Run 4798 returned a gateway timeout at the settlement step.