Ipregistry - INVALID_API_KEY error for API calls in some regions – Incident details

INVALID_API_KEY error for API calls in some regions

Resolved
Partial outage 30 %
Started about 22 hours agoLasted 44 minutes

Affected

API

Partial outage from 9:05 PM to 9:49 PM

Globally distributed nodes — https://api.ipregistry.co

Partial outage from 9:05 PM to 9:49 PM

Europe nodes — https://eu.api.ipregistry.co

Partial outage from 9:05 PM to 9:49 PM

Updates
  • Postmortem
    Postmortem

    A surge in load for a replicated database cluster saturated available connections. A recent backend library change introduced an overly aggressive connection timeout, which led to cascading timeouts in a subset of regions, while others were unaffected.

    We sincerely apologize for the disruption caused. We adjusted connection settings, and put additional safeguards in place to reduce the risk of similar issues. We are also improving testing and monitoring around connection handling to better detect and mitigate such situations in the future.

  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Identified
    Identified
    We are continuing to work on a fix for this incident.
  • Investigating
    Investigating
    We are currently investigating this incident.