The 3rd party has restored power successfully. All public services are back online. Thanks for your patience!
Posted May 22, 2024 - 11:00 EDT
Update
Apologies for the lack of update, as noted previously no staff is on site currently and in fact, we are in a whole different time zone :(
We believe the electric utility has restored power, but our UPS most likely requires being manually powered on. We are engaging a 3rd party to attempt this.
Posted May 22, 2024 - 10:38 EDT
Update
Due to the extended outage condition, our UPS has fully depleted and our router on site is no longer announcing the on premise prefix. Unreachable errors for the on premise IP space is expected.
Posted May 22, 2024 - 01:35 EDT
Update
The local utility service has acknowledged the outage but there is no current estimate for restoration. The current status from the utility is “Assessing Condition”.
Posted May 22, 2024 - 01:29 EDT
Identified
A power outage occurred at the on premise site, which resulted in our UPS batteries being depleted. We are awaiting further info regarding the outages. Unfortunately, nobody on our team is on site, so it may take time to restore service.
GitLab and metrics will be unavailable during this time. Alerts based on Prometheus metrics will not fire. Our tertiary MX will be unavailable, but there is no impact on Petalpost as the higher priority MXs are functional.
We will provide updates as we have them.
Posted May 22, 2024 - 01:08 EDT
This incident affected: Hosted Services (GitLab, Grafana/Prometheus Monitoring) and Hosts/PoPs (Eveland - Elkhart, IN, GitLab - Elkhart, IN, Kaneshiro - Elkhart, IN, Penrose - Elkhart, IN).