Vexera - Notice history

All systems operational

Website - Operational

100% - uptime
Feb 2022 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2022
Mar 2022
Apr 2022

Notice history

View current status

Apr 2022

APR21
Planned maintenance
CompletedMaintenance3 hours 59 minutes
  • Completed
    April 21st, 2022 at 13:59UTC
    Completed
    April 21st, 2022 at 13:59UTC

    Maintenance has completed. Vexera should have recovered in most guilds since 12:30 UTC. We are currently handling an issue in a subset of guilds were Vexera is still unresponsive. These guilds should have recovered in the next 10-15 minutes.

  • Update
    April 21st, 2022 at 11:39UTC
    Update
    April 21st, 2022 at 11:39UTC

    Our hosting provider maintenance has ended. Vexera Pro is working again. The main Vexera instance is currently reconnecting to your guilds. This process may take a few minutes, hang tight!

  • In progress
    April 21st, 2022 at 10:00UTC
    In progress
    April 21st, 2022 at 10:00UTC

    Maintenance is now in progress

  • Planned
    April 21st, 2022 at 10:00UTC
    Planned
    April 21st, 2022 at 10:00UTC

    Our hosting provider has planned a maintenance on April 21st at 10:00 UTC, to replace a network equipment that has issues.

    During this maintenance, both Vexera and Vexera Pro will be temporarily offline. The maintenance is planned to take approximately 2 hours.

    Link to the hosting provider status page: https://bare-metal-servers.status-ovhcloud.com/incidents/hhmw7kglyrb3

Vexera unresponsive
ResolvedMajor outage20 minutes
  • Resolved
    UTC
    Resolved

    Vexera was unresponsive for about 10 minutes (18:44-18:53 UTC). This issue is due to the deployment of a bogus update.

    The update was quickly tested on our beta instance before the deployment, and no issues were found (we used only very simple tests/commands). Edit: See at the end
    The issue was therefore only identified when the update got deployed to our production instance, as alarms about high errors rate fired.

    We immediately initiated a rollback, but an issue in our orchestrator prevented it from automatically receiving the rollback instruction, delaying the recovery of Vexera.

    Please accept our apologies for this outage 😦

    Edit: After investigation, it looks like we had almost zero chances to identify the bug.
    The bug occurs only if the user sending the message doesn't have any configuration option set in our database (eg. if you never used +userlocale or such commands). We actually have user config set on our accounts on both our dev and beta instances…

  • Monitoring
    UTC
    Monitoring

    We rolled back an update and currently monitoring the result.

  • Investigating
    UTC
    Investigating

    Our monitoring system fired an alert regarding elevated errors rate. We are investigating!

Mar 2022

No notices reported this month

Feb 2022

High latency
ResolvedMajor outage2 hours 22 minutes
  • Resolved
    UTC
    Resolved

    Everything is back to normal for all servers. The issue that triggered this incident is know since some time, and work is in progress to fix it.

    Sorry for the inconvenience 😦

  • Update
    UTC
    Update

    As the underlying issue got quickly out of control, we are going to do a cold restart of Vexera to recover. Vexera will slowly become available in servers again. The whole operation can take up to 1 hour.

  • Investigating
    UTC
    Investigating

    Vexera is currently handling commands with very high latency (30+ seconds). We are investigating.

Previous

Feb 2022 to Apr 2022

Next