Vexera - Notice history

All systems operational

Website - Operational

99.07% uptime
Mar 2022 · 100.0%Apr · 100.0%May · 97.26%

Previous page

Next page

Mar 2022
Apr 2022
May 2022

Notice history

View current status

May 2022

Vexera may not respond to some commands
ResolvedMinor outage1 hour 37 minutes
  • ResolvedUTC

    We just resolved the issue!

  • MonitoringUTC

    We implemented a fix and currently monitoring the result.

  • IdentifiedUTC

    Vexera may currently not respond to some commands. Re-sending the command again may work.

    This is due to a lapse in processing resources that occurred following a rollout. This lapse of resources is causing Vexera to backlog events continuously, causing a very high ping until it became unresponsive.

    We have made the decision to drop ~20% of received messages on purpose to partially mitigate the issue and keep Vexera sort of responsive while we work on restoring everything. As such, some commands may be ignored. Re-sending them should work.

MAY26
Planned maintenance
CompletedMaintenance2 hours 30 minutes
  • CompletedMay 26th, 2022 at 10:30UTC

    Maintenance has completed successfully.

  • UpdateMay 26th, 2022 at 09:52UTC

    We are continuing with the main instance now.

  • UpdateMay 26th, 2022 at 08:30UTC

    Vexera Pro is back up. We are postponing the operation of the main instance by ~1 hour to check a few things.

  • In progressMay 26th, 2022 at 08:00UTC

    Maintenance is now in progress

  • PlannedMay 26th, 2022 at 08:00UTC

    This maintenance is the first of two-phase maintenance planned to fix recent instability (see the announcement in our server for more information).

    Vexera will be brought down for a few minutes while we make some preparatory changes and migrate our database server to a new host.
    We expect the downtime to be 1 hour or less (until Vexera is available again in all guilds).
    Vexera Pro will also be unavailable for less than 5 minutes shortly before the start of the maintenance on the regular instance. At least one instance will therefore be available at any point in time.

    This maintenance will not fix the underlying issue by itself but will allow us to work towards the next maintenance.

Elevated errors rate
ResolvedMinor outage5 hours 58 minutes
  • ResolvedUTC

    This issue has been resolved.

  • MonitoringUTC

    All affected shards have been restarted. Vexera is fully operational. We'll keep monitoring the situation for another few minutes until we mark the issue resolved.

  • UpdateUTC

    The recovery process is taking longer than expected due to a huge amount of shards (~700 out of 1535) not having the correct guild count. We are still in the process of restarting all affected shards. Current ETA is 21:45 (UTC). Thank you for your continued patience while we are working towards a resolution to this issue.

  • IdentifiedUTC

    We have located the root cause of the issue. Recovery is currently in progress, we hope to be back within the next hour. Thank you for your patience!

  • InvestigatingUTC

    Our monitoring system fired an alert regarding elevated errors rate. We are investigating!

Multiple services unavailable
ResolvedMajor outage8 hours 51 minutes
  • ResolvedUTC

    Vexera is operating normally.

  • MonitoringUTC

    Vexera has successfully recovered. We are currently monitoring the service for further instability.

  • IdentifiedUTC

    We have identified the issue and Vexera is currently recovering. It may take some time for Vexera to be available again in your server. Vexera Pro is back up and running. We'll have a detailed summary of events ready when the issue has been resolved.

  • InvestigatingUTC

    Vexera is currently unresponsive on all servers. Unfortunately, this issue occurred at night for the whole staff team, and there are no available developers at the moment. We hope to resume normal service during the early hours of Wednesday morning (UTC timezone).

Image manipulation and wolfram not working
ResolvedPartial outage1 hour 15 minutes
  • ResolvedUTC

    The issue has been resolved!

  • IdentifiedUTC

    We have identified an issue with our outbound proxy which impacts image manipulation commands and +wolfram.

Apr 2022

APR21
Planned maintenance
CompletedMaintenance3 hours 59 minutes
  • CompletedApril 21st, 2022 at 13:59UTC

    Maintenance has completed. Vexera should have recovered in most guilds since 12:30 UTC. We are currently handling an issue in a subset of guilds were Vexera is still unresponsive. These guilds should have recovered in the next 10-15 minutes.

  • UpdateApril 21st, 2022 at 11:39UTC

    Our hosting provider maintenance has ended. Vexera Pro is working again. The main Vexera instance is currently reconnecting to your guilds. This process may take a few minutes, hang tight!

  • In progressApril 21st, 2022 at 10:00UTC

    Maintenance is now in progress

  • PlannedApril 21st, 2022 at 10:00UTC

    Our hosting provider has planned a maintenance on April 21st at 10:00 UTC, to replace a network equipment that has issues.

    During this maintenance, both Vexera and Vexera Pro will be temporarily offline. The maintenance is planned to take approximately 2 hours.

    Link to the hosting provider status page: https://bare-metal-servers.status-ovhcloud.com/incidents/hhmw7kglyrb3

Vexera unresponsive
ResolvedMajor outage20 minutes
  • ResolvedUTC

    Vexera was unresponsive for about 10 minutes (18:44-18:53 UTC). This issue is due to the deployment of a bogus update.

    The update was quickly tested on our beta instance before the deployment, and no issues were found (we used only very simple tests/commands). Edit: See at the end
    The issue was therefore only identified when the update got deployed to our production instance, as alarms about high errors rate fired.

    We immediately initiated a rollback, but an issue in our orchestrator prevented it from automatically receiving the rollback instruction, delaying the recovery of Vexera.

    Please accept our apologies for this outage 😦

    Edit: After investigation, it looks like we had almost zero chances to identify the bug.
    The bug occurs only if the user sending the message doesn't have any configuration option set in our database (eg. if you never used +userlocale or such commands). We actually have user config set on our accounts on both our dev and beta instances…

  • MonitoringUTC

    We rolled back an update and currently monitoring the result.

  • InvestigatingUTC

    Our monitoring system fired an alert regarding elevated errors rate. We are investigating!

Mar 2022

No notices reported this month

Previous

Mar 2022 to May 2022

Next