Vexera - Notice history

All systems operational

Website - Operational

99% - uptime
Mar 2022 · 100.0%Apr · 100.0%May · 97.26%

Previous page

Next page

Mar 2022
Apr 2022
May 2022

Notice history

May 2022

Vexera may not respond to some commands
  • Resolved
    Resolved

    We just resolved the issue!

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Identified
    Identified

    Vexera may currently not respond to some commands. Re-sending the command again may work.

    This is due to a lapse in processing resources that occurred following a rollout. This lapse of resources is causing Vexera to backlog events continuously, causing a very high ping until it became unresponsive.

    We have made the decision to drop ~20% of received messages on purpose to partially mitigate the issue and keep Vexera sort of responsive while we work on restoring everything. As such, some commands may be ignored. Re-sending them should work.

Planned maintenance
  • Completed
    May 26th, 2022 at 10:30
    Completed
    May 26th, 2022 at 10:30

    Maintenance has completed successfully.

  • Update
    May 26th, 2022 at 09:52
    In progress
    May 26th, 2022 at 09:52

    We are continuing with the main instance now.

  • Update
    May 26th, 2022 at 08:30
    In progress
    May 26th, 2022 at 08:30

    Vexera Pro is back up. We are postponing the operation of the main instance by ~1 hour to check a few things.

  • In progress
    May 26th, 2022 at 08:00
    In progress
    May 26th, 2022 at 08:00

    Maintenance is now in progress

  • Planned
    May 26th, 2022 at 08:00
    Planned
    May 26th, 2022 at 08:00

    This maintenance is the first of two-phase maintenance planned to fix recent instability (see the announcement in our server for more information).

    Vexera will be brought down for a few minutes while we make some preparatory changes and migrate our database server to a new host.
    We expect the downtime to be 1 hour or less (until Vexera is available again in all guilds).
    Vexera Pro will also be unavailable for less than 5 minutes shortly before the start of the maintenance on the regular instance. At least one instance will therefore be available at any point in time.

    This maintenance will not fix the underlying issue by itself but will allow us to work towards the next maintenance.

Elevated errors rate
  • Resolved
    Resolved

    This issue has been resolved.

  • Monitoring
    Monitoring

    All affected shards have been restarted. Vexera is fully operational. We'll keep monitoring the situation for another few minutes until we mark the issue resolved.

  • Update
    Update

    The recovery process is taking longer than expected due to a huge amount of shards (~700 out of 1535) not having the correct guild count. We are still in the process of restarting all affected shards. Current ETA is 21:45 (UTC). Thank you for your continued patience while we are working towards a resolution to this issue.

  • Identified
    Identified

    We have located the root cause of the issue. Recovery is currently in progress, we hope to be back within the next hour. Thank you for your patience!

  • Investigating
    Investigating

    Our monitoring system fired an alert regarding elevated errors rate. We are investigating!

Multiple services unavailable
  • Resolved
    Resolved

    Vexera is operating normally.

  • Monitoring
    Monitoring

    Vexera has successfully recovered. We are currently monitoring the service for further instability.

  • Identified
    Identified

    We have identified the issue and Vexera is currently recovering. It may take some time for Vexera to be available again in your server. Vexera Pro is back up and running. We'll have a detailed summary of events ready when the issue has been resolved.

  • Investigating
    Investigating

    Vexera is currently unresponsive on all servers. Unfortunately, this issue occurred at night for the whole staff team, and there are no available developers at the moment. We hope to resume normal service during the early hours of Wednesday morning (UTC timezone).

Image manipulation and wolfram not working
  • Resolved
    Resolved

    The issue has been resolved!

  • Identified
    Identified

    We have identified an issue with our outbound proxy which impacts image manipulation commands and +wolfram.

Apr 2022

Planned maintenance
  • Completed
    April 21st, 2022 at 13:59
    Completed
    April 21st, 2022 at 13:59

    Maintenance has completed. Vexera should have recovered in most guilds since 12:30 UTC. We are currently handling an issue in a subset of guilds were Vexera is still unresponsive. These guilds should have recovered in the next 10-15 minutes.

  • Update
    April 21st, 2022 at 11:39
    In progress
    April 21st, 2022 at 11:39

    Our hosting provider maintenance has ended. Vexera Pro is working again. The main Vexera instance is currently reconnecting to your guilds. This process may take a few minutes, hang tight!

  • In progress
    April 21st, 2022 at 10:00
    In progress
    April 21st, 2022 at 10:00

    Maintenance is now in progress

  • Planned
    April 21st, 2022 at 10:00
    Planned
    April 21st, 2022 at 10:00

    Our hosting provider has planned a maintenance on April 21st at 10:00 UTC, to replace a network equipment that has issues.

    During this maintenance, both Vexera and Vexera Pro will be temporarily offline. The maintenance is planned to take approximately 2 hours.

    Link to the hosting provider status page: https://bare-metal-servers.status-ovhcloud.com/incidents/hhmw7kglyrb3

Vexera unresponsive
  • Resolved
    Resolved

    Vexera was unresponsive for about 10 minutes (18:44-18:53 UTC). This issue is due to the deployment of a bogus update.

    The update was quickly tested on our beta instance before the deployment, and no issues were found (we used only very simple tests/commands). Edit: See at the end
    The issue was therefore only identified when the update got deployed to our production instance, as alarms about high errors rate fired.

    We immediately initiated a rollback, but an issue in our orchestrator prevented it from automatically receiving the rollback instruction, delaying the recovery of Vexera.

    Please accept our apologies for this outage 😦

    Edit: After investigation, it looks like we had almost zero chances to identify the bug.
    The bug occurs only if the user sending the message doesn't have any configuration option set in our database (eg. if you never used +userlocale or such commands). We actually have user config set on our accounts on both our dev and beta instances…

  • Monitoring
    Monitoring

    We rolled back an update and currently monitoring the result.

  • Investigating
    Investigating

    Our monitoring system fired an alert regarding elevated errors rate. We are investigating!

Mar 2022

No notices reported this month

Mar 2022 to May 2022

Next