Vexera - Notice history

All systems operational

Website - Operational

99% - uptime
May 2022 · 97.26%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2022
Jun 2022
Jul 2022

Notice history

Jul 2022

Bot down
  • Resolved
    UTC
    Resolved

    Vexera is now back online in all servers.

  • Identified
    UTC
    Identified

    The underlying issue has been found, and the bot is currently restarting. Please allow up to 20 minutes for full recovery.

  • Investigating
    UTC
    Investigating

    We are currently investigating this incident.

Jun 2022

No notices reported this month

May 2022

Vexera may not respond to some commands
  • Resolved
    UTC
    Resolved

    We just resolved the issue!

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Identified
    UTC
    Identified

    Vexera may currently not respond to some commands. Re-sending the command again may work.

    This is due to a lapse in processing resources that occurred following a rollout. This lapse of resources is causing Vexera to backlog events continuously, causing a very high ping until it became unresponsive.

    We have made the decision to drop ~20% of received messages on purpose to partially mitigate the issue and keep Vexera sort of responsive while we work on restoring everything. As such, some commands may be ignored. Re-sending them should work.

Planned maintenance
  • Completed
    May 26th, 2022 at 10:30UTC
    Completed
    May 26th, 2022 at 10:30UTC

    Maintenance has completed successfully.

  • Update
    May 26th, 2022 at 09:52UTC
    Update
    May 26th, 2022 at 09:52UTC

    We are continuing with the main instance now.

  • Update
    May 26th, 2022 at 08:30UTC
    Update
    May 26th, 2022 at 08:30UTC

    Vexera Pro is back up. We are postponing the operation of the main instance by ~1 hour to check a few things.

  • In progress
    May 26th, 2022 at 08:00UTC
    In progress
    May 26th, 2022 at 08:00UTC

    Maintenance is now in progress

  • Planned
    May 26th, 2022 at 08:00UTC
    Planned
    May 26th, 2022 at 08:00UTC

    This maintenance is the first of two-phase maintenance planned to fix recent instability (see the announcement in our server for more information).

    Vexera will be brought down for a few minutes while we make some preparatory changes and migrate our database server to a new host.
    We expect the downtime to be 1 hour or less (until Vexera is available again in all guilds).
    Vexera Pro will also be unavailable for less than 5 minutes shortly before the start of the maintenance on the regular instance. At least one instance will therefore be available at any point in time.

    This maintenance will not fix the underlying issue by itself but will allow us to work towards the next maintenance.

Elevated errors rate
  • Resolved
    UTC
    Resolved

    This issue has been resolved.

  • Monitoring
    UTC
    Monitoring

    All affected shards have been restarted. Vexera is fully operational. We'll keep monitoring the situation for another few minutes until we mark the issue resolved.

  • Update
    UTC
    Update

    The recovery process is taking longer than expected due to a huge amount of shards (~700 out of 1535) not having the correct guild count. We are still in the process of restarting all affected shards. Current ETA is 21:45 (UTC). Thank you for your continued patience while we are working towards a resolution to this issue.

  • Identified
    UTC
    Identified

    We have located the root cause of the issue. Recovery is currently in progress, we hope to be back within the next hour. Thank you for your patience!

  • Investigating
    UTC
    Investigating

    Our monitoring system fired an alert regarding elevated errors rate. We are investigating!

Multiple services unavailable
  • Resolved
    UTC
    Resolved

    Vexera is operating normally.

  • Monitoring
    UTC
    Monitoring

    Vexera has successfully recovered. We are currently monitoring the service for further instability.

  • Identified
    UTC
    Identified

    We have identified the issue and Vexera is currently recovering. It may take some time for Vexera to be available again in your server. Vexera Pro is back up and running. We'll have a detailed summary of events ready when the issue has been resolved.

  • Investigating
    UTC
    Investigating

    Vexera is currently unresponsive on all servers. Unfortunately, this issue occurred at night for the whole staff team, and there are no available developers at the moment. We hope to resume normal service during the early hours of Wednesday morning (UTC timezone).

Image manipulation and wolfram not working
  • Resolved
    UTC
    Resolved

    The issue has been resolved!

  • Identified
    UTC
    Identified

    We have identified an issue with our outbound proxy which impacts image manipulation commands and +wolfram.

May 2022 to Jul 2022

Next