Duration: 1 hour and 48 minutes

Affected Components: Commands processing, Scheduled events

Jun 3, 15:58:00 GMT+0
Identified - We have identified an issue that makes Vexera unresponsive in a subset of servers. We are working on restoring service in the affected servers..

Jun 3, 16:55:56 GMT+0
Identified - About 50% of the impacted servers have recovered at this point..

Jun 3, 17:46:25 GMT+0
Resolved - This incident has been resolved..

Duration: 1 day, 1 hour and 34 minutes

Affected Components: Commands processing, Scheduled events

Apr 10, 20:15:00 GMT+0
Investigating - Our monitoring system fired an alert regarding elevated errors rate. We are investigating!.

Apr 11, 20:09:51 GMT+0
Identified - This issue is related to yesterday's Discord incident: <https://discordstatus.com/incidents/flzv6f6p7l4k> The bot is currently recovering. Full recovery is expected in an hour..

Apr 11, 21:00:41 GMT+0
Identified - Vexera has recovered for about 75% of the servers. We expect the last servers to recover in the next 20 minutes..

Apr 11, 21:27:13 GMT+0
Monitoring - Vexera should be available again in all servers. While some backend caches finish to fill up, you may experience slowness on some commands. We are monitoring for possible remnant errors. Scheduled events missed during the outage will be replayed in the next few minutes..

Apr 11, 21:49:27 GMT+0
Resolved - This incident has been resolved..

Duration: 2 days and 12 hours

Affected Components: Commands processing, Scheduled events, Voice orchestration, Image manipulation

Sep 15, 23:00:00 GMT+0
Investigating - We are aware of an issue affecting Vexera's availability and are currently investigating..

Sep 17, 15:18:44 GMT+0
Identified - The issue has been identified and we are working on restoring service to the affected components..

Sep 17, 18:24:50 GMT+0
Monitoring - Recovery of the affected components has been successful. We are currently monitoring for further issues and unavailability..

Sep 18, 10:32:00 GMT+0
Resolved - This incident has been resolved..

Duration: 15 hours and 40 minutes

Affected Components: Voice orchestration

Apr 14, 23:00:00 GMT+0
Investigating - We're aware of an issue causing music playback to fail mostly due to the error `Internal Server Error` (another error may also occur). Investigation is currently pending..

Apr 15, 14:20:00 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Apr 15, 14:40:13 GMT+0
Resolved - This incident has been resolved..

Duration: 16 hours and 45 minutes

Affected Components: Commands processing

Feb 11, 17:42:13 GMT+0
Investigating - Vexera is currently unresponsive. We are investigating this incident..

Feb 11, 17:55:45 GMT+0
Identified - We have found the cause of the incident. Vexera is now cold-booting and should be available again in all servers in the next 30 minutes..

Feb 11, 18:36:28 GMT+0
Identified - We are having issues related to the recent growth of Vexera, and the bot is still starting up..

Feb 11, 22:15:24 GMT+0
Monitoring - Vexera should now be available and working again on all servers. Please note that you may experience some weirdness in the next few hours until we come up with a definitive fix..

Feb 12, 10:27:34 GMT+0
Resolved - Everything is now operating again in a nominal state..

Feb 11, 19:20:18 GMT+0
Identified - Around 75% of servers have already recovered. Recovery is slower for the last servers..

Duration: 6 hours and 52 minutes

Affected Components: Commands processing

Jan 21, 02:05:00 GMT+0
Investigating - Our monitoring system fired an alert regarding elevated errors rate. We are investigating!.

Jan 21, 08:37:20 GMT+0
Identified - Hi everyone, we are sorry for this incident, which sadly happened during the night for us. Vexera managed to self-heal from the incident and recovered for most servers within the next hour. We are now manually fixing the remaining servers where Vexera is still unresponsive..

Jan 21, 08:57:29 GMT+0
Resolved - Vexera should be working again in all servers!.

Duration: 55 minutes

Affected Components: Commands processing

Jan 10, 20:04:48 GMT+0
Investigating - We are aware of an issue with Vexera not responding in some servers. We are investigating..

Jan 10, 20:22:49 GMT+0
Identified - We are in the process of restarting Vexera. It should come back online on all servers in the next 20 minutes..

Jan 10, 20:59:55 GMT+0
Resolved - Vexera is online again in all servers, and everything looks back to normal so far..

Duration: 3 hours and 4 minutes

Affected Components: Scheduled events, Website, Commands processing, Image manipulation, Voice orchestration

Dec 19, 17:03:00 GMT+0
Investigating - Vexera is currently experiencing a major outage, leading to complete downtime in all servers. We are aware and are investigating this issue. As a side effect, the dashboard on our website is also unavailable at the time. You will see a 403 error until the aforementioned issue has been fixed. Vexera Pro is unaffected by this..

Dec 19, 18:11:00 GMT+0
Identified - Vexera is currently recovering. We are in the process of starting back up all affected shards, this may take a while. Another incident update will be ready when Vexera has fully recovered..

Dec 19, 18:57:23 GMT+0
Monitoring - All services have recovered from the outage. We are currently monitoring the situation until we're sure this incident is resolved..

Dec 19, 20:06:49 GMT+0
Resolved - After monitoring the situation, we're certain that Vexera is stable and fully back up. Thank you for your patience!.

Duration: 2 hours and 21 minutes

Affected Components: Scheduled events, Commands processing

Dec 15, 19:47:52 GMT+0
Investigating - Our monitoring system fired an alert regarding elevated errors rate. We are investigating!.

Dec 15, 20:12:24 GMT+0
Identified - Vexera has been down for 25 minutes due to a database issue. It's currently being restarted, please allow up to 30 minutes to reconnect in all servers. *Vexera Pro is back online.*.

Dec 15, 20:41:47 GMT+0
Monitoring - Vexera should be online again in all servers. A few things (members cache...) are still loading in the background, but we expect that everything is working again as expected..

Dec 15, 22:08:48 GMT+0
Resolved - Everything works as expected. The incident is now closed..

Duration: 54 minutes

Dec 4, 15:30:00 GMT+0
Identified - An incident currently impacts a hosting provider hosting some of our voice nodes (servers handling voice encoding and playback). The load was transferred to the remaining nodes. As such, some side effects like lags may be observed until this issue is fixed..

Dec 4, 15:55:00 GMT+0
Monitoring - Impacted voice nodes are back in production. We are monitoring the situation while the load rebalances itself between nodes..

Dec 4, 16:24:27 GMT+0
Resolved - Everything is back to normal!.

Duration: 16 minutes

Affected Components: Voice orchestration

Nov 18, 07:00:00 GMT+0
Identified - We will be deploying an update to some internal services of Vexera handling voice connections and playbacks. The update will cause a temporary disconnection of Vexera from all voice channels. The duration of the unavailability should be around 5 minutes..

Nov 18, 07:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 18, 07:15:30 GMT+0
Completed - Maintenance has completed successfully..

Duration: 7 hours and 15 minutes

Affected Components: Website

Aug 31, 02:00:00 GMT+0
Resolved - The issue has been resolved..

Aug 30, 18:45:23 GMT+0
Investigating - The website is currently returning "Server Error" messages to some users. We are currently investigating. Clearing cookies can fix the issue for some users..

Duration: 1 hour and 17 minutes

Affected Components: Commands processing, Scheduled events

Aug 18, 19:52:15 GMT+0
Investigating - Vexera is currently offline in most servers. We are investigating!.

Aug 18, 20:21:06 GMT+0
Identified - Our hosting provider had a short outage with their internal networking, which brought down Vexera. The issue has been resolved, and Vexera is booting back again. Please give it ~20 minutes..

Aug 18, 20:52:05 GMT+0
Monitoring - At this point, Vexera should be working again on all servers. We are monitoring the situation before closing this incident..

Aug 18, 21:08:50 GMT+0
Resolved - The issue has been resolved..

Duration: 52 minutes

Affected Components: Commands processing

Jul 21, 19:30:00 GMT+0
Investigating - We are currently investigating this incident..

Jul 21, 19:52:46 GMT+0
Identified - The underlying issue has been found, and the bot is currently restarting. Please allow up to 20 minutes for full recovery..

Jul 21, 20:21:30 GMT+0
Resolved - Vexera is now back online in all servers..

Duration: 1 hour and 38 minutes

Affected Components: Commands processing

May 29, 14:24:13 GMT+0
Identified - Vexera may currently not respond to some commands. Re-sending the command again may work. This is due to a lapse in processing resources that occurred following a rollout. This lapse of resources is causing Vexera to backlog events continuously, causing a very high ping until it became unresponsive. We have made the decision to drop ~20% of received messages on purpose to partially mitigate the issue and keep Vexera sort of responsive while we work on restoring everything. As such, some commands may be ignored. Re-sending them should work..

May 29, 15:52:47 GMT+0
Monitoring - We implemented a fix and currently monitoring the result..

May 29, 16:02:09 GMT+0
Resolved - We just resolved the issue!.

Duration: 2 hours and 30 minutes

Affected Components: Image manipulation, Commands processing, Scheduled events, Voice orchestration

May 26, 09:52:54 GMT+0
Identified - We are continuing with the main instance now..

May 26, 10:30:00 GMT+0
Completed - Maintenance has completed successfully..

May 26, 08:00:00 GMT+0
Identified - This maintenance is the first of two-phase maintenance planned to fix recent instability (see the announcement in [our server](https://discord.gg/vexera) for more information). Vexera will be brought down for a few minutes while we make some preparatory changes and migrate our database server to a new host. **We expect the downtime to be 1 hour or less (until Vexera is available again in all guilds).** *Vexera Pro will also be unavailable for less than 5 minutes shortly before the start of the maintenance on the regular instance. At least one instance will therefore be available at any point in time.* This maintenance will not fix the underlying issue by itself but will allow us to work towards the next maintenance..

May 26, 08:30:00 GMT+0
Identified - Vexera Pro is back up. We are postponing the operation of the main instance by ~1 hour to check a few things..

May 26, 08:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 5 hours and 59 minutes

Affected Components: Commands processing

May 19, 17:44:49 GMT+0
Identified - We have located the root cause of the issue. Recovery is currently in progress, we hope to be back within the next hour. Thank you for your patience!.

May 19, 19:04:52 GMT+0
Identified - The recovery process is taking longer than expected due to a huge amount of shards (~700 out of 1535) not having the correct guild count. We are still in the process of restarting all affected shards. Current ETA is 21:45 (UTC). Thank you for your continued patience while we are working towards a resolution to this issue..

May 19, 22:58:39 GMT+0
Resolved - This issue has been resolved..

May 19, 17:00:00 GMT+0
Investigating - Our monitoring system fired an alert regarding elevated errors rate. We are investigating!.

May 19, 22:02:05 GMT+0
Monitoring - All affected shards have been restarted. Vexera is fully operational. We'll keep monitoring the situation for another few minutes until we mark the issue resolved..

Duration: 22 days and 3 hours

Affected Components: Commands processing, Image manipulation, Website, Scheduled events, Voice orchestration

May 18, 09:15:00 GMT+0
Resolved - Vexera is operating normally..

May 18, 07:37:11 GMT+0
Identified - We have identified the issue and Vexera is currently recovering. It may take some time for Vexera to be available again in your server. Vexera Pro is back up and running. We'll have a detailed summary of events ready when the issue has been resolved..

May 18, 08:45:02 GMT+0
Monitoring - Vexera has successfully recovered. We are currently monitoring the service for further instability..

May 18, 00:24:00 GMT+0
Investigating - Vexera is currently unresponsive on all servers. Unfortunately, this issue occurred at night for the whole staff team, and there are no available developers at the moment. We hope to resume normal service during the early hours of Wednesday morning (UTC timezone)..

Duration: 1 hour and 15 minutes

Affected Components: Commands processing, Image manipulation

May 9, 12:50:00 GMT+0
Resolved - The issue has been resolved!.

May 9, 11:34:57 GMT+0
Identified - We have identified an issue with our outbound proxy which impacts image manipulation commands and `+wolfram`..

Duration: 28 days and 18 hours

Affected Components: Website, Voice orchestration, Commands processing

May 3, 17:17:47 GMT+0
Identified - We have identified the cause of the issue and are working on fixing the impacted voice service, as well as the dashboard. Vexera Pro has been fixed and is back up & running! ----------------------- **Edit:** We are currently restarting all affected shards..

May 3, 08:00:00 GMT+0
Investigating - Vexera is currently affected by 3 various issues: 1. One of our voice components, specifically "voice-manager-1", is currently unresponsive. Some servers may experience issues playing music. 2. The dashboard returns a "403 Forbidden" error once you select a server in your server list. Configuring a server through the dashboard is not possible at the time. 3. Vexera Pro is unresponsive to all commands. We're investigating the issues above and will update once further information becomes available. Thank you for your patience..

May 3, 19:30:00 GMT+0
Resolved - There may have been some lags/non-responses while we were disabling some workaround code we used to assist in recovery. But the issue is now considered resolved..

May 3, 18:40:00 GMT+0
Monitoring - As of now, all guilds should have recovered. We are keeping health metrics under close monitoring while our cache is finishing to warm up. For transparency, previous updates contained an error: the outage is not due to an issue with our voice service (although early investigations have reported this service as a possible cause). There will be more information to come when we will be sure that everything is fixed..

May 3, 20:47:00 GMT+0
Postmortem - The root cause is a network micro-cut that happened around 07:00 AM UTC. This caused a large number of shards to disconnect and reconnect at the same time. This caused some issues in our cache: 240 shards (out of a total of 704) had incorrect data cached. This cache contains critical data for the bot operations (like in which guilds Vexera is present, what are the channels in these guilds, the permissions, the members, etc.). Missing and invalid data prevented the bot to work correctly in some guilds. The issue, that first got identified by the support team based on user reports, wasn't correctly escalated to devs for remediation. All the people who run the bot are based in Europe, and at this hour, we were already on our worksites. This finally got escalated to someone with access to our servers at 02:17 PM UTC who tried basic actions to recover the bot without success. Being at work, they weren't able to do much more. Real investigations finally started around 05:00 PM UTC, and the first remediation steps were able to be executed by 05:10 PM UTC. After having identified the 240 shards affected by the issue, a rolling restart of them was started to fix the issue. This action was completed by 06:40 PM UTC, at which point almost all guilds were fixed. We then dug into our logs to search for and fix a few remaining guilds in an inconsistent state. At 07:30 PM UTC, we believed everything was back to normal, which was confirmed by our metrics. --- The time it took to fix this issue is unacceptable, and there is clearly a lack of proper escalation and handling processes for major outages that happen during workdays. We will see how we can improve that..

Duration: 3 hours and 59 minutes

Affected Components: Website, Commands processing, Scheduled events, Image manipulation, Voice orchestration

Apr 21, 10:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 21, 11:39:41 GMT+0
Identified - Our hosting provider maintenance has ended. Vexera Pro is working again. The main Vexera instance is currently reconnecting to your guilds. This process may take a few minutes, hang tight!.

Apr 21, 13:59:21 GMT+0
Completed - Maintenance has completed. Vexera should have recovered in most guilds since 12:30 UTC. We are currently handling an issue in a subset of guilds were Vexera is still unresponsive. These guilds should have recovered in the next 10-15 minutes..

Apr 21, 10:00:00 GMT+0
Identified - Our hosting provider has planned a maintenance on April 21st at 10:00 UTC, to replace a network equipment that has issues. During this maintenance, both Vexera and Vexera Pro will be temporarily offline. The maintenance is planned to take approximately 2 hours. Link to the hosting provider status page: [https://bare-metal-servers.status-ovhcloud.com/incidents/hhmw7kglyrb3](https://bare-metal-servers.status-ovhcloud.com/incidents/hhmw7kglyrb3).

Duration: 20 hours

Affected Components: Commands processing

Apr 6, 18:45:00 GMT+0
Investigating - Our monitoring system fired an alert regarding elevated errors rate. We are investigating!.

Apr 6, 19:05:00 GMT+0
Resolved - Vexera was unresponsive for about 10 minutes (18:44-18:53 UTC). This issue is due to the deployment of a bogus update. The update was quickly tested on our beta instance before the deployment, and no issues were found (we used only very simple tests/commands). **Edit: See at the end** The issue was therefore only identified when the update got deployed to our production instance, as alarms about high errors rate fired. We immediately initiated a rollback, but an issue in our orchestrator prevented it from automatically receiving the rollback instruction, delaying the recovery of Vexera. Please accept our apologies for this outage 😦 ----------------------- **Edit:** After investigation, it looks like we had almost zero chances to identify the bug. The bug occurs only if the user sending the message doesn't have any configuration option set in our database (eg. if you never used `+userlocale` or such commands). We actually have user config set on our accounts on both our dev and beta instances….

Apr 6, 18:53:00 GMT+0
Monitoring - We rolled back an update and currently monitoring the result..

Duration: 5 days and 22 hours

Affected Components: Commands processing, Scheduled events

Feb 13, 17:30:00 GMT+0
Investigating - Vexera is currently handling commands with very high latency (30+ seconds). We are investigating..

Feb 13, 17:43:00 GMT+0
Investigating - As the underlying issue got quickly out of control, we are going to do a cold restart of Vexera to recover. Vexera will slowly become available in servers again. The whole operation can take up to 1 hour. .

Feb 13, 19:52:00 GMT+0
Resolved - Everything is back to normal for all servers. The issue that triggered this incident is know since some time, and work is in progress to fix it. Sorry for the inconvenience 😦.

Duration: 3 hours and 43 minutes

Affected Components: Commands processing

Jan 19, 18:31:00 GMT+0
Resolved - Vexera is back up!.

Jan 19, 14:48:00 GMT+0
Investigating - Vexera is currently unresponsive and shows as offline in all servers. Both **Vexera** and **Vexera Pro** are affected by this. We're looking into the issue and will send out updates as they become available. Thank you for your patience!.

Jan 19, 16:29:00 GMT+0
Identified - Vexera is starting back up. Vexera Pro is back up & has been for around 15 minutes. We will provide update when Vexera is completely back up..

Duration: 3 hours

Affected Components: Commands processing, Voice orchestration, Scheduled events

Jan 15, 10:00:00 GMT+0
Identified - Due to a database issue, Vexera is currently unresponsive. We're resolving this now and Vexera will be back up soon..

Jan 15, 11:48:00 GMT+0
Identified - Vexera is restarting, it will be online again in your guild within 15-30 minutes..

Jan 15, 13:00:00 GMT+0
Resolved - Vexera is back up everywhere 🙂.

Duration: 2 hours and 57 minutes

Affected Components: Commands processing, Voice orchestration

Jan 11, 16:10:00 GMT+0
Investigating - We are currently investigating a high bot latency..

Jan 11, 19:07:00 GMT+0
Resolved - During the last 3 hours, Vexera experienced again the same issue than 3 days ago. The probable cause of these issues is a contexts leak in an internal service that causes slowness after a few days of operations and ultimately a backlogging of Discord events when encountering our daily peak hours around 17:00 UTC. We will implement scheduled restarts of the affected service to mitigate the issue until we can identify and fix the cause of these leaks. Sorry for the inconvenience.

Duration: 2 days and 5 hours

Affected Components: Commands processing, Voice orchestration

Jan 8, 16:15:00 GMT+0
Investigating - Vexera may respond with a delay. We are investigating the issue..

Jan 8, 17:08:00 GMT+0
Resolved - .

Jan 8, 17:07:00 GMT+0
Monitoring - Vexera experienced issues during the last hour which have lead to delayed responses and `Internal Server Error` messages (mainly with music related commands). In an attempt to remediate the issue, we have restarted the voice manager (our internal service controlling the voice encoders). This may have disconnected Vexera from voice channels. We believe the issue is now fixed, and our error rate is back to normal levels since 15 minutes. However, we are still monitoring the situation, because the root cause of the issue is still unknown..

Duration: 46 minutes

Affected Components: Commands processing

Jan 5, 11:56:00 GMT+0
Monitoring - Vexera should have recovered in all servers. We are monitoring our error rate..

Jan 5, 12:06:00 GMT+0
Resolved - The issue is now resolved..

Jan 5, 11:20:00 GMT+0
Identified - We are aware that Vexera is unresponsive on some servers. We are currently taking remediation steps to resolve this issue..

Duration: 2 hours

Affected Components: Website, Commands processing, Voice orchestration, Scheduled events

Dec 28, 13:00:00 GMT+0
Completed - Maintenance has completed successfully..

Dec 28, 11:00:00 GMT+0
Identified - Our hosting provider has planned a maintenance starting today at 11:00 UTC, to reboot a network equipment that have issues. During this maintenance, Vexera may go temporarily offline. If that happens, this shouldn't take longer than 1 hour..

Dec 28, 12:42:00 GMT+0
Identified - We took this opportunity to upgrade the infrastructure behind Vexera. We have completed the infrastructure upgrades, and our hosting provider has completed their maintenance. We will be starting Vexera back up in the next 15 minutes or so. Vexera will be back online in your server in under an hour..