Vexera Status - Incident history https://status.vexera.io Vexera Tue, 3 Jun 2025 15:58:00 +0000 Vexera unresponsive in a subset of servers Type: Incident Duration: 1 hour and 48 minutes Affected Components: Commands processing, Scheduled events Jun 3, 15:58:00 GMT+0 - Identified - We have identified an issue that makes Vexera unresponsive in a subset of servers. We are working on restoring service in the affected servers. Jun 3, 16:55:56 GMT+0 - Identified - About 50% of the impacted servers have recovered at this point. Jun 3, 17:46:25 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 1 hour and 48 minutes

Affected Components: ,

<p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>15:58:00</var> GMT+0</small><br><strong>Identified</strong> - We have identified an issue that makes Vexera unresponsive in a subset of servers. We are working on restoring service in the affected servers..</p> <p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>16:55:56</var> GMT+0</small><br><strong>Identified</strong> - About 50% of the impacted servers have recovered at this point..</p> <p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>17:46:25</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Tue, 3 Jun 2025 15:58:00 +0000 https://status.vexera.io/incident/cmbgq7z7x000yf9lv1xaz20xj https://status.vexera.io/incident/cmbgq7z7x000yf9lv1xaz20xj Bot is down Type: Incident Duration: 1 day, 1 hour and 34 minutes Affected Components: Commands processing, Scheduled events Apr 10, 20:15:00 GMT+0 - Investigating - Our monitoring system fired an alert regarding elevated errors rate. We are investigating! Apr 11, 20:09:51 GMT+0 - Identified - This issue is related to yesterday's Discord incident: <https://discordstatus.com/incidents/flzv6f6p7l4k> The bot is currently recovering. Full recovery is expected in an hour. Apr 11, 21:00:41 GMT+0 - Identified - Vexera has recovered for about 75% of the servers. We expect the last servers to recover in the next 20 minutes. Apr 11, 21:27:13 GMT+0 - Monitoring - Vexera should be available again in all servers. While some backend caches finish to fill up, you may experience slowness on some commands. We are monitoring for possible remnant errors. Scheduled events missed during the outage will be replayed in the next few minutes. Apr 11, 21:49:27 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 1 day, 1 hour and 34 minutes

Affected Components: ,

<p><small>Apr <var data-var='date'> 10</var>, <var data-var='time'>20:15:00</var> GMT+0</small><br><strong>Investigating</strong> - Our monitoring system fired an alert regarding elevated errors rate. We are investigating!.</p> <p><small>Apr <var data-var='date'> 11</var>, <var data-var='time'>20:09:51</var> GMT+0</small><br><strong>Identified</strong> - This issue is related to yesterday's Discord incident: <https://discordstatus.com/incidents/flzv6f6p7l4k> The bot is currently recovering. Full recovery is expected in an hour..</p> <p><small>Apr <var data-var='date'> 11</var>, <var data-var='time'>21:00:41</var> GMT+0</small><br><strong>Identified</strong> - Vexera has recovered for about 75% of the servers. We expect the last servers to recover in the next 20 minutes..</p> <p><small>Apr <var data-var='date'> 11</var>, <var data-var='time'>21:27:13</var> GMT+0</small><br><strong>Monitoring</strong> - Vexera should be available again in all servers. While some backend caches finish to fill up, you may experience slowness on some commands. We are monitoring for possible remnant errors. Scheduled events missed during the outage will be replayed in the next few minutes..</p> <p><small>Apr <var data-var='date'> 11</var>, <var data-var='time'>21:49:27</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Thu, 10 Apr 2025 20:15:00 +0000 https://status.vexera.io/incident/cm9d7zbgg007jo33iqamgfkwe https://status.vexera.io/incident/cm9d7zbgg007jo33iqamgfkwe Impacted Availability Type: Incident Duration: 2 days and 12 hours Affected Components: Commands processing, Scheduled events, Voice orchestration, Image manipulation Sep 15, 23:00:00 GMT+0 - Investigating - We are aware of an issue affecting Vexera's availability and are currently investigating. Sep 17, 15:18:44 GMT+0 - Identified - The issue has been identified and we are working on restoring service to the affected components. Sep 17, 18:24:50 GMT+0 - Monitoring - Recovery of the affected components has been successful. We are currently monitoring for further issues and unavailability. Sep 18, 10:32:00 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 2 days and 12 hours

Affected Components: , , ,

<p><small>Sep <var data-var='date'> 15</var>, <var data-var='time'>23:00:00</var> GMT+0</small><br><strong>Investigating</strong> - We are aware of an issue affecting Vexera's availability and are currently investigating..</p> <p><small>Sep <var data-var='date'> 17</var>, <var data-var='time'>15:18:44</var> GMT+0</small><br><strong>Identified</strong> - The issue has been identified and we are working on restoring service to the affected components..</p> <p><small>Sep <var data-var='date'> 17</var>, <var data-var='time'>18:24:50</var> GMT+0</small><br><strong>Monitoring</strong> - Recovery of the affected components has been successful. We are currently monitoring for further issues and unavailability..</p> <p><small>Sep <var data-var='date'> 18</var>, <var data-var='time'>10:32:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Sun, 15 Sep 2024 23:00:00 +0000 https://status.vexera.io/incident/cm16kihxs0001oo26iwuw0a4u https://status.vexera.io/incident/cm16kihxs0001oo26iwuw0a4u Elevated Music Errors Rate Type: Incident Duration: 15 hours and 40 minutes Affected Components: Voice orchestration Apr 14, 23:00:00 GMT+0 - Investigating - We're aware of an issue causing music playback to fail mostly due to the error `Internal Server Error` (another error may also occur). Investigation is currently pending. Apr 15, 14:20:00 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Apr 15, 14:40:13 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 15 hours and 40 minutes

Affected Components:

<p><small>Apr <var data-var='date'> 14</var>, <var data-var='time'>23:00:00</var> GMT+0</small><br><strong>Investigating</strong> - We're aware of an issue causing music playback to fail mostly due to the error `Internal Server Error` (another error may also occur). Investigation is currently pending..</p> <p><small>Apr <var data-var='date'> 15</var>, <var data-var='time'>14:20:00</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Apr <var data-var='date'> 15</var>, <var data-var='time'>14:40:13</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Fri, 14 Apr 2023 23:00:00 +0000 https://status.vexera.io/incident/clgi1bzdd270147ghogcve9d29z https://status.vexera.io/incident/clgi1bzdd270147ghogcve9d29z Bot down Type: Incident Duration: 16 hours and 45 minutes Affected Components: Commands processing Feb 11, 17:42:13 GMT+0 - Investigating - Vexera is currently unresponsive. We are investigating this incident. Feb 11, 17:55:45 GMT+0 - Identified - We have found the cause of the incident. Vexera is now cold-booting and should be available again in all servers in the next 30 minutes. Feb 11, 18:36:28 GMT+0 - Identified - We are having issues related to the recent growth of Vexera, and the bot is still starting up. Feb 11, 22:15:24 GMT+0 - Monitoring - Vexera should now be available and working again on all servers. Please note that you may experience some weirdness in the next few hours until we come up with a definitive fix. Feb 12, 10:27:34 GMT+0 - Resolved - Everything is now operating again in a nominal state. Feb 11, 19:20:18 GMT+0 - Identified - Around 75% of servers have already recovered. Recovery is slower for the last servers. Type: Incident

Duration: 16 hours and 45 minutes

Affected Components:

<p><small>Feb <var data-var='date'> 11</var>, <var data-var='time'>17:42:13</var> GMT+0</small><br><strong>Investigating</strong> - Vexera is currently unresponsive. We are investigating this incident..</p> <p><small>Feb <var data-var='date'> 11</var>, <var data-var='time'>17:55:45</var> GMT+0</small><br><strong>Identified</strong> - We have found the cause of the incident. Vexera is now cold-booting and should be available again in all servers in the next 30 minutes..</p> <p><small>Feb <var data-var='date'> 11</var>, <var data-var='time'>18:36:28</var> GMT+0</small><br><strong>Identified</strong> - We are having issues related to the recent growth of Vexera, and the bot is still starting up..</p> <p><small>Feb <var data-var='date'> 11</var>, <var data-var='time'>22:15:24</var> GMT+0</small><br><strong>Monitoring</strong> - Vexera should now be available and working again on all servers. Please note that you may experience some weirdness in the next few hours until we come up with a definitive fix..</p> <p><small>Feb <var data-var='date'> 12</var>, <var data-var='time'>10:27:34</var> GMT+0</small><br><strong>Resolved</strong> - Everything is now operating again in a nominal state..</p> <p><small>Feb <var data-var='date'> 11</var>, <var data-var='time'>19:20:18</var> GMT+0</small><br><strong>Identified</strong> - Around 75% of servers have already recovered. Recovery is slower for the last servers..</p> ]]> Sat, 11 Feb 2023 17:42:13 +0000 https://status.vexera.io/incident/cle08w49r146470jgmx34nwofbz https://status.vexera.io/incident/cle08w49r146470jgmx34nwofbz Elevated errors rate Type: Incident Duration: 6 hours and 52 minutes Affected Components: Commands processing Jan 21, 02:05:00 GMT+0 - Investigating - Our monitoring system fired an alert regarding elevated errors rate. We are investigating! Jan 21, 08:37:20 GMT+0 - Identified - Hi everyone, we are sorry for this incident, which sadly happened during the night for us. Vexera managed to self-heal from the incident and recovered for most servers within the next hour. We are now manually fixing the remaining servers where Vexera is still unresponsive. Jan 21, 08:57:29 GMT+0 - Resolved - Vexera should be working again in all servers! Type: Incident

Duration: 6 hours and 52 minutes

Affected Components:

<p><small>Jan <var data-var='date'> 21</var>, <var data-var='time'>02:05:00</var> GMT+0</small><br><strong>Investigating</strong> - Our monitoring system fired an alert regarding elevated errors rate. We are investigating!.</p> <p><small>Jan <var data-var='date'> 21</var>, <var data-var='time'>08:37:20</var> GMT+0</small><br><strong>Identified</strong> - Hi everyone, we are sorry for this incident, which sadly happened during the night for us. Vexera managed to self-heal from the incident and recovered for most servers within the next hour. We are now manually fixing the remaining servers where Vexera is still unresponsive..</p> <p><small>Jan <var data-var='date'> 21</var>, <var data-var='time'>08:57:29</var> GMT+0</small><br><strong>Resolved</strong> - Vexera should be working again in all servers!.</p> ]]> Sat, 21 Jan 2023 02:05:00 +0000 https://status.vexera.io/incident/cld5p0peu11089henc1uzke6lu https://status.vexera.io/incident/cld5p0peu11089henc1uzke6lu Vexera down Type: Incident Duration: 55 minutes Affected Components: Commands processing Jan 10, 20:04:48 GMT+0 - Investigating - We are aware of an issue with Vexera not responding in some servers. We are investigating. Jan 10, 20:22:49 GMT+0 - Identified - We are in the process of restarting Vexera. It should come back online on all servers in the next 20 minutes. Jan 10, 20:59:55 GMT+0 - Resolved - Vexera is online again in all servers, and everything looks back to normal so far. Type: Incident

Duration: 55 minutes

Affected Components:

<p><small>Jan <var data-var='date'> 10</var>, <var data-var='time'>20:04:48</var> GMT+0</small><br><strong>Investigating</strong> - We are aware of an issue with Vexera not responding in some servers. We are investigating..</p> <p><small>Jan <var data-var='date'> 10</var>, <var data-var='time'>20:22:49</var> GMT+0</small><br><strong>Identified</strong> - We are in the process of restarting Vexera. It should come back online on all servers in the next 20 minutes..</p> <p><small>Jan <var data-var='date'> 10</var>, <var data-var='time'>20:59:55</var> GMT+0</small><br><strong>Resolved</strong> - Vexera is online again in all servers, and everything looks back to normal so far..</p> ]]> Tue, 10 Jan 2023 20:04:48 +0000 https://status.vexera.io/incident/clcqnw7od66334h6ocqmbzewqu https://status.vexera.io/incident/clcqnw7od66334h6ocqmbzewqu Vexera down Type: Incident Duration: 3 hours and 4 minutes Affected Components: Scheduled events, Website, Commands processing, Image manipulation, Voice orchestration Dec 19, 17:03:00 GMT+0 - Investigating - Vexera is currently experiencing a major outage, leading to complete downtime in all servers. We are aware and are investigating this issue. As a side effect, the dashboard on our website is also unavailable at the time. You will see a 403 error until the aforementioned issue has been fixed. Vexera Pro is unaffected by this. Dec 19, 18:11:00 GMT+0 - Identified - Vexera is currently recovering. We are in the process of starting back up all affected shards, this may take a while. Another incident update will be ready when Vexera has fully recovered. Dec 19, 18:57:23 GMT+0 - Monitoring - All services have recovered from the outage. We are currently monitoring the situation until we're sure this incident is resolved. Dec 19, 20:06:49 GMT+0 - Resolved - After monitoring the situation, we're certain that Vexera is stable and fully back up. Thank you for your patience! Type: Incident

Duration: 3 hours and 4 minutes

Affected Components: , , , ,

<p><small>Dec <var data-var='date'> 19</var>, <var data-var='time'>17:03:00</var> GMT+0</small><br><strong>Investigating</strong> - Vexera is currently experiencing a major outage, leading to complete downtime in all servers. We are aware and are investigating this issue. As a side effect, the dashboard on our website is also unavailable at the time. You will see a 403 error until the aforementioned issue has been fixed. Vexera Pro is unaffected by this..</p> <p><small>Dec <var data-var='date'> 19</var>, <var data-var='time'>18:11:00</var> GMT+0</small><br><strong>Identified</strong> - Vexera is currently recovering. We are in the process of starting back up all affected shards, this may take a while. Another incident update will be ready when Vexera has fully recovered..</p> <p><small>Dec <var data-var='date'> 19</var>, <var data-var='time'>18:57:23</var> GMT+0</small><br><strong>Monitoring</strong> - All services have recovered from the outage. We are currently monitoring the situation until we're sure this incident is resolved..</p> <p><small>Dec <var data-var='date'> 19</var>, <var data-var='time'>20:06:49</var> GMT+0</small><br><strong>Resolved</strong> - After monitoring the situation, we're certain that Vexera is stable and fully back up. Thank you for your patience!.</p> ]]> Mon, 19 Dec 2022 17:03:00 +0000 https://status.vexera.io/incident/clbv4ty3t7209i7ls2bj7tjoh https://status.vexera.io/incident/clbv4ty3t7209i7ls2bj7tjoh Vexera down Type: Incident Duration: 2 hours and 21 minutes Affected Components: Scheduled events, Commands processing Dec 15, 19:47:52 GMT+0 - Investigating - Our monitoring system fired an alert regarding elevated errors rate. We are investigating! Dec 15, 20:12:24 GMT+0 - Identified - Vexera has been down for 25 minutes due to a database issue. It's currently being restarted, please allow up to 30 minutes to reconnect in all servers. *Vexera Pro is back online.* Dec 15, 20:41:47 GMT+0 - Monitoring - Vexera should be online again in all servers. A few things (members cache...) are still loading in the background, but we expect that everything is working again as expected. Dec 15, 22:08:48 GMT+0 - Resolved - Everything works as expected. The incident is now closed. Type: Incident

Duration: 2 hours and 21 minutes

Affected Components: ,

<p><small>Dec <var data-var='date'> 15</var>, <var data-var='time'>19:47:52</var> GMT+0</small><br><strong>Investigating</strong> - Our monitoring system fired an alert regarding elevated errors rate. We are investigating!.</p> <p><small>Dec <var data-var='date'> 15</var>, <var data-var='time'>20:12:24</var> GMT+0</small><br><strong>Identified</strong> - Vexera has been down for 25 minutes due to a database issue. It's currently being restarted, please allow up to 30 minutes to reconnect in all servers. *Vexera Pro is back online.*.</p> <p><small>Dec <var data-var='date'> 15</var>, <var data-var='time'>20:41:47</var> GMT+0</small><br><strong>Monitoring</strong> - Vexera should be online again in all servers. A few things (members cache...) are still loading in the background, but we expect that everything is working again as expected..</p> <p><small>Dec <var data-var='date'> 15</var>, <var data-var='time'>22:08:48</var> GMT+0</small><br><strong>Resolved</strong> - Everything works as expected. The incident is now closed..</p> ]]> Thu, 15 Dec 2022 19:47:52 +0000 https://status.vexera.io/incident/clbphuacr11997nolnpz6isyag https://status.vexera.io/incident/clbphuacr11997nolnpz6isyag Multiple voice nodes down Type: Incident Duration: 54 minutes Dec 4, 15:30:00 GMT+0 - Identified - An incident currently impacts a hosting provider hosting some of our voice nodes (servers handling voice encoding and playback). The load was transferred to the remaining nodes. As such, some side effects like lags may be observed until this issue is fixed. Dec 4, 15:55:00 GMT+0 - Monitoring - Impacted voice nodes are back in production. We are monitoring the situation while the load rebalances itself between nodes. Dec 4, 16:24:27 GMT+0 - Resolved - Everything is back to normal! Type: Incident

Duration: 54 minutes

<p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>15:30:00</var> GMT+0</small><br><strong>Identified</strong> - An incident currently impacts a hosting provider hosting some of our voice nodes (servers handling voice encoding and playback). The load was transferred to the remaining nodes. As such, some side effects like lags may be observed until this issue is fixed..</p> <p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>15:55:00</var> GMT+0</small><br><strong>Monitoring</strong> - Impacted voice nodes are back in production. We are monitoring the situation while the load rebalances itself between nodes..</p> <p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>16:24:27</var> GMT+0</small><br><strong>Resolved</strong> - Everything is back to normal!.</p> ]]> Sun, 4 Dec 2022 15:30:00 +0000 https://status.vexera.io/incident/clb9jd2r92671hvmyxiout2i4 https://status.vexera.io/incident/clb9jd2r92671hvmyxiout2i4 Planned maintenance Type: Maintenance Duration: 16 minutes Affected Components: Voice orchestration Nov 18, 07:00:00 GMT+0 - Identified - We will be deploying an update to some internal services of Vexera handling voice connections and playbacks. The update will cause a temporary disconnection of Vexera from all voice channels. The duration of the unavailability should be around 5 minutes. Nov 18, 07:00:01 GMT+0 - Identified - Maintenance is now in progress Nov 18, 07:15:30 GMT+0 - Completed - Maintenance has completed successfully. Type: Maintenance

Duration: 16 minutes

Affected Components:

<p><small>Nov <var data-var='date'> 18</var>, <var data-var='time'>07:00:00</var> GMT+0</small><br><strong>Identified</strong> - We will be deploying an update to some internal services of Vexera handling voice connections and playbacks. The update will cause a temporary disconnection of Vexera from all voice channels. The duration of the unavailability should be around 5 minutes..</p> <p><small>Nov <var data-var='date'> 18</var>, <var data-var='time'>07:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Nov <var data-var='date'> 18</var>, <var data-var='time'>07:15:30</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> ]]> Fri, 18 Nov 2022 07:00:00 +0000 https://status.vexera.io/maintenance/clalphhzt3709h7opstrwmspp https://status.vexera.io/maintenance/clalphhzt3709h7opstrwmspp Website shows Server Errors Type: Incident Duration: 7 hours and 15 minutes Affected Components: Website Aug 31, 02:00:00 GMT+0 - Resolved - The issue has been resolved. Aug 30, 18:45:23 GMT+0 - Investigating - The website is currently returning "Server Error" messages to some users. We are currently investigating. Clearing cookies can fix the issue for some users. Type: Incident

Duration: 7 hours and 15 minutes

Affected Components:

<p><small>Aug <var data-var='date'> 31</var>, <var data-var='time'>02:00:00</var> GMT+0</small><br><strong>Resolved</strong> - The issue has been resolved..</p> <p><small>Aug <var data-var='date'> 30</var>, <var data-var='time'>18:45:23</var> GMT+0</small><br><strong>Investigating</strong> - The website is currently returning "Server Error" messages to some users. We are currently investigating. Clearing cookies can fix the issue for some users..</p> ]]> Tue, 30 Aug 2022 18:45:23 +0000 https://status.vexera.io/incident/cl7gjhss428131k4ofg0qb5417 https://status.vexera.io/incident/cl7gjhss428131k4ofg0qb5417 Vexera offline Type: Incident Duration: 1 hour and 17 minutes Affected Components: Commands processing, Scheduled events Aug 18, 19:52:15 GMT+0 - Investigating - Vexera is currently offline in most servers. We are investigating! Aug 18, 20:21:06 GMT+0 - Identified - Our hosting provider had a short outage with their internal networking, which brought down Vexera. The issue has been resolved, and Vexera is booting back again. Please give it ~20 minutes. Aug 18, 20:52:05 GMT+0 - Monitoring - At this point, Vexera should be working again on all servers. We are monitoring the situation before closing this incident. Aug 18, 21:08:50 GMT+0 - Resolved - The issue has been resolved. Type: Incident

Duration: 1 hour and 17 minutes

Affected Components: ,

<p><small>Aug <var data-var='date'> 18</var>, <var data-var='time'>19:52:15</var> GMT+0</small><br><strong>Investigating</strong> - Vexera is currently offline in most servers. We are investigating!.</p> <p><small>Aug <var data-var='date'> 18</var>, <var data-var='time'>20:21:06</var> GMT+0</small><br><strong>Identified</strong> - Our hosting provider had a short outage with their internal networking, which brought down Vexera. The issue has been resolved, and Vexera is booting back again. Please give it ~20 minutes..</p> <p><small>Aug <var data-var='date'> 18</var>, <var data-var='time'>20:52:05</var> GMT+0</small><br><strong>Monitoring</strong> - At this point, Vexera should be working again on all servers. We are monitoring the situation before closing this incident..</p> <p><small>Aug <var data-var='date'> 18</var>, <var data-var='time'>21:08:50</var> GMT+0</small><br><strong>Resolved</strong> - The issue has been resolved..</p> ]]> Thu, 18 Aug 2022 19:52:15 +0000 https://status.vexera.io/incident/cl6zglk7o15224x7ogbiqwtar5 https://status.vexera.io/incident/cl6zglk7o15224x7ogbiqwtar5 Bot down Type: Incident Duration: 52 minutes Affected Components: Commands processing Jul 21, 19:30:00 GMT+0 - Investigating - We are currently investigating this incident. Jul 21, 19:52:46 GMT+0 - Identified - The underlying issue has been found, and the bot is currently restarting. Please allow up to 20 minutes for full recovery. Jul 21, 20:21:30 GMT+0 - Resolved - Vexera is now back online in all servers. Type: Incident

Duration: 52 minutes

Affected Components:

<p><small>Jul <var data-var='date'> 21</var>, <var data-var='time'>19:30:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Jul <var data-var='date'> 21</var>, <var data-var='time'>19:52:46</var> GMT+0</small><br><strong>Identified</strong> - The underlying issue has been found, and the bot is currently restarting. Please allow up to 20 minutes for full recovery..</p> <p><small>Jul <var data-var='date'> 21</var>, <var data-var='time'>20:21:30</var> GMT+0</small><br><strong>Resolved</strong> - Vexera is now back online in all servers..</p> ]]> Thu, 21 Jul 2022 19:30:00 +0000 https://status.vexera.io/incident/cl5vfr7c38316h0odw9w4jp6v https://status.vexera.io/incident/cl5vfr7c38316h0odw9w4jp6v Vexera may not respond to some commands Type: Incident Duration: 1 hour and 38 minutes Affected Components: Commands processing May 29, 14:24:13 GMT+0 - Identified - Vexera may currently not respond to some commands. Re-sending the command again may work. This is due to a lapse in processing resources that occurred following a rollout. This lapse of resources is causing Vexera to backlog events continuously, causing a very high ping until it became unresponsive. We have made the decision to drop ~20% of received messages on purpose to partially mitigate the issue and keep Vexera sort of responsive while we work on restoring everything. As such, some commands may be ignored. Re-sending them should work. May 29, 15:52:47 GMT+0 - Monitoring - We implemented a fix and currently monitoring the result. May 29, 16:02:09 GMT+0 - Resolved - We just resolved the issue! Type: Incident

Duration: 1 hour and 38 minutes

Affected Components:

<p><small>May <var data-var='date'> 29</var>, <var data-var='time'>14:24:13</var> GMT+0</small><br><strong>Identified</strong> - Vexera may currently not respond to some commands. Re-sending the command again may work. This is due to a lapse in processing resources that occurred following a rollout. This lapse of resources is causing Vexera to backlog events continuously, causing a very high ping until it became unresponsive. We have made the decision to drop ~20% of received messages on purpose to partially mitigate the issue and keep Vexera sort of responsive while we work on restoring everything. As such, some commands may be ignored. Re-sending them should work..</p> <p><small>May <var data-var='date'> 29</var>, <var data-var='time'>15:52:47</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and currently monitoring the result..</p> <p><small>May <var data-var='date'> 29</var>, <var data-var='time'>16:02:09</var> GMT+0</small><br><strong>Resolved</strong> - We just resolved the issue!.</p> ]]> Sun, 29 May 2022 14:24:13 +0000 https://status.vexera.io/incident/cl3re7pk28652lwoht574ppgk https://status.vexera.io/incident/cl3re7pk28652lwoht574ppgk Planned maintenance Type: Maintenance Duration: 2 hours and 30 minutes Affected Components: Image manipulation, Commands processing, Scheduled events, Voice orchestration May 26, 09:52:54 GMT+0 - Identified - We are continuing with the main instance now. May 26, 10:30:00 GMT+0 - Completed - Maintenance has completed successfully. May 26, 08:00:00 GMT+0 - Identified - This maintenance is the first of two-phase maintenance planned to fix recent instability (see the announcement in [our server](https://discord.gg/vexera) for more information). Vexera will be brought down for a few minutes while we make some preparatory changes and migrate our database server to a new host. **We expect the downtime to be 1 hour or less (until Vexera is available again in all guilds).** *Vexera Pro will also be unavailable for less than 5 minutes shortly before the start of the maintenance on the regular instance. At least one instance will therefore be available at any point in time.* This maintenance will not fix the underlying issue by itself but will allow us to work towards the next maintenance. May 26, 08:30:00 GMT+0 - Identified - Vexera Pro is back up. We are postponing the operation of the main instance by ~1 hour to check a few things. May 26, 08:00:01 GMT+0 - Identified - Maintenance is now in progress Type: Maintenance

Duration: 2 hours and 30 minutes

Affected Components: , , ,

<p><small>May <var data-var='date'> 26</var>, <var data-var='time'>09:52:54</var> GMT+0</small><br><strong>Identified</strong> - We are continuing with the main instance now..</p> <p><small>May <var data-var='date'> 26</var>, <var data-var='time'>10:30:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> <p><small>May <var data-var='date'> 26</var>, <var data-var='time'>08:00:00</var> GMT+0</small><br><strong>Identified</strong> - This maintenance is the first of two-phase maintenance planned to fix recent instability (see the announcement in [our server](https://discord.gg/vexera) for more information). Vexera will be brought down for a few minutes while we make some preparatory changes and migrate our database server to a new host. **We expect the downtime to be 1 hour or less (until Vexera is available again in all guilds).** *Vexera Pro will also be unavailable for less than 5 minutes shortly before the start of the maintenance on the regular instance. At least one instance will therefore be available at any point in time.* This maintenance will not fix the underlying issue by itself but will allow us to work towards the next maintenance..</p> <p><small>May <var data-var='date'> 26</var>, <var data-var='time'>08:30:00</var> GMT+0</small><br><strong>Identified</strong> - Vexera Pro is back up. We are postponing the operation of the main instance by ~1 hour to check a few things..</p> <p><small>May <var data-var='date'> 26</var>, <var data-var='time'>08:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> ]]> Thu, 26 May 2022 08:00:00 +0000 https://status.vexera.io/maintenance/cl3m6irzt1610leojrvisuevp https://status.vexera.io/maintenance/cl3m6irzt1610leojrvisuevp Elevated errors rate Type: Incident Duration: 5 hours and 59 minutes Affected Components: Commands processing May 19, 17:44:49 GMT+0 - Identified - We have located the root cause of the issue. Recovery is currently in progress, we hope to be back within the next hour. Thank you for your patience! May 19, 19:04:52 GMT+0 - Identified - The recovery process is taking longer than expected due to a huge amount of shards (~700 out of 1535) not having the correct guild count. We are still in the process of restarting all affected shards. Current ETA is 21:45 (UTC). Thank you for your continued patience while we are working towards a resolution to this issue. May 19, 22:58:39 GMT+0 - Resolved - This issue has been resolved. May 19, 17:00:00 GMT+0 - Investigating - Our monitoring system fired an alert regarding elevated errors rate. We are investigating! May 19, 22:02:05 GMT+0 - Monitoring - All affected shards have been restarted. Vexera is fully operational. We'll keep monitoring the situation for another few minutes until we mark the issue resolved. Type: Incident

Duration: 5 hours and 59 minutes

Affected Components:

<p><small>May <var data-var='date'> 19</var>, <var data-var='time'>17:44:49</var> GMT+0</small><br><strong>Identified</strong> - We have located the root cause of the issue. Recovery is currently in progress, we hope to be back within the next hour. Thank you for your patience!.</p> <p><small>May <var data-var='date'> 19</var>, <var data-var='time'>19:04:52</var> GMT+0</small><br><strong>Identified</strong> - The recovery process is taking longer than expected due to a huge amount of shards (~700 out of 1535) not having the correct guild count. We are still in the process of restarting all affected shards. Current ETA is 21:45 (UTC). Thank you for your continued patience while we are working towards a resolution to this issue..</p> <p><small>May <var data-var='date'> 19</var>, <var data-var='time'>22:58:39</var> GMT+0</small><br><strong>Resolved</strong> - This issue has been resolved..</p> <p><small>May <var data-var='date'> 19</var>, <var data-var='time'>17:00:00</var> GMT+0</small><br><strong>Investigating</strong> - Our monitoring system fired an alert regarding elevated errors rate. We are investigating!.</p> <p><small>May <var data-var='date'> 19</var>, <var data-var='time'>22:02:05</var> GMT+0</small><br><strong>Monitoring</strong> - All affected shards have been restarted. Vexera is fully operational. We'll keep monitoring the situation for another few minutes until we mark the issue resolved..</p> ]]> Thu, 19 May 2022 17:00:00 +0000 https://status.vexera.io/incident/cl3dabz9e256471m2ojdnqn0fvd https://status.vexera.io/incident/cl3dabz9e256471m2ojdnqn0fvd Multiple services unavailable Type: Incident Duration: 22 days and 3 hours Affected Components: Commands processing, Image manipulation, Website, Scheduled events, Voice orchestration May 18, 09:15:00 GMT+0 - Resolved - Vexera is operating normally. May 18, 07:37:11 GMT+0 - Identified - We have identified the issue and Vexera is currently recovering. It may take some time for Vexera to be available again in your server. Vexera Pro is back up and running. We'll have a detailed summary of events ready when the issue has been resolved. May 18, 08:45:02 GMT+0 - Monitoring - Vexera has successfully recovered. We are currently monitoring the service for further instability. May 18, 00:24:00 GMT+0 - Investigating - Vexera is currently unresponsive on all servers. Unfortunately, this issue occurred at night for the whole staff team, and there are no available developers at the moment. We hope to resume normal service during the early hours of Wednesday morning (UTC timezone). Type: Incident

Duration: 22 days and 3 hours

Affected Components: , , , ,

<p><small>May <var data-var='date'> 18</var>, <var data-var='time'>09:15:00</var> GMT+0</small><br><strong>Resolved</strong> - Vexera is operating normally..</p> <p><small>May <var data-var='date'> 18</var>, <var data-var='time'>07:37:11</var> GMT+0</small><br><strong>Identified</strong> - We have identified the issue and Vexera is currently recovering. It may take some time for Vexera to be available again in your server. Vexera Pro is back up and running. We'll have a detailed summary of events ready when the issue has been resolved..</p> <p><small>May <var data-var='date'> 18</var>, <var data-var='time'>08:45:02</var> GMT+0</small><br><strong>Monitoring</strong> - Vexera has successfully recovered. We are currently monitoring the service for further instability..</p> <p><small>May <var data-var='date'> 18</var>, <var data-var='time'>00:24:00</var> GMT+0</small><br><strong>Investigating</strong> - Vexera is currently unresponsive on all servers. Unfortunately, this issue occurred at night for the whole staff team, and there are no available developers at the moment. We hope to resume normal service during the early hours of Wednesday morning (UTC timezone)..</p> ]]> Wed, 18 May 2022 00:24:00 +0000 https://status.vexera.io/incident/cl3auds4817646aaoasiq929mx https://status.vexera.io/incident/cl3auds4817646aaoasiq929mx Image manipulation and wolfram not working Type: Incident Duration: 1 hour and 15 minutes Affected Components: Commands processing, Image manipulation May 9, 12:50:00 GMT+0 - Resolved - The issue has been resolved! May 9, 11:34:57 GMT+0 - Identified - We have identified an issue with our outbound proxy which impacts image manipulation commands and `+wolfram`. Type: Incident

Duration: 1 hour and 15 minutes

Affected Components: ,

<p><small>May <var data-var='date'> 9</var>, <var data-var='time'>12:50:00</var> GMT+0</small><br><strong>Resolved</strong> - The issue has been resolved!.</p> <p><small>May <var data-var='date'> 9</var>, <var data-var='time'>11:34:57</var> GMT+0</small><br><strong>Identified</strong> - We have identified an issue with our outbound proxy which impacts image manipulation commands and `+wolfram`..</p> ]]> Mon, 9 May 2022 11:34:57 +0000 https://status.vexera.io/incident/cl2ynczun776913j2op3te049p7 https://status.vexera.io/incident/cl2ynczun776913j2op3te049p7 Voice Service Impacted, Dashboard & Vexera Pro Unavailable Type: Incident Duration: 28 days and 18 hours Affected Components: Website, Voice orchestration, Commands processing May 3, 17:17:47 GMT+0 - Identified - We have identified the cause of the issue and are working on fixing the impacted voice service, as well as the dashboard. Vexera Pro has been fixed and is back up & running! ----------------------- **Edit:** We are currently restarting all affected shards. May 3, 08:00:00 GMT+0 - Investigating - Vexera is currently affected by 3 various issues: 1. One of our voice components, specifically "voice-manager-1", is currently unresponsive. Some servers may experience issues playing music. 2. The dashboard returns a "403 Forbidden" error once you select a server in your server list. Configuring a server through the dashboard is not possible at the time. 3. Vexera Pro is unresponsive to all commands. We're investigating the issues above and will update once further information becomes available. Thank you for your patience. May 3, 19:30:00 GMT+0 - Resolved - There may have been some lags/non-responses while we were disabling some workaround code we used to assist in recovery. But the issue is now considered resolved. May 3, 18:40:00 GMT+0 - Monitoring - As of now, all guilds should have recovered. We are keeping health metrics under close monitoring while our cache is finishing to warm up. For transparency, previous updates contained an error: the outage is not due to an issue with our voice service (although early investigations have reported this service as a possible cause). There will be more information to come when we will be sure that everything is fixed. May 3, 20:47:00 GMT+0 - Postmortem - The root cause is a network micro-cut that happened around 07:00 AM UTC. This caused a large number of shards to disconnect and reconnect at the same time. This caused some issues in our cache: 240 shards (out of a total of 704) had incorrect data cached. This cache contains critical data for the bot operations (like in which guilds Vexera is present, what are the channels in these guilds, the permissions, the members, etc.). Missing and invalid data prevented the bot to work correctly in some guilds. The issue, that first got identified by the support team based on user reports, wasn't correctly escalated to devs for remediation. All the people who run the bot are based in Europe, and at this hour, we were already on our worksites. This finally got escalated to someone with access to our servers at 02:17 PM UTC who tried basic actions to recover the bot without success. Being at work, they weren't able to do much more. Real investigations finally started around 05:00 PM UTC, and the first remediation steps were able to be executed by 05:10 PM UTC. After having identified the 240 shards affected by the issue, a rolling restart of them was started to fix the issue. This action was completed by 06:40 PM UTC, at which point almost all guilds were fixed. We then dug into our logs to search for and fix a few remaining guilds in an inconsistent state. At 07:30 PM UTC, we believed everything was back to normal, which was confirmed by our metrics. --- The time it took to fix this issue is unacceptable, and there is clearly a lack of proper escalation and handling processes for major outages that happen during workdays. We will see how we can improve that. Type: Incident

Duration: 28 days and 18 hours

Affected Components: , ,

<p><small>May <var data-var='date'> 3</var>, <var data-var='time'>17:17:47</var> GMT+0</small><br><strong>Identified</strong> - We have identified the cause of the issue and are working on fixing the impacted voice service, as well as the dashboard. Vexera Pro has been fixed and is back up & running! ----------------------- **Edit:** We are currently restarting all affected shards..</p> <p><small>May <var data-var='date'> 3</var>, <var data-var='time'>08:00:00</var> GMT+0</small><br><strong>Investigating</strong> - Vexera is currently affected by 3 various issues: 1. One of our voice components, specifically "voice-manager-1", is currently unresponsive. Some servers may experience issues playing music. 2. The dashboard returns a "403 Forbidden" error once you select a server in your server list. Configuring a server through the dashboard is not possible at the time. 3. Vexera Pro is unresponsive to all commands. We're investigating the issues above and will update once further information becomes available. Thank you for your patience..</p> <p><small>May <var data-var='date'> 3</var>, <var data-var='time'>19:30:00</var> GMT+0</small><br><strong>Resolved</strong> - There may have been some lags/non-responses while we were disabling some workaround code we used to assist in recovery. But the issue is now considered resolved..</p> <p><small>May <var data-var='date'> 3</var>, <var data-var='time'>18:40:00</var> GMT+0</small><br><strong>Monitoring</strong> - As of now, all guilds should have recovered. We are keeping health metrics under close monitoring while our cache is finishing to warm up. For transparency, previous updates contained an error: the outage is not due to an issue with our voice service (although early investigations have reported this service as a possible cause). There will be more information to come when we will be sure that everything is fixed..</p> <p><small>May <var data-var='date'> 3</var>, <var data-var='time'>20:47:00</var> GMT+0</small><br><strong>Postmortem</strong> - The root cause is a network micro-cut that happened around 07:00 AM UTC. This caused a large number of shards to disconnect and reconnect at the same time. This caused some issues in our cache: 240 shards (out of a total of 704) had incorrect data cached. This cache contains critical data for the bot operations (like in which guilds Vexera is present, what are the channels in these guilds, the permissions, the members, etc.). Missing and invalid data prevented the bot to work correctly in some guilds. The issue, that first got identified by the support team based on user reports, wasn't correctly escalated to devs for remediation. All the people who run the bot are based in Europe, and at this hour, we were already on our worksites. This finally got escalated to someone with access to our servers at 02:17 PM UTC who tried basic actions to recover the bot without success. Being at work, they weren't able to do much more. Real investigations finally started around 05:00 PM UTC, and the first remediation steps were able to be executed by 05:10 PM UTC. After having identified the 240 shards affected by the issue, a rolling restart of them was started to fix the issue. This action was completed by 06:40 PM UTC, at which point almost all guilds were fixed. We then dug into our logs to search for and fix a few remaining guilds in an inconsistent state. At 07:30 PM UTC, we believed everything was back to normal, which was confirmed by our metrics. --- The time it took to fix this issue is unacceptable, and there is clearly a lack of proper escalation and handling processes for major outages that happen during workdays. We will see how we can improve that..</p> ]]> Tue, 3 May 2022 08:00:00 +0000 https://status.vexera.io/incident/cl2qcrgxd12317gwloh95y9dgx https://status.vexera.io/incident/cl2qcrgxd12317gwloh95y9dgx Planned maintenance Type: Maintenance Duration: 3 hours and 59 minutes Affected Components: Website, Commands processing, Scheduled events, Image manipulation, Voice orchestration Apr 21, 10:00:01 GMT+0 - Identified - Maintenance is now in progress Apr 21, 11:39:41 GMT+0 - Identified - Our hosting provider maintenance has ended. Vexera Pro is working again. The main Vexera instance is currently reconnecting to your guilds. This process may take a few minutes, hang tight! Apr 21, 13:59:21 GMT+0 - Completed - Maintenance has completed. Vexera should have recovered in most guilds since 12:30 UTC. We are currently handling an issue in a subset of guilds were Vexera is still unresponsive. These guilds should have recovered in the next 10-15 minutes. Apr 21, 10:00:00 GMT+0 - Identified - Our hosting provider has planned a maintenance on April 21st at 10:00 UTC, to replace a network equipment that has issues. During this maintenance, both Vexera and Vexera Pro will be temporarily offline. The maintenance is planned to take approximately 2 hours. Link to the hosting provider status page: [https://bare-metal-servers.status-ovhcloud.com/incidents/hhmw7kglyrb3](https://bare-metal-servers.status-ovhcloud.com/incidents/hhmw7kglyrb3) Type: Maintenance

Duration: 3 hours and 59 minutes

Affected Components: , , , ,

<p><small>Apr <var data-var='date'> 21</var>, <var data-var='time'>10:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Apr <var data-var='date'> 21</var>, <var data-var='time'>11:39:41</var> GMT+0</small><br><strong>Identified</strong> - Our hosting provider maintenance has ended. Vexera Pro is working again. The main Vexera instance is currently reconnecting to your guilds. This process may take a few minutes, hang tight!.</p> <p><small>Apr <var data-var='date'> 21</var>, <var data-var='time'>13:59:21</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed. Vexera should have recovered in most guilds since 12:30 UTC. We are currently handling an issue in a subset of guilds were Vexera is still unresponsive. These guilds should have recovered in the next 10-15 minutes..</p> <p><small>Apr <var data-var='date'> 21</var>, <var data-var='time'>10:00:00</var> GMT+0</small><br><strong>Identified</strong> - Our hosting provider has planned a maintenance on April 21st at 10:00 UTC, to replace a network equipment that has issues. During this maintenance, both Vexera and Vexera Pro will be temporarily offline. The maintenance is planned to take approximately 2 hours. Link to the hosting provider status page: [https://bare-metal-servers.status-ovhcloud.com/incidents/hhmw7kglyrb3](https://bare-metal-servers.status-ovhcloud.com/incidents/hhmw7kglyrb3).</p> ]]> Thu, 21 Apr 2022 10:00:00 +0000 https://status.vexera.io/maintenance/cl24qjdx9145675k7n4qo8wjlug https://status.vexera.io/maintenance/cl24qjdx9145675k7n4qo8wjlug Vexera unresponsive Type: Incident Duration: 20 hours Affected Components: Commands processing Apr 6, 18:45:00 GMT+0 - Investigating - Our monitoring system fired an alert regarding elevated errors rate. We are investigating! Apr 6, 19:05:00 GMT+0 - Resolved - Vexera was unresponsive for about 10 minutes (18:44-18:53 UTC). This issue is due to the deployment of a bogus update. The update was quickly tested on our beta instance before the deployment, and no issues were found (we used only very simple tests/commands). **Edit: See at the end** The issue was therefore only identified when the update got deployed to our production instance, as alarms about high errors rate fired. We immediately initiated a rollback, but an issue in our orchestrator prevented it from automatically receiving the rollback instruction, delaying the recovery of Vexera. Please accept our apologies for this outage 😦 ----------------------- **Edit:** After investigation, it looks like we had almost zero chances to identify the bug. The bug occurs only if the user sending the message doesn't have any configuration option set in our database (eg. if you never used `+userlocale` or such commands). We actually have user config set on our accounts on both our dev and beta instances… Apr 6, 18:53:00 GMT+0 - Monitoring - We rolled back an update and currently monitoring the result. Type: Incident

Duration: 20 hours

Affected Components:

<p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>18:45:00</var> GMT+0</small><br><strong>Investigating</strong> - Our monitoring system fired an alert regarding elevated errors rate. We are investigating!.</p> <p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>19:05:00</var> GMT+0</small><br><strong>Resolved</strong> - Vexera was unresponsive for about 10 minutes (18:44-18:53 UTC). This issue is due to the deployment of a bogus update. The update was quickly tested on our beta instance before the deployment, and no issues were found (we used only very simple tests/commands). **Edit: See at the end** The issue was therefore only identified when the update got deployed to our production instance, as alarms about high errors rate fired. We immediately initiated a rollback, but an issue in our orchestrator prevented it from automatically receiving the rollback instruction, delaying the recovery of Vexera. Please accept our apologies for this outage 😦 ----------------------- **Edit:** After investigation, it looks like we had almost zero chances to identify the bug. The bug occurs only if the user sending the message doesn't have any configuration option set in our database (eg. if you never used `+userlocale` or such commands). We actually have user config set on our accounts on both our dev and beta instances….</p> <p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>18:53:00</var> GMT+0</small><br><strong>Monitoring</strong> - We rolled back an update and currently monitoring the result..</p> ]]> Wed, 6 Apr 2022 18:45:00 +0000 https://status.vexera.io/incident/cl1scxilz487523rjnc8cjyy75l https://status.vexera.io/incident/cl1scxilz487523rjnc8cjyy75l High latency Type: Incident Duration: 5 days and 22 hours Affected Components: Commands processing, Scheduled events Feb 13, 17:30:00 GMT+0 - Investigating - Vexera is currently handling commands with very high latency (30+ seconds). We are investigating. Feb 13, 17:43:00 GMT+0 - Investigating - As the underlying issue got quickly out of control, we are going to do a cold restart of Vexera to recover. Vexera will slowly become available in servers again. The whole operation can take up to 1 hour. Feb 13, 19:52:00 GMT+0 - Resolved - Everything is back to normal for all servers. The issue that triggered this incident is know since some time, and work is in progress to fix it. Sorry for the inconvenience 😦 Type: Incident

Duration: 5 days and 22 hours

Affected Components: ,

<p><small>Feb <var data-var='date'> 13</var>, <var data-var='time'>17:30:00</var> GMT+0</small><br><strong>Investigating</strong> - Vexera is currently handling commands with very high latency (30+ seconds). We are investigating..</p> <p><small>Feb <var data-var='date'> 13</var>, <var data-var='time'>17:43:00</var> GMT+0</small><br><strong>Investigating</strong> - As the underlying issue got quickly out of control, we are going to do a cold restart of Vexera to recover. Vexera will slowly become available in servers again. The whole operation can take up to 1 hour. .</p> <p><small>Feb <var data-var='date'> 13</var>, <var data-var='time'>19:52:00</var> GMT+0</small><br><strong>Resolved</strong> - Everything is back to normal for all servers. The issue that triggered this incident is know since some time, and work is in progress to fix it. Sorry for the inconvenience 😦.</p> ]]> Sun, 13 Feb 2022 17:30:00 +0000 https://status.vexera.io/incident/cl12h8o2r209387b6ljmav4nvix https://status.vexera.io/incident/cl12h8o2r209387b6ljmav4nvix Vexera unresponsive Type: Incident Duration: 3 hours and 43 minutes Affected Components: Commands processing Jan 19, 18:31:00 GMT+0 - Resolved - Vexera is back up! Jan 19, 14:48:00 GMT+0 - Investigating - Vexera is currently unresponsive and shows as offline in all servers. Both **Vexera** and **Vexera Pro** are affected by this. We're looking into the issue and will send out updates as they become available. Thank you for your patience! Jan 19, 16:29:00 GMT+0 - Identified - Vexera is starting back up. Vexera Pro is back up & has been for around 15 minutes. We will provide update when Vexera is completely back up. Type: Incident

Duration: 3 hours and 43 minutes

Affected Components:

<p><small>Jan <var data-var='date'> 19</var>, <var data-var='time'>18:31:00</var> GMT+0</small><br><strong>Resolved</strong> - Vexera is back up!.</p> <p><small>Jan <var data-var='date'> 19</var>, <var data-var='time'>14:48:00</var> GMT+0</small><br><strong>Investigating</strong> - Vexera is currently unresponsive and shows as offline in all servers. Both **Vexera** and **Vexera Pro** are affected by this. We're looking into the issue and will send out updates as they become available. Thank you for your patience!.</p> <p><small>Jan <var data-var='date'> 19</var>, <var data-var='time'>16:29:00</var> GMT+0</small><br><strong>Identified</strong> - Vexera is starting back up. Vexera Pro is back up & has been for around 15 minutes. We will provide update when Vexera is completely back up..</p> ]]> Wed, 19 Jan 2022 14:48:00 +0000 https://status.vexera.io/incident/cl12hlomb223155b6ljh9whueut https://status.vexera.io/incident/cl12hlomb223155b6ljh9whueut Vexera unresponsive Type: Incident Duration: 3 hours Affected Components: Commands processing, Voice orchestration, Scheduled events Jan 15, 10:00:00 GMT+0 - Identified - Due to a database issue, Vexera is currently unresponsive. We're resolving this now and Vexera will be back up soon. Jan 15, 11:48:00 GMT+0 - Identified - Vexera is restarting, it will be online again in your guild within 15-30 minutes. Jan 15, 13:00:00 GMT+0 - Resolved - Vexera is back up everywhere 🙂 Type: Incident

Duration: 3 hours

Affected Components: , ,

<p><small>Jan <var data-var='date'> 15</var>, <var data-var='time'>10:00:00</var> GMT+0</small><br><strong>Identified</strong> - Due to a database issue, Vexera is currently unresponsive. We're resolving this now and Vexera will be back up soon..</p> <p><small>Jan <var data-var='date'> 15</var>, <var data-var='time'>11:48:00</var> GMT+0</small><br><strong>Identified</strong> - Vexera is restarting, it will be online again in your guild within 15-30 minutes..</p> <p><small>Jan <var data-var='date'> 15</var>, <var data-var='time'>13:00:00</var> GMT+0</small><br><strong>Resolved</strong> - Vexera is back up everywhere 🙂.</p> ]]> Sat, 15 Jan 2022 10:00:00 +0000 https://status.vexera.io/incident/cl12i2kys235519b6ljfr7uowix https://status.vexera.io/incident/cl12i2kys235519b6ljfr7uowix High latency Type: Incident Duration: 2 hours and 57 minutes Affected Components: Commands processing, Voice orchestration Jan 11, 16:10:00 GMT+0 - Investigating - We are currently investigating a high bot latency. Jan 11, 19:07:00 GMT+0 - Resolved - During the last 3 hours, Vexera experienced again the same issue than 3 days ago. The probable cause of these issues is a contexts leak in an internal service that causes slowness after a few days of operations and ultimately a backlogging of Discord events when encountering our daily peak hours around 17:00 UTC. We will implement scheduled restarts of the affected service to mitigate the issue until we can identify and fix the cause of these leaks. Sorry for the inconvenience Type: Incident

Duration: 2 hours and 57 minutes

Affected Components: ,

<p><small>Jan <var data-var='date'> 11</var>, <var data-var='time'>16:10:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating a high bot latency..</p> <p><small>Jan <var data-var='date'> 11</var>, <var data-var='time'>19:07:00</var> GMT+0</small><br><strong>Resolved</strong> - During the last 3 hours, Vexera experienced again the same issue than 3 days ago. The probable cause of these issues is a contexts leak in an internal service that causes slowness after a few days of operations and ultimately a backlogging of Discord events when encountering our daily peak hours around 17:00 UTC. We will implement scheduled restarts of the affected service to mitigate the issue until we can identify and fix the cause of these leaks. Sorry for the inconvenience.</p> ]]> Tue, 11 Jan 2022 16:10:00 +0000 https://status.vexera.io/incident/cl12iimng196566yljpckk9bj2 https://status.vexera.io/incident/cl12iimng196566yljpckk9bj2 High latency/Errors when executing commands Type: Incident Duration: 2 days and 5 hours Affected Components: Commands processing, Voice orchestration Jan 8, 16:15:00 GMT+0 - Investigating - Vexera may respond with a delay. We are investigating the issue. Jan 8, 17:08:00 GMT+0 - Resolved - Jan 8, 17:07:00 GMT+0 - Monitoring - Vexera experienced issues during the last hour which have lead to delayed responses and `Internal Server Error` messages (mainly with music related commands). In an attempt to remediate the issue, we have restarted the voice manager (our internal service controlling the voice encoders). This may have disconnected Vexera from voice channels. We believe the issue is now fixed, and our error rate is back to normal levels since 15 minutes. However, we are still monitoring the situation, because the root cause of the issue is still unknown. Type: Incident

Duration: 2 days and 5 hours

Affected Components: ,

<p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>16:15:00</var> GMT+0</small><br><strong>Investigating</strong> - Vexera may respond with a delay. We are investigating the issue..</p> <p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>17:08:00</var> GMT+0</small><br><strong>Resolved</strong> - .</p> <p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>17:07:00</var> GMT+0</small><br><strong>Monitoring</strong> - Vexera experienced issues during the last hour which have lead to delayed responses and `Internal Server Error` messages (mainly with music related commands). In an attempt to remediate the issue, we have restarted the voice manager (our internal service controlling the voice encoders). This may have disconnected Vexera from voice channels. We believe the issue is now fixed, and our error rate is back to normal levels since 15 minutes. However, we are still monitoring the situation, because the root cause of the issue is still unknown..</p> ]]> Sat, 8 Jan 2022 16:15:00 +0000 https://status.vexera.io/incident/cl12i8qo346896yljq09lm4xh https://status.vexera.io/incident/cl12i8qo346896yljq09lm4xh Vexera unresponsive in some servers Type: Incident Duration: 46 minutes Affected Components: Commands processing Jan 5, 11:56:00 GMT+0 - Monitoring - Vexera should have recovered in all servers. We are monitoring our error rate. Jan 5, 12:06:00 GMT+0 - Resolved - The issue is now resolved. Jan 5, 11:20:00 GMT+0 - Identified - We are aware that Vexera is unresponsive on some servers. We are currently taking remediation steps to resolve this issue. Type: Incident

Duration: 46 minutes

Affected Components:

<p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>11:56:00</var> GMT+0</small><br><strong>Monitoring</strong> - Vexera should have recovered in all servers. We are monitoring our error rate..</p> <p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>12:06:00</var> GMT+0</small><br><strong>Resolved</strong> - The issue is now resolved..</p> <p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>11:20:00</var> GMT+0</small><br><strong>Identified</strong> - We are aware that Vexera is unresponsive on some servers. We are currently taking remediation steps to resolve this issue..</p> ]]> Wed, 5 Jan 2022 11:20:00 +0000 https://status.vexera.io/incident/cl12in0xq243756yljeh5p6dyj https://status.vexera.io/incident/cl12in0xq243756yljeh5p6dyj Planned maintenance Type: Maintenance Duration: 2 hours Affected Components: Website, Commands processing, Voice orchestration, Scheduled events Dec 28, 13:00:00 GMT+0 - Completed - Maintenance has completed successfully. Dec 28, 11:00:00 GMT+0 - Identified - Our hosting provider has planned a maintenance starting today at 11:00 UTC, to reboot a network equipment that have issues. During this maintenance, Vexera may go temporarily offline. If that happens, this shouldn't take longer than 1 hour. Dec 28, 12:42:00 GMT+0 - Identified - We took this opportunity to upgrade the infrastructure behind Vexera. We have completed the infrastructure upgrades, and our hosting provider has completed their maintenance. We will be starting Vexera back up in the next 15 minutes or so. Vexera will be back online in your server in under an hour. Type: Maintenance

Duration: 2 hours

Affected Components: , , ,

<p><small>Dec <var data-var='date'> 28</var>, <var data-var='time'>13:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> <p><small>Dec <var data-var='date'> 28</var>, <var data-var='time'>11:00:00</var> GMT+0</small><br><strong>Identified</strong> - Our hosting provider has planned a maintenance starting today at 11:00 UTC, to reboot a network equipment that have issues. During this maintenance, Vexera may go temporarily offline. If that happens, this shouldn't take longer than 1 hour..</p> <p><small>Dec <var data-var='date'> 28</var>, <var data-var='time'>12:42:00</var> GMT+0</small><br><strong>Identified</strong> - We took this opportunity to upgrade the infrastructure behind Vexera. We have completed the infrastructure upgrades, and our hosting provider has completed their maintenance. We will be starting Vexera back up in the next 15 minutes or so. Vexera will be back online in your server in under an hour..</p> ]]> Tue, 28 Dec 2021 11:00:00 +0000 https://status.vexera.io/maintenance/cl12is4dc302626yljfuemxwt1 https://status.vexera.io/maintenance/cl12is4dc302626yljfuemxwt1