Joel A. A. Bredaoutages
Curated list of all outages with statements and timestamps.
Outage timestamps are recorded in UTC +2 time.
NOTE: Commit messages may be slightly changed from their original
state to make things more clear. This should not be more than 2 words.
Outages have been recorded since 22.08.26. There are no prior outages
that have been recorded.
Outages
Live DB-Establish-Check
@ 29.08.26 23:30-23:52 | Statement
After trying to block HTTP access to /api via Commit #102a82b 'feat: add rule to deny /api via HTTP' after
recent security updates, the entire root was accidently blocked. This was discovered 17 minutes later.
Then, 2 minutes later, Commit #5a7c298 'fix: block /api explicitly' blocked as said, /api explicitly.
@ 29.08.26 23:30-23:52 | Timeline
29.08.26 23:30 -- commit #102a82b was pushed
29.08.26 23:47 -- error was found
29.08.26 23:49 -- commit #c102a82b identified as the culprit
29.08.26 23:52 -- committing #102a82b
29.08.26 23:52 -- pushing #102a82b
29.08.26 23:52 -- site is back 'online'
@ 23.08.26 22:15-22:18 | Statement
After removing leftover terminal-like objects, the skills tab was also edited and a skill_path was added. The problem was, the skill_path was never added in the backend breaking root. This was quickly caught just about 30 seconds after the push. Commit #c2831f6 'chore: remove terminal alias' was the cause. 3 minutes later Commit #a2d16e0 'chore: add skill_path tag to skills array' delivered the fix.
@ 23.08.26 22:15-22:18 | Timeline
22.08.26 22:15 -- commit #c2831f6 was pushed
22.08.26 22:15 -- error was found. site left as is
22.08.26 22:15 -- commit #c2831f6 identified as the culprit
22.08.26 22:18 -- committing #a2d16e0
22.08.26 22:18 -- pushing #a2d16e0
22.08.26 22:18 -- site is back online
@ 22.08.26 17:59-18:50 | Statement
After what seems like a normal refactor, a major relative-path error was found after the main page of the subdomain docs had raw html. Then, after investigating I found every file breaking backend and being turned into raw html. The commit that made those changes was Commit #4d32fe9 'refactor: imgs to api, css to api' (Commit #475c5bf 'refactor')
Commit #475c5bf 'refactor' was a follow up commit after #4d32fe9 brought in a merge conflict and was silently lost across multiple terminals. #475c5bf was force pushed without further review as Commit #4d32fe9 was already reviewed. It turned out that they were not the same as one expected. In the end, there were now multiple broken relative-paths as #4d32fe9 already was pushed and changed various paths while #475c5bf did not change the paths in the files which I did not realise until deployment. 48 minutes later Commit #7233d1d 'fix: fixed routing, fixed .htaccess' delivered the right paths. In the future, even commits that seem like a copy will be reviewed and refactor commits will require more information.
@ 22.08.26 17:59-18:50 | Timeline
22.08.26 17:59 -- refactor api architecture
22.08.26 18:02 -- site shutdown after errors were found
22.08.26 18:13 -- scheduled fix @ 19:00
22.08.26 18:18 -- fixed relative path at root
22.08.26 18:20 -- error found in root. one line change
22.08.26 18:23 -- reviewed root
22.08.26 18:24 -- reschedule opening 20 minutes up
22.08.26 18:25 -- moving onto subdomains
22.08.26 18:42 -- fixed relative-paths @ all subdomains
22.08.26 18:46 -- running tests
22.08.26 18:49 -- no errors during tests
22.08.26 18:49 -- committing #7233d1d
22.08.26 18:50 -- pushing #7233d1d
22.08.26 18:50 -- site is back online
23.08.26 00:13 -- commit #475c5bf verified as the culprit