Forum Update: We are back and running, but with some losses

TwistedTwiglegTwistedTwigleg Posts: 3,330Admin
edited October 1 in Forum Chat

Hey everyone,

As many of you probably know, the forums went down for an extend period recently. Recently, when you visited the forums, there would be an error message telling you an error had occur, and to return to the forums later. The issue seems to have occurred only for the past couple days, but unfortunately what was causing the error was rather grave.

What happened

Quick context: I am responsible for the hosting of the Godot Community Forums and any server maintenance, data, etc.

After visiting the forums, I debugged the issue the forums was reporting. The issue the forums was giving was it was having difficulty accessing some of the data in the database the forums uses. Upon investigation, I found that almost all of the forum data in the database was missing and the database itself was corrupt. I do not know what happened exactly and I am still investigating the issue, but it appears that something triggered the database to delete itself.

After many hours of work looking through the databases, the server logs, contacting the server provider and talking with their technicians, and salvaging data in the databases, I was able to get the forums back up. The data that was successfully recovered was the discussions, questions, and comments. However, all of the other data, including users, was lost. Thankfully I had a backup of the forum data from a couple months ago, so I was able to fill in most of the gaps in the data with that backup.

However, you may notice that there are a lot of posts by a user called "UnknownUser". This is a user I created. "UnknownUser" is a placeholder for all of the discussions, questions, and comments that were salvaged but didn't have a working user ID assigned to them. Based on the data, it is estimated that roughly 500-600 discussions and around 800 comments lost their user ID. All of these discussions and comments are now assigned to "UnknownUser" for now. Additionally, around 1,000 user accounts, around 2 months of private messages, a bunch of YAGA reactions, and all of the activity log data after the backup was lost.

The important part of the forum, the discussions and comments, were mercifully able to be salvaged from the corrupt database. Any comments or discussions made by users whose account was created prior to the last couple months should see all of their posts and comments correctly attributed to them. Accounts made in the last couple months will need to be remade and their previous posts and comments will be attributed to "UnknownUser" for the time being.

Again, I am still investigating the issue and am working to protect the forums and its data so this issue cannot happen again.

What does this mean?

Well, short-term, all users should change their passwords. I do not think whatever happened was a data breach, I think it was just a deletion, but it is better to be safe than sorry. I do not know if the deletion was triggered by a malicious individual or group, or if the issue was just general database corruption. Again, I do not believe it was a data breach, but please change your password if you have an account on the forums.

Additionally, if you created a account on the forums recently, you may need to remake it. To know whether you need to remake your account, please try to reset your password by clicking "I forgot my password". This should send you an email with a password reset, though because of the increased mail traffic, please give it a few hours to send the email. If you do not receive an email a few hours after resetting your password, please make a new account.

The forums are currently back up and running like before. You should be able to interact with the forums like normal, though some functionality has been disabled while I figure out what happened.

I know this is an alarming and unfortunate situation, and I want to thank everyone for their patience and support as we figure this out moving forward. If you have any questions or concerns, please let us know! Thanks!

Update: I have a topic that was lost and is now attributed to "unknown user", how do I recover it?

Please make a new topic with the same name as the old topic, ideally linking to the old topic in the opening post of the new topic. Then please let us on forum staff know (either by using @name, private messaging, or flagging the topic and explaining) and we will see about merging the old topic into the new one.
Thanks!

Comments

  • HippoHippo Posts: 134Member

    Thanks for your hard work in resolving this issue and getting the forums back up and running again!

  • cyberealitycybereality Posts: 1,063Moderator

    Unfortunate event, but good to see you recovered most of the forum.

    Any thoughts on having a auto-backup of the database or something to mitigate this in the future?

  • MegalomaniakMegalomaniak Posts: 2,958Admin

    For now weekly backups instead of every few months, but manual afaik.

  • TwistedTwiglegTwistedTwigleg Posts: 3,330Admin

    Yeah, as Megalomaniak said. I plan to see if I can automate the process though, but for now, I’ll be backing it up manually every week.

  • ChevifierChevifier Posts: 8Member
    edited September 28

    Im sure its the 3rd time I had to make an account. I thought I was going crazy.
    I didnt notice the giant red notification at first lol

  • KanataEXEKanataEXE Posts: 5Member

    Okay, I should have reset my password first instead of making a new account, but it keeps saying that no user/email exists, but I make a new one.

  • TwistedTwiglegTwistedTwigleg Posts: 3,330Admin

    @KanataEXE said:
    Okay, I should have reset my password first instead of making a new account, but it keeps saying that no user/email exists, but I make a new one.

    Then making a new one is probably the right thing to do. The account was probably lost if it doesn't know of an account attached to the email/user.

  • TwistedTwiglegTwistedTwigleg Posts: 3,330Admin

    Update: I added a explanation for what to do if you have a topic that was lost and you want to recover it to the opening post.
    Also, on the 3rd, I think I'll make the red banner about what happened to the forums closable, and then fully remove it on the 10th it will be removed entirely, since everyone has likely seen it at that point.

  • ojn17041991ojn17041991 Posts: 4Member

    Thanks for your work in getting the forums back up and running. If you don't mind me asking, why do you think the database deletion was not done maliciously?

  • TwistedTwiglegTwistedTwigleg Posts: 3,330Admin

    @ojn17041991 said:
    Thanks for your work in getting the forums back up and running. If you don't mind me asking, why do you think the database deletion was not done maliciously?

    Thanks @ojn17041991! The forums should be good for the foreseeable future, especially now that I’m more aware of the potential (that and doing regular backups)

    As for the database deletion, there are a number of things it could be and I don’t want to jump to conclusions. For example, it could have been a storage drive related error, though its unlikely since the server provider said everything was fine on their end, but its possible. It could also be a bug in Vanilla, the forum software we use, that caused the deletion, perhaps a instruction that went haywire or something.

    To be fair, I think it was an injection attack of some form, meaning someone or something specifically targeted the forums. I do not think the database was copied/stolen though, just deleted. However, since we do not know for sure, that is why I am encouraging everyone to change their passwords, because it is best to careful than have problems down the line.

    Ultimately, the reason I am not assuming malicious intent is because I do not want to assume something or someone with negative intent was necessarily attacking the forums. Until I have a better idea, I want to try to keep an open mind as much as possible. :smile:

  • xquaxqua Posts: 7Unconfirmed Member

    Whoa I am sorry for you ! I feel you being the main hoster for some services ! when this happens, cold sweat starts running down !

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file