Ubuntu gone wonko

About 2pm today my Ubuntu server went wonko . Wrong.. broke.. like it's haunted.

And oddly at the same time my main router got reset back to factory settings....

I have a few user accounts on my Ubuntu run server and the main account naga just vanished... Poof gone. The others where still there but I could not login via ssh to any of them. So I rebooted the server... And on reboot ALL user accounts where gone and I was greeted with a set up screen to add a user!

What happend?! I NEED one of those user folders back if at all possible.

Why on earth would they just get deleted like that?
I'm so confused.

On top of that, once I made a user again the desktop vanished after a few mins and all I had was a black screen with a flashing cursor.

I had to reboot it and was once again greeted with a setup screen. If left there the same thing happens after a while, black screen and I have to reboot.

What has happened to my server 😭😭

I have things like uptime kumar and webminninstalled and they still work so it's not like the server got wiped

Edited by Seansmit17 (Wed 14-Feb-24 18:22:11)

Could be that the disk that has /etc on it is flakey, or has got corrupted. As you had a problem with your router at the same time it could have been a power blip.

First thing I would do is boot from a rescue disk and then run fsck on all your file systems.

Are just the accounts gone - what about the home directories under /home? Check this before running fsck. Use the -N option first time you run fsck.

Don’t you have any backups of important data?

The router could be power related but not the server, it's on a ups.

It's the users and all files in home.. it's all gone.

/Home/amp are where the files I needed were.

Sadly any backups made by amp was also in the same dir. I had not got round to sorting out a cloud back up or even one to my windows machine.

I am running this in hyper v on my windows machine.

I'll do a disk check tonight when I am home as doing all that remotely is a pain.

In reply to a post by Seansmit17:

I had not got round to sorting out a cloud back up or even one to my windows machine.

Backup regimes aren't something I "get around to", they are priority number 1.

Lesson learned then isn't it.

Im hoping there's some logs somewhere that might shed some light on what or who. I think someone logged in via ssh and did some [censored] about... And if that is the case I hope there's some evidence left.

I'll see if I can get lucky with file recovery software.

The files lost are for 2 game servers. It's no real big issue just a pain in the ass. Amp stores it's files under home and also it's backups...

I'll have to change that. Might have amp move it's data store to a network share on my windows pc instead. Would make doing backups more simple.

You might want to tighten up the security of your ssh connection.

https://www.makeuseof.com/ways-to-secure-ssh-connect...

Ok, So I am home and I have been taking a look at the logs.

It seems that someone or multiple people have been trying to brute force their way in via SSH and FTP.

A whole load of failed login attempts. But it only takes on. I have trying to decipher the logs and see what user got logged in, what method and what command they ran.

It looks like user david logged in and then changed the root password and then deleted another user "root2" this was from a SKY IP address and he is on sky. but he says thats not his IP address...

Would someone mind taking a quick nose at the log?

https://drive.google.com/file/d/1TpViNr38BROiPhtzvq9...

The part regarding david logging in from a sky IP is at Feb 14 14:35:52 nagatek sshd[20437]:

almost at the end of the log.

Was it that user that did this.. that is the question. I will continue to view the logs. In the mean time I am trying some file recovery options to try and get the game server files back and if i do or not I am going to nuke the install and start again and follow the advice in the link shared for better security.

In reply to a post by Seansmit17:

It seems that someone or multiple people have been trying to brute force their way in via SSH and FTP.

FTP is redundant these days, SSH has SFTP and works just as well, and uses a key file instead of a password which is much more secure. Password logins should always be disabled (as advised on the SSH tips page).

I have passwords set still but I've moved ssh to anothwr port, only a few users can login, IP banned after 3 failed attempts to login.

I'll sort out ssh keys tomorrow as well as a proper back up.