In reply to a post by jchamier:

depending how you've disabled SSH, you may need to check your SSHD config.

Well, er, its all GUI for me so to check my SSHD config (after a trip around the google) I'd have to enable SSH in order to access the Terminal. <- That may or may not expose my level of ignorance...

Anyways, SSH is "off". My backup app can connect using SFTP on port 22. I can also do another backup using rsync, also on port 22. So that all seems fine to me. But I can change the port number if that's recommended. I think.

In reply to a post by Woolwich:

In reply to a post by jchamier:

depending how you've disabled SSH, you may need to check your SSHD config.

Well, er, its all GUI for me so to check my SSHD config (after a trip around the google) I'd have to enable SSH in order to access the Terminal. <- That may or may not expose my level of ignorance...

Anyways, SSH is "off". My backup app can connect using SFTP on port 22. I can also do another backup using rsync, also on port 22. So that all seems fine to me. But I can change the port number if that's recommended. I think.

Unless the system is very old, the sftp process is part of ssh. There shouldn't be any reason not to use ssh certs for access and disable password authentication to keep it secure.

In reply to a post by mrkevlh:

In reply to a post by Woolwich:

How can that work for an automated backup script or app? I set the app up to connect to port whatever using name and password. I think that's the limit of what it can do. Long and complex passwords? Or is that another myth? Non-standard usernames maybe?

Thanks again.

If you need to access remote SSH systems securely you don't use usernames and passwords. Instead you use strong SSH certs and disable password authentication.

Really, you want to explain that to the EPCC, Tu-Dresden and Leibniz Supercomputing Centre and a good few others.

The problem with SSH keys is you have no way of knowing if the private key has been secured with a passphrase. Turns out that lots of users don't actually secure them with a passphrase and hackers where able to use this to hop from HPC to HPC system across Europe.

Even if they are secured with a passphrase they are no more secure than the passphrase used to secure them, which is likely to be the same as an existing password.

If you believe SSH keys are more secure than usernames and passwords then frankly you are an uniformed idiot.

In reply to a post by mrkevlh:

Unless the system is very old, the sftp process is part of ssh. There shouldn't be any reason not to use ssh certs for access and disable password authentication to keep it secure.

Yes there is SSH certs are a panacea for the clueless and uniformed who have no actual experience in running secure systems. There is nothing wrong with password authentication, they are in fact *MORE* secure than SSH keys.

In reply to a post by caffn8me:

Opening up SSH to the outside world is obviously a security risk as some folks have recently discovered.

There are many things going on here and without context this is disingenuous. My day job is running an High Performance Computer (HPC) or supercomputer to the layman. I have been running public facing servers now for over 25 years and never had a machine of mine compromised.

Firstly HPC sites like ARCHER run by the EPCC, have to make themselves available to remote login. To not do so would make them largely pointless.

Secondly EPCC have to some extent been hoisted by their own petard in this case. There is a lot of "tail wagging the dog" going on in account management IMHO at the EPCC. Basically if I have more than one project I have to have separate accounts because for each project because that makes it easier for EPCC to manage things and then each account has to have a different password. As you can imagine under such awkward conditions bad practice by the end user prevails.

In my case, appropriate firewall rules means allowing inbound access from specific IP addresses to these servers. All my servers run SSH but not all are accessible from any IP address.

For HPC sites this is basically impossible and anyway would not have helped the EPCC.

I do run some SSH servers accessible from everywhere (except for countries I've decided are rogue) and for these I run SSH on a non-standard port. I know people will say "but if people scan you they will find your SSH servers anyway" but that's highly unlikely.

Most SSH scans target port 22 and any IP address attempting to connect to me on this port gets autoblocked for a significant period of time. Also, a port
scan results in an automatic block for a long time.

And I have logs that show none standard SSH ports being found on never before used IP addresses within 36 hours. If you think that a none standard SSH port helps I have a slightly used bridge with only a few snapped cables and brand new truss end links to sell you.

If someone does find my SSH server by chance (which has never happened, and I've got the logs to prove it) then they get three attempts at the password before Fail2ban locks them out - again for a significant time. Even if they get the username and password correct, they still need to use multifactor authentication - which is tied to devices in my possession

Good idea, but this would not have helped the EPCC and multifactor authentication is not trivial to implement when you have hundreds of users. If you only have a handful or it's just yourself then MFA is pointless just pick a decent random password and don't write it down.

So what happened at the EPCC and other HPC sites across the UK and Europe back in April? One thing to bear in mind is that users of HPC machines often have accounts on multiple machines across countries and continents. Certainly the machine I look after has users with accounts on ARCHER, and other systems that where compromised including the ones in Munich and Dresden. These are sites that absolutely would be on your allow list of IP addresses. Saying use a VPN is impractical because an end user on one system does not get to setup a VPN to another system.

Well firstly an account belonging to a user in Poland (or at least a user account on a system in Poland) was compromised. The method for this compromise has not been published.

They where able to use this local access to gain root on the systems (more on this later). Once you have root on a system it is trivial to scan all the accounts looking for private SSH keys that have not been secured with a passphrase (which turns out to be lots). On the HPC system I run it takes under one second to scan just shy of 900 user accounts. You can then take a look at the users known_hosts file (if it is not hashed) to look for potential hosts to try those keys on. Note even if known_hosts is hashed I could just scan your Bash history file, it is also trivial to do. In a small amount of defence if your Google setting up SSH keys a lot of tutorials on the web tell you not to supply a passphrase for your private SSH key. This includes "trusted" sources like RedHat for crying out load.

Using this method the hackers (who where coming from IP addresses originating in China) where able to move from HPC system to system across Europe. They where not able to gain root on all systems which with the lack of new local privilege escalation vulnerabilities published in the last two months would indicate they where using a pre existing vulnerability that was not patched. HPC systems are notorious for not applying patches regularly for a range of reasons a good few of which revolve around patches changing the results of computations which screws up your research...

My guess is that https://nvd.nist.gov/vuln/detail/CVE-2019-18634 was the mechanism used for the local to root privilege escalation. It is the most likely candidate and the deafening silence when I have asked the sites effected is noticeable. I want to know is there a zero day I need to be worried about on my HPC site or is down to a lack of patching. Of course HPC sites don't want to admit to a lack of patching...

You might say well ban IP addresses in China or Russia from connecting to your system. The problem with that is we have users from those countries as students and researchers at our University who have accounts on our system for the research. At the moment a few of them are stuck back in their home countries due to COVID-19 lockdowns....

A good password that only exists in your head with a none standard account name (root, admin or a first name are all bad) something like ahd12935 is much better, combined with rate limiting connection attempts is way more secure than SSH keys. We have actually disabled SSH key based access because we cannot secure it. We have just had to [censored] a couple of users for sharing passwords....

The take away from all this is the HPC systems where hacked due to the stupidity of users over their handling of SSH keys which as an admin you have absolutely no way of checking up on and probably a lack of timely patching by admins.

In reply to a post by jabuzzard:

If you believe SSH keys are more secure than usernames and passwords then frankly you are an uniformed idiot.

Both mechanisms have their own pro and cons but I don't think it is very polite to be calling people idiots when someone has a different opinion to yours.

Very apposite Dilbert cartoon today.

https://dilbert.com/strip/2020-06-23

In reply to a post by mrkevlh:

In reply to a post by jabuzzard:

If you believe SSH keys are more secure than usernames and passwords then frankly you are an uniformed idiot.

Both mechanisms have their own pro and cons but I don't think it is very polite to be calling people idiots when someone has a different opinion to yours.

When you respond down the line to a post that says look at the risks of SSH keys, these guys got hacked, by suggesting to switch to a method of authentication that was used in said hack you are by definition uninformed and an idiot. Don't want to be called those things don't comment on things you know nothing about...

Consequently this is nothing to do with opinion, it's all about facts. Fact ARCHER (and Cirrus and a number of other HPC systems in the UK and Europe) where hacked where the entry point was an SSH key. Not a password but an SSH key.

Your method of authentication turned out in the real world to be not as secure as you thought. In fact without SSH keys the hack would have been unlikely to happen.

Ultimately all an SSH key is, is a very long random password (about 300 characters) that is impossible to remember so to have any hope of being secure has to be secured with another much shorter password. However it turns out the much shorter password is optional and users being lazy don't bother with it in about 1/5th of cases. On my HPC system ~21% of all private SSH keys on my HPC system had no passphrase, over 890 accounts where about 126 had private SSH keys. I know scan for and delete such keys several times a day in true BOFH fashion.

So why are SSH keys allegedly more secure? Well attempting to guess them is a mugs game. However if you rate limit your failed login attempts this turns out to be moot point.

So on my system assuming you have managed to work out a valid username which you won't do because they all look something like cng23584 (they are in effect random) you have only 2592 guess every 24 hours, except the password in use has complexity rules enforced (it's authenticated of the Universities AD) so good luck on cracking it.

Meanwhile that private SSH key which you have no idea whether or not has a passphrase can be used to access your system, and you think that is more secure. Well more fool you.

if you want security two factor is the only way to go in general. With passwords in general being more secure than SSH keys in the real world.

Don't get me wrong, SSH keys have their place, and we use them in tightly controlled scenarios. However for remote access to a system they are a nightmare and insecure as hell.

In reply to a post by jabuzzard:

And I have logs that show none standard SSH ports being found on never before used IP addresses within 36 hours.

And?

Read and learn.

In reply to a post by Woolwich:

Why is it "highly" unlikely scanners will find that port? Do you recommend any (or a range)? Did I recently read ports for SSH should be below a certain number? 1024 perhaps? I've seen 2222 suggested in the past.

It's because my firewalls are configured to blacklist any IP address which attempts to connect to them on any port and interface address combination that isn't running an explicitly permitted service.

You can, for example, get to unrestricted web servers I run via ports 80 and 443 but if you try to telnet to those same IP addresses, you'll be locked out for a considerable time period, inlcuding from previously available services. The lockout is from all ports on all firewall interface addresses. A scanner attempting to connect has to wait for the lockout to expire before they can probe a second port. Given the number of ports to choose from and the lockout period, it will take somewhat more than ten years to scan all ports on a single interface on my firewall from a single IP address. Not all interfaces run SSH either.

Secondly, I monitor logs for failed connection attempts and where I see coordinated probes from specific address ranges I put a permanent block in place. Most probes I see are from Chinese and Russian controlled IP addresses - which may not be geographically located in China or Russia respectively.

Some firewalls support geolocation blocking and it's also possible to block specific countries using https://www.ipdeny.com/ block lists. If you don't need ssh access from China and Russia, I'd strongly suggest blocking ssh connections from these countries. There may be other countries you also consider to be rogue nations.

One thing which is very important is that you log all failed access attempts and you analyse all failures. In my case, if I get a failed login attempt via SSH on the correct open port, Fail2ban emails me immediately with details. I don't have email notifications of all failed attempts to closed ports because there are far too many, but I do review the logs regularly.

Don't use port 2222 because that's a commonly used ssh alternative and I see regular probes to this and other n22 ports. It will be found very quickly by scanners.

Even if they get the username and password correct, they still need to use multifactor authentication - which is tied to devices in my possession

How can that work for an automated backup script or app? I set the app up to connect to port whatever using name and password. I think that's the limit of what it can do. Long and complex passwords? Or is that another myth? Non-standard usernames maybe?

It's perfectly possible to use ssh keys for rsync securely. An ssh key is only insecure, with or without a password, if someone unauthorized gets hold of it. You need to stop that happening. You can't 'guess' a key but it can be very easy to guess a password such as 123ABCdef.

You can also restrict what a key is allowed to do. You can, for example, state that the key can only run rsync on the remote server and no other commands. Have a look at Rsync - via SSH with no password, utilizing SSH ForceCommand in the authorized_keys file to limit the commands that can be run with that SSH key for how to do that.