More and more organizations failed to apply cloud security controls and recommendations, even with the essential stuff, such as Posture Management. While security incidents occur all time, one common scenario is the issue with VM/EC2 open port to the public network.

You are probably asking yourself why I’m writing a post about an old topic. Such as exposed VM… It’s still relevant. More than ever is in the wild and is risky because many attacks start directly in the cloud and are initiated from the cloud machines.

Cloud incident is on the rise. For example, Uber has been hacked yet again with code and employee data released online. Based on the information, Uber has suffered from another high-profile data leak that exposed sensitive employee and company data. This time, attackers breached the company by compromising an AWS server used by a third party that provides Uber with asset management and tracking services.

Every basic compliance audit or risk assessment will notify you to secure a public port, such as SSH or RDP. In addition, every time a scanner checks the configuration of your cloud assets, you’ll be warned about it. For example, if you don’t secure SSH on Azure VM, AWS EC2, or Google CE. You’ll undoubtedly get a critical alert stating that one of your Security rules has its port open. This security issue happens often, and when this happens, you may think, have I been pwned? Sometimes the security team doesn’t know.

You wouldn’t be at fault because misconfiguration happens all time, and CSPM tools are often quite alarming on this topic. In this article, I will go through the known issue with exposed VM and explain the threats you are susceptible to when you have this misconfiguration with SSH.

ℹ️ Note: The post focuses on Azure VM, but the behavior is the same for AWS EC2 or Goole CE.

SSH on VM

SSH on VM, what could be wrong? And Why SSH and not RDP? While there are commonly attacked ports exposed to the public network, would they include: RDP (3389), FTP (21), HTTP (80), SQL (1433), etc.? That can be because of the following reasons:

• Linux is the most used OS in Azure and other cloud vendors.
• SSH is a common issue in many environments.
• SSH is enabled by default on Linux machines.
• Linux machines with SSH expose to the internet by misstep.
• When you circle up a Linux machine, SSH access is configured via an RSA key pair or other creds.
• SSH is not monitored on SOC as they need to be.

SSH is the standard for remotely managing machines and replaces telnet as the secure alternative. This is the main entry point for attackers since once you can access SSH, you are a few steps closer to gaining control of the entire cloud environment. Therefore, it’s essential to minimize misconfiguration, create friction with illegal actions, and audit SSH activities.

However, this ease of access also makes it convenient for attackers. Azure and other Cloud providers explain how to secure SSH with their best practices. In summary:

• Don’t use an Administrator account with a username and password.
• Don’t expose the SSH port to the public network.
• Don’t allow the root user to use an SSH terminal.
• Force all users to log in with an SSH key pair.
• Deactivate password authentication.
• Use SSH key enforcer with Azure Policy.
• Enforce conditional access to ssh connections.

What are security issues with opening SSH connections from untrusted/unknown sources? There are many answers. Here are some of them:

• Password-Based Attacks.
• Increase the exposed external attack surface.
• Key Pair Leaks.
• Potential of lateral movement after taking over.

The Reality of exposing SSH

Depending on what we’ve, if it is the defense approach, the CSPM tool, or the Azure VM configuration, an attacker may take control of more or less of your resources. The final result depends on the variety of misconfigurations in your cloud environment. The following scenarios in which your SSH port is accessible can be:

If we look at the MITRE ATT&CK, the main tactics and techniques will be the following:

Let’s take these scenarios and put them into context.

SSH without Auth

If you want to see your Cloud environment burn, keep your VM without authentication and the public network exposed with an ssh port. Bear in mind that an exposed VM with no authentication looks very strange in the case of a honeypot. A particular scenario can occur if there are high misconfigurations or, in a case that creates Honeypot.

SSH with Creds Auth

This is a typical scenario, but in the case of VM\EC2, you need to configure it manually. As a user, you may be motivated to configure your machines in this way to be able to access via SSH from anywhere. And you will create the credentials with a false feeling of being safe. Here’s the problem; in this specific scenario, with SSH open to the world behind simple credentials like a username and password, the security depends entirely on the robustness of your password.

A straightforward search in Shodan provides some findings for exposed ssh. You can search with additional parameters, and the number will be higher.

SSH with Key Pair Auth

This is the default configuration for Azure VM. In this situation, the private key isn’t stored in Azure and can only be retrieved when it’s created. The threat could be leaking your key pair by mistake or by an insider. This is something that can happen but is not unique to SSH access.

The creation of a “key pair” is similar to that of a Linux system if your VM is based on Linux and the public key is stored in the ‘~/.ssh/authorized_keys’ folder. If you are using GitHub to store it, tools like DumpsterDiver are handy tools to scrap repositories and able to find keys. The main issue is how to manage the key pairs.

Cryptographic stuff – The key pair generation is based on RSA standards. The possibility of generating two different “key pairs” accessing the same instance is minor. Therefore, a brute force attack is not practical in this scenario. Still, it can happen.

SSH BF Testing

To run security testing on cloud machines is pretty standard – you need a BF tool + password list + user list and target. Still, you can take it to the next level and run a custom PowerShell script with that evade from the CSPM\CWP tools.

While running security testing against an Azure Linux VM with a Brute-Forcer (custom-made PowerShell) with 100k passwords with port 22, I saw nothing on some of the Top leader CSPM\CWP tools (checked several times with different environments). No alert, no warning. Nothing suspicious. Sometimes it’s identified and alerted after a while. That is extremely bad!

ℹ️ Note: The excellent news for MDC is that SSH brute force was detected after a short period. 

This specific brute force was tested with the following conditions:

• Different user names.
• A list with 100k passwords.
• Attacked from different public IPs.
• Custom PowerShell.
• 10-15 hours of testing, total.
• Few targets on the same Azure subscription and external subnet.
• A known tool such as NMAP or Hydra wasn’t part of the testing.

A brute force in action against my Linux VM on Azure. The script ran for a few days, each for a few hours.

To display a list of the failed SSH logins in Linux, the most straightforward command you can use is grep “failed password.”

ℹ️ Note: Because the Azure VM is open to the public network, I saw other IPs attempting to guess the password.

ℹ️ TIP: In Shodan, you’ve got a filter that can search for cloud machines with no key pair. It can be a good starting point.

The MXDR Behavior

There are a few methods to detect SSH Brute-Force, whether by MDC, MDE on a cloud machine, or with a Microsoft Sentinel agent. The Microsoft Defender for Cloud identified the attack based on specific metrics and alert rules. With MDC, the identification is done within an hour (around 20 minutes).

In the MXDR method with Microsoft Sentinel, Defender for Cloud, and Defender for Endpoint, the attack was identified after no more than 15-20 minutes – for all of them and is super accurate because of the correlation between the signals.

As always, I love to take it to the extreme, so I configured a bunch of queries on Microsoft Sentinel and Defender for Endpoint to identify the most accurate attacks or potential and made auto-triage when a false-positive appears.

So, what does it look like from the MXDR?

Microsoft Sentinel

Microsoft Sentinel can work in several ways:

• Microsoft Sentinel agent
• MDC integration
• MDE integration
• Zeek
• Syslog
• Multi-cloud scenario with EC2\GCE

ℹ️ TIP: Many cloud VMs not cover by EDR or equivalent – make sure to apply EDR to your cloud VMs to identify the ‘little things.’

In Microsoft Sentinel, we received the incidents and started the investigation. This one was raised from Defender for Cloud. The investigation by the console is excellent, but with KQL is much better. With the console, we need to go through the following:

• Main incidents
• Timeline
• Similar incident
• Entities
• Incident Graph

While the console is pretty good and does the job, I prefer to investigate with KQL.

Microsoft Sentinel with Syslog (guest mode) – With Microsoft Sentinel, we can install an agent and configure the Azure VM with guest configuration with Syslog mode. Also, we can install the Microsoft Sentinel agent on a (or EC2\GCE). In this situation, we can dig into the Linux logs (auth.log, etc.).

Once we’ve got the Syslog table, we can run queries against Syslog

let sysmsg = dynamic ([
“Disconnected from authenticating”,
“Failed password”,
“error: Received disconnect”
]);
Syslog
| where Facility == “auth” // an optional to accurate for a specific log
| where SourceSystem == “Linux”
| where SyslogMessage has_any (sysmsg)
| project TimeGenerated, HostName, SeverityLevel, SyslogMessage, ProcessName

Syslog
| where SyslogMessage startswith “Failed Password”
| extend User = extract(“for(?s)(.*)from”,1,SyslogMessage)
| extend IPaddr = extract(“(([0-9]{1,3})\\.([0-9]{1,3})\\.([0-9]{1,3})\\.(([0-9]{1,3})))”,1,SyslogMessage)
| project TimeGenerated, HostName, SyslogMessage, IPaddr, User

let sysmsg = dynamic ([
“Disconnected from authenticating”,
“Failed password”,
“error: Received disconnect”
]);
Syslog
| where SyslogMessage startswith “Failed Password”
| extend User = extract(“for(?s)(.*)from”,1,SyslogMessage)
| extend IPaddr = extract(“(([0-9]{1,3})\\.([0-9]{1,3})\\.([0-9]{1,3})\\.(([0-9]{1,3})))”,1,SyslogMessage)
| where SyslogMessage has_any (sysmsg)
| project TimeGenerated, HostName, SyslogMessage, IPaddr, User

Defender for Cloud

The Microsoft Defender for Cloud provides the data, and we can investigate the alert vis MDC or Microsoft Sentinel. The exciting thing with Defender for Cloud is that the references shown in the environment depend on the resources and services you’re protecting and your customized configuration. For SSH, there are few alerts, and for Brute Force, we’ve got two specific alerts.

• VM_SshBruteForceFailed
• VM_SshBruteForceSuccess

ℹ️ TIP: Take a look at the reference because there are additional detections for SSH.

First, Security alerts raise alerts about Brute Forcing Linux VM.

In the real case scenario when ‘Accounts used to sign in to host successfully,’ the MDC raises two or more alerts, one for Failed SSH brute force attack with Pre-Attack and the second with ‘Successful SSH brute force attack.’

All of the alerts provide the information for the attack:

• Number of failed authentication attempts to host
• Accounts used on failed sign-in to host attempts
• Was the SSH session initiated
• Related entities
• Attacker source computer name
• Several distinct users failed to authenticate
• Number of successful authentication attempts to host
• Accounts used to sign in to host successfully

Defender for Endpoint

Many people asked me why they need EDR on cloud machines – we have CNAPP… and then we run security testing (PT/RTO) on cloud machines, and we get the facts that CNAPP cannot identifies many testing (operations), such as malware, memory dump, etc. With EDR, the SSH Brute force scenario can detect in minutes.

ℹ️ Note: CNAPP tools are great, and we need them! But they cannot do everything! If you’ve got an EDR, cover the cloud machines too.

The DeviceLogonEvents table in the advanced hunting schema contains information about user logins and device authentication events.

The following query can be of custom rule

Password-Based Attack Detection

As a security person, I know that a misconfiguration will occur on cloud machines sooner or later. If it is exposed to a public network or some sys-admin creates a Linux machine with a user and password, and maybe both. 😮 Misconfig, it’s unstoppable.

Many tools are commonly used to brute force user account passwords over SSH, including those used in Linux brute force compromise cases in Azure. A specific example is password spray attacks, where an attacker uses one or a few passwords against multiple well-known user accounts in an environment. These attacks typically attempt common passwords hoping one or more accounts can be compromised across multiple virtual machines. Attackers limit the number of password-guess attempts to avoid account lockout and detection by defenders.

When it comes to Linux VM on the cloud, it does not behave as they should. This scenario is one of many that behave differently for Linux. The known tools, such as Hydra or NMAP, are excellent, but a custom-made PowerShell or Python is much more powerful. The security tools don’t know how to ‘eat’ that behavior, and the combination of custom-made tools and the fact that CSPM\CWP lacks detection create another blindspot on the cloud.

ℹ️ TIP: you can compare the ‘auth. logs’ on the Linux vm against the CSPM\CWP tool, and you will understand the differences – visibility will be the first.

Create friction with SSH Brute-Forcing

If you have a Linux VM facing the internet, there will be loads of SSH brute-force attempts daily, many of which are automated (probably by a Bot). There are a few solutions to help defend against this and reduce the number of login attempts. Possibly one of the easiest things to do is change the port number on which SSH operates. Although this will prevent the essential brute-force attempts, scanning for SSH running on alternate ports is trivial.

General Methods

If password-based authentication is vital, use strong passwords and follow best practices. A better method is to implement a service like Fail2ban or DenyHosts to block brute-force attempts at the host level. This, combined with private key authentication instead of passwords, will put you out of the reach of most attackers.

Azure Methods

SSH key enforcer – As part of Microsoft’s quest to get rid of passwords, they published an Azure Policy that helps ensure Azure Linux VMs use SSH key authentication instead of passwords. You can deploy this policy to your Azure subscription or management group to prevent the creation of Linux VMs with passwords as the SSH authentication type. This policy blocks the deployment if the ‘disablePasswordAuthentication’ property in a VM isn’t defined or is set to ‘false’ when the VM image publisher and offer are well-known Linux offerings. After deploying this policy, any new deployments, including a Linux VM with password-based authentication, fail.

Staying Secure – The Microsoft Defender for Cloud recommendation, “Authentication to Linux machines should require SSH keys,” checks for SSH authentication after a VM has been deployed. The recommendation is a built-in Guest Configuration policy that monitors changes made to the SSH password authentication setting on the machine itself; if a VM was initially deployed with SSH key-based authentication and later password-based authentication is enabled in the SSH settings, this recommendation will report the VM as non-compliant. This recommendation is part of the Secure Score and follows Azure Security Benchmark best practices.

Azure AD Alternative – An alternative to SSH keys is using Azure AD authentication. This allows Azure AD credentials to log in to Azure Linux VMs.

AWS Methods

Use AWS Systems Manager Session Manager for shell access to EC2 instances – Session Manager allows AWS IAM users to log in to your instances with encryption and logging capabilities. The systems Manager’s traffic goes through the Systems Manager Endpoint, allowing easy and secure access to private instances without opening inbound ports.

Use EC2 Instance Connect for shell access to EC2 instances – EC2 Instance Connect allows you to connect to your Linux instances using Secure Shell IAM roles and policies.

References

Security best practices for IaaS workloads in Azure

Eliminate Password-Based Attacks on Azure Linux VMs

Keep EC2 Linux instances secure when using SSH

Automation to Block Brute-force Attacked IP detected by Microsoft Defender for Cloud.

Reference table for all security alerts in Microsoft Defender for Cloud

Common-Credentials-Password

Microsoft Defender for Endpoint on Linux | Microsoft Learn

Shreder is a powerful multi-threaded SSH protocol password brute-force tool