CYBERDOM

Another day of Cybersecurity

AWS CloudTrail – HIDDEN_DUE_TO_SECURITY_REASONS

Published · Updated

As part of ongoing research and hunting, as well as investigating security incidents, I encounter many cases where there are gaps in security tools, systems do not document and collect logs properly or do not display them as we would like, attackers are very skilled, and finding evidence can be complex.

Visibility and logs missing or not displayed correctly can decide the results of an incident investigation and even lead to a lack of visibility. In the cloud, lack of visibility is one of the worst things.

There are many cases where logs are not displayed correctly, whether at the AWS CloudTrail, GCP audit, or Azure’s signin logs. In such scenarios, we can look for compensatory controls that will close the gaps, but sometimes, we may find ourselves without closing the required controls and be with a gap and a known risk.

The following article concerns one of the many issues of missing documentation and logging in AWS CloudTrail. Some will say that it is not a security matter and, therefore, it is not a problem, and there will be others for whom it is an issue.

This issue appeared in an environment where I connected the AWS CloudTrail to Microsoft Sentinel. After running the SOC process and, in particular, publishing the Analytical rules, I saw an attacker that was sitting against the AWS environment performing brute force and password spray. As part of the hunting and investigation, it was found that the logs give accurate information, but some information is not precise enough.

Tens of thousands of logs with the message “HIDDEN_DUE_TO_SECURITY_REASONS” were part of the inaccurate information.

AWS CloudTrail in a Nutshell

AWS CloudTrail is a vital service for monitoring and auditing activity within AWS accounts. It essentially creates an event history that you can use to track what happened, who did it, when it happened, and from where. This information is crucial for tasks like:

  • Security: CloudTrail can help you identify and investigate suspicious activity. For instance, if there’s a sudden surge in API calls, CloudTrail can help you pinpoint the source and determine if there’s a potential security breach.
  • Compliance: Many compliance regulations require organizations to track user activity and API calls. CloudTrail provides a centralized log of this activity, making it easier to meet compliance requirements.
  • Operational Auditing: CloudTrail helps you understand how your AWS resources are used. This information can help troubleshoot issues, identify inefficiencies, etc.

CloudTrail offers several ways to store and analyze your event history:

  • CloudTrail trails: These configurations deliver events to an Amazon S3 bucket you specify. You can also integrate CloudTrail with Amazon CloudWatch Logs and Amazon EventBridge for further analysis.
  • CloudTrail Lake: This managed data lake is specifically designed for storing, querying, and analyzing CloudTrail events. CloudTrail Lake simplifies security and audit logging by providing a centralized location for your event data.

Three types of events can be logged in CloudTrail:

  • Management events
  • Data events
  • Insights events

AWS CloudTrail features

Amazon CloudTrail has several features you would expect from a monitoring and governance tool. These features include:

  • AWS CloudTrail is always on, enabling you to view data from the most recent 90 days.
  • Event History to see all changes made regarding the creation, modification, or deletion of AWS resources.
  • Multi-region configuration, extended to all newly launched regions, and monitoring of changes with AWS Config.
  • Log file integrity validation and encryption can be used with further encryption services, including AWS KMS.
  • Data events, management events, and CloudTrail Insights.

By default, trails and event data stores log management events but not data or Insights events. All event types use a CloudTrail JSON log format.

What is cloud forensics exactly? Is it log analysis? Every cloud-logging conversation and discussion reminds me of the following tweet.

CloudTrail for Threat Hunters

In the context of AWS CloudTrail, threat hunting aims to proactively identify and mitigate security threats within an AWS environment. Here are the specific goals of threat hunting in CloudTrail:

  • Finding Breaches and Malicious Activity: The primary objective is to detect any unauthorized or malicious activities within the AWS environment. This involves identifying potential security breaches, unauthorized access attempts, or suspicious behavior that may indicate an ongoing or potential threat.
  • Fueling the Design and Creation of New Detection Rules: Through threat hunting, security teams can gather insights into emerging attack patterns, tactics, and techniques used by threat actors targeting AWS environments. These insights can then be used to develop and refine detection rules and algorithms for identifying similar threats in the future.
  • Building Subject Matter Expertise on Threat Activity: By actively investigating potential threats and analyzing CloudTrail logs, security teams can better understand the specific threats and attack vectors relevant to their AWS environment. This expertise enables more effective threat detection, response, and mitigation strategies tailored to the organization’s unique risk profile.
  • Documenting Learnings to Share with Others: Threat-hunting activities generate valuable insights and lessons learned about AWS security threats and effective detection techniques. Documenting these findings allows organizations to share knowledge and best practices with the broader security community, contributing to collective efforts to improve cloud security posture and resilience against evolving threats.

Each hunt process typically begins with a hypothesis about potential attacker behavior based on known threat intelligence, previous incidents, or emerging security trends. The goal is to validate or refute these hypotheses by analyzing CloudTrail logs and other relevant data sources, following a systematic and iterative approach akin to the scientific method.

Performing hunt sessions based on hypotheses is an effective way to proactively identify and mitigate potential threats in AWS environments. Here are the hypotheses you’ve listed, along with the objectives of each hunt session:

  • Hypothesis: An attacker may attempt to create an IAM user to persist in an environment they’ve breached.
    • Objective: Validate whether there are any unauthorized attempts to create new IAM users or modify existing ones within the AWS environment. This involves analyzing CloudTrail logs and IAM activity to detect suspicious or unauthorized user permissions or role changes.
  • Hypothesis: An attacker may attempt to leverage Amazon SES for email spam purposes.
    • Objective: Investigate whether there are any anomalous patterns of email sending activity through Amazon Simple Email Service (SES), which may indicate unauthorized use of SES for spamming or phishing purposes. This involves monitoring SES API calls and analyzing email sending patterns to identify potential misuse or abuse.
  • Hypothesis: An attacker who has compromised an environment with private EC2 instances may attempt to access it by creating or importing EC2 key pairs and modifying security groups.
    • Objective: Verify whether any suspicious changes to EC2 key pairs or security group configurations may indicate unauthorized access attempts to private EC2 instances. This involves examining CloudTrail logs for EC2 API calls related to key pair management and security group modifications and correlating this activity with other indicators of compromise.
  • Hypothesis: An attacker with compromised AWS credentials may attempt to enumerate the permissions assigned to a stolen identity.
    • Objective: Determine whether there are any unauthorized attempts to enumerate or escalate privileges associated with compromised AWS credentials. This involves analyzing CloudTrail logs for IAM-related API calls, such as List* and Get* actions, to identify suspicious attempts to gather information about permissions and access rights associated with compromised credentials.

Effective threat hunting and security incident response depends on adequate logging, as described in the AWS Security Incident Response Guide. If you have the proper logs and can query them, you can respond more effectively to security events. If a security event occurs, you can use various log sources to validate what occurred and understand the scope.

Microsoft Sentinel and AWS CloudTrail

Integrating Microsoft Sentinel and AWS CloudTrail strengthens the security posture of cloud environments and enhances the capabilities for proactive threat hunting.

Integration Benefits: A Unified Front Against Threats

The integration of Microsoft Sentinel, a cloud-native SIEM solution, with AWS CloudTrail, a comprehensive auditing and logging service, creates a synergy that offers several key benefits:

Centralized Visibility: By aggregating logs from Azure and AWS environments into a single platform, organizations gain centralized visibility into their entire cloud infrastructure. This unified view allows security teams to correlate events across platforms and identify potential threats more efficiently.

Enhanced Analytics: Leveraging Sentinel’s advanced analytics capabilities, organizations can analyze CloudTrail logs alongside other security data sources to uncover patterns indicative of malicious activity. This holistic approach to threat analysis provides deeper insights into potential threats and improves threat detection accuracy.

Automated Response: Sentinel’s automation capabilities enable organizations to automate response actions based on predefined playbooks. By integrating with CloudTrail, Sentinel can trigger response actions in real time, allowing security teams to respond swiftly to emerging threats and contain incidents before they escalate.

Continuous Monitoring: CloudTrail’s ability to capture detailed information about API calls made within AWS environments facilitates continuous monitoring of user activity and resource usage. When integrated with Sentinel, this continuous monitoring becomes even more robust, enabling organizations to detect and respond to suspicious behavior proactively.

Empowering Proactive Threat Hunting

Proactive threat hunting involves actively searching for indications of compromise within an environment rather than waiting for alerts to trigger. The integration of Microsoft Sentinel and AWS CloudTrail provides security teams with the tools and capabilities needed to conduct proactive threat hunting effectively:

Correlation of Events: By correlating CloudTrail logs with other security data sources, such as Azure Activity Logs or firewall logs, Sentinel enables security teams to identify anomalies and potential indicators of compromise that may go unnoticed when analyzed in isolation.

Behavioral Analysis: Sentinel’s AI and machine learning capabilities can analyze historical CloudTrail logs to establish baseline behavior patterns for users and resources. Deviations from these baselines can indicate potentially malicious activity, prompting further investigation.

Hunting Queries: Sentinel’s flexible query language allows security analysts to craft custom hunting queries tailored to specific threat scenarios or use cases. By querying CloudTrail logs for suspicious activity indicators, analysts can uncover hidden threats and proactively mitigate risks.

Threat Intelligence Integration: Sentinel integrates with external threat intelligence feeds, enabling security teams to enrich CloudTrail logs with contextual information about known threats and adversaries. This enrichment enhances the effectiveness of threat-hunting activities by providing additional context for investigation.

This connector allows you to ingest AWS service logs, collected in AWS S3 buckets, to Microsoft Sentinel. The currently supported data types are:

  • AWS CloudTrail
  • VPC Flow Logs
  • AWS GuardDuty
  • AWSCloudWatch

Brute forcing AWS IAM User

Because I could not show a production environment with massive findings and information on identity attacks (spray, brute force, enumeration, etc.), I created a Python script to simulate and brute-force the AWS users and show what is happening in the logs. This script was built to brute-force users who do not exist in the AWS directory.

The main goal is to simulate brute force non-existent AWS users and break down the message: “HIDDEN_DUE_TO_SECURITY_REASONS.”

The following Python script combines a few libraries, functions, and more.

Here are a few highlights from this Python script:

  • Objective: The script automates login attempts to an AWS account using a brute-force approach, aiming to find valid credentials.
  • Key Components:
    • Argument Parsing: Utilizesargparse to handle command-line arguments, including the AWS Account ID and an option for random username generation.
    • Password Generation: Functions generate passwords meeting typical security criteria and fake real names or usernames using the Faker library.
    • Login Attempt: Sends POST requests to the AWS authentication endpoint with generated credentials to check for successful login.
    • Random URL Request: This function makes GET requests to random URL paths within the AWS base URL, printing response status codes and content.
    • Main Functionality: This function runs a loop for a specified number of attempts, generating credentials, attempting login, and making URL requests.
  • Functionality:
    • Brute-Force Login: Iterates through login attempts, trying different combinations of usernames and passwords.
    • Feedback: Provide feedback on each login attempt, indicating whether it was successful or not.
    • Random Exploration: After a successful login, the user makes random URL requests, potentially to explore AWS resources further.
  • Execution: The script checks if it’s being run as the main module (__name__ == '__main__') and executes the main functionality accordingly.
  • Libraries:
    • Utilizes various Python libraries, such as requests for HTTP requests, string for string manipulation, secrets for secure random number generation, Faker for generating fake data and tqdm for creating progress bars.

The script provides a versatile tool for automated login attempts to AWS accounts, potentially useful for security testing or exploration purposes.

The image below shows the script output with progress and the fake users and passwords.

When the script is completed, the output will be as follows: Login unsuccessful after 250 attempts.

Conversely, the AWS console shows the following information once the script is completed or even after a few attempts.

TIP: Bear in mind that AWS CloudTrail does not count all failed attempts, whether in the lab or production environments.

The AWS Console Side – HIDDEN_DUE_TO_SECURITY_REASONS

AWS CloudTrail records AWS API calls and user activity. CloudTrail events contain detailed information about each call, including:

  • Who made the call (user, role, or service)
  • When the call was made
  • What API was called
  • The parameters used in the call
  • The source IP address of the call
  • The resources affected by the call

This information can be used for various purposes, including security analysis, resource monitoring, and compliance auditing.

The event data in the image you sent is likely a JSON formatted record of an AWS CloudTrail event. Here are some of the common fields you might find in such a record:

  • eventVersion: This specifies the version of the event schema.
  • userIdentity: This section contains information about the user who made the API call, including their user type (e.g., IAMUser) and account ID.
  • eventTime: This is the timestamp of the event.
  • eventSource: This indicates the source of the event, such as signin.amazonaws.com, for a sign-in event.
  • eventName: This field describes the type of event, such as ConsoleLogin.
  • awsRegion: The AWS region where the event occurred.
  • sourceIPAddress: The IP address of the device that made the call.
  • userAgent: Information about the user-agent string of the browser or application that made the call.
  • errorMessage (optional): If the event resulted in an error, this field will contain the error message.
  • requestParameters: This section may contain details about the parameters used in the API call.
  • responseElements: This section may contain details about the response elements of the API call.
  • additionalEventData: This section may contain additional data specific to the event type.
  • eventID: A unique identifier for the event.
  • eventType: This describes the type of CloudTrail event.
  • managementEvent: Specifies whether this is a management event.
  • recipientAccountId: The ID of the recipient account for the event.
  • eventCategory: The category of the event, such as Management.

The images above show the message for “HIDDEN_DUE_TO_SECURITY_REASONS.”

The AWS Console Side – Failed authentication

In addition, ” a good failed authentication” will be different, including additional information, such as the user, the failed message, etc. How should a correct failed message look for failed authentication with an existing user (a correct and real IAM user)? The image below shows the message “errorMessage”: “Failed authentication.”

The json log shows the following important information:

  • “userName”: “test1”
  • “errorMessage”: “Failed authentication”

Message: HIDDEN_DUE_TO_SECURITY_REASONS

It’s important to note that the message “HIDDEN_DUE_TO_SECURITY_REASONS” can be irrelevant in some scenarios and important in many others. On the one hand, this message isn’t a pure security issue, but on the other hand, it can be.

What are the important fields from the CloudTrail log for this message? When discussing and hunting the message: “HIDDEN_DUE_TO_SECURITY_REASONS,” we should check specific fields.

  • “userName”: “HIDDEN_DUE_TO_SECURITY_REASONS.”
  • “errorMessage”: “No username found in supplied account.”
Note: The HIDDEN_DUE_TO_SECURITY_REASONS message can appear on additional actions on AWS.

While checking with AWS about this message (Documents, support, and with their internal security team), I noticed that it is mentioned and explained a few times with different approaches in their documents.

From the “CloudTrail userIdentity element,” it is the message below.

In the Logging user sign-in events document, the description is as follows:

As a security best practice, AWS does not log the entered IAM user name text when the sign-in failure is caused by an incorrect user name. The user name text is masked by the value HIDDEN_DUE_TO_SECURITY_REASONS.

In a nutshell, AWS masks the userName text with HIDDEN_DUE_TO_SECURITY_REASONS to help prevent exposing potentially sensitive information.

TIP: If you’ve got GuardDuty, it can report the finding HIDDEN_DUE_TO_SECURITY_REASONS.

Mitigation

You cannot mitigate this message because if someone wants to run brute force on your AWS users (and AWS account), he can do so without limitation. The hidden message belongs to non-existent users. The risk is very minimal and may be only informative. Still, we want to know if an attacker is running an attack against the AWS account ID.

  • In some scenarios, a threat-hunting team will create a detection rule if someone is sitting on their account and trying to enumerate users and non-existent objects.
  • Bear in mind that the hidden message doesn’t appear only in a brute-force scenario for IAM users. It can also appear on other AWS components, even between internal components.

CloudTrail limitations

You may face some challenges if you attempt to build detections around this message. This is mainly because a significant portion of useful telemetry (basically all of it) is overlooked in these logs.

For this reason, you can use CloudTrail to track the number of calls and their associated IP address but not access their details.

MITRE

The Mitre Att&ck Framework and information for brute force are as follows:

In some cases, Cognito can help in this scenario and improve the visibility.

In conclusion

This article highlights the “HIDDEN_DUE_TO_SECURITY_REASONS” issue in AWS CloudTrail, which masks log details to protect sensitive information but can hinder security investigations. It explains how incorrect usernames in brute-force attacks trigger this message, complicating threat detection. Integrating AWS CloudTrail with Microsoft Sentinel can improve log analysis and threat hunting. A provided Python script demonstrates how logs can obscure important details during security incidents, emphasizing the need for robust logging and monitoring strategies.

Tags:

Discover more from CYBERDOM

Subscribe now to keep reading and get access to the full archive.

Continue reading