Detection & Compromise: Secrets from the AWS Secrets Manager
Public exposure of secrets — privileged credentials used to access resources— can lead to adverse outcomes for organizations. Today, we’ll walk over the Secrets Manager service on the AWS Cloud which is fairly handy when it comes to the creation and management of secrets.
I’ll not be digging into the specifics of “how” the service works; rather we’ll see “how” to detect these secret compromises and “what” to do in such a case — considering it does fall under the customer’s responsibility in the Shared Responsibility model.
Why go for “Secrets”?
Secrets, in essence, offer a gateway to the crown jewels of an organization. Whether it’s credentials to the database, API keys to your accounts, encryption keys, or whatever — they’re potentially used to access privileged environments and are, therefore, lucrative targets for your adversaries.
As a case study, we’ll take Tesla’s infamous crypto-jacking compromise in account. How’d it go down? A publicly exposed Kubernetes administration console with no password. That’s not the end of it — one of the pods or containers included “privileged credentials” to an entirely different cloud environment.
That’s how the attackers dug in, starting off from a console to retrieving secrets, escalating privileges, and deploying their scripts to complete the objectives of their crypto-jacking operation.
Whew; that’s precisely why Secrets Management is an important aspect of operational security today.
Compromising Secrets
Before we move on to detecting a compromise of secrets on the cloud, let’s first script together potential attack vectors. One thing’s for sure — to access or abuse these secrets, an adversary would have to get access to credentials allowing permissions to AWS Secrets Manager.
For a demonstration, I’ve configured (read: misconfigured) a few services in my AWS account. For brevity’s sake, I’ll also assume there’s a compromised EC2 instance and the ‘adversary’ has gained access to it. Let’s see what can I do with my access.
Let’s start by checking if there’s an Instance Profile (role) assigned to the instance using the metadata service:
So, there’s a role attached to the instance. If I try to list more information about the role — no success. As the error suggests, I don’t have enough permissions to retrieve data from the iam service. Okay, let’s not ring alarms by brute forcing permissions.
(Keeping it short) Let’s try the list-secrets API call to the Secrets Manager service (you can also try to go for services like S3):
Success! I see the role does have (some) permissions to the manager. Let’s dig into this secret by retrieving the value using the get-secret-value API call:
Let’s also parse this JSON into readable credentials (or values of the SecretString key) using jq:
What are the chances that these are valid credentials? I’ve configured a profile with these credentials using the aws configure utility. Let’s retry some of the previous IAM calls which failed on the configured instance profile:
Success! I won’t explore what other permissions are assigned to this user but generally speaking, permissions to use IAM opens up a lot of potential attack vectors to pivot into.
Detecting the Compromise
Now that the offense part of this compromise is complete — let’s discuss detection methodologies. If you’ve got a trail configured in CloudTrail, you can analyze the telemetry from your users, API, or service events to dig into figuring out the root cause of the compromise.
Parsing CloudTrail Data (Athena)
If you’re a fan of Athena, integrate your CloudTrail logs via S3 and parsing them won’t be a difficult task at all. Since this article has spanned a couple weeks, I don’t have the telemetry anymore but I did manage to dump the logs out in CSV. So, I’ll also give a quick brief on how to add CSV data to Athena (since I couldn’t find much on it over the web).
Not interested in learning about Athena or how to parse CTL data using the service? Skip to the next heading: “CloudTrail Log Analysis”
Head over to Athena. Let’s start by setting up an Output Location for the workspace we’re currently working in — which is the location where query results are stored. Create a new bucket for this configuration data, switch to the Settings tab in Athena, and press Manage to populate the Query Result Location field with an S3 buckets location.
Head back into the query editor and let’s create a new table by hovering over the Create option under Tables & Views. Since my CSV sheet is in an S3 bucket, I’ll choose S3 Bucket as my data source. Fill in details of the database (new or existing), table, and the S3 input (the actual bucket’s location ending with the prefix not the file name). Select CSV under Data Format and bulk-add columns. Here’s what I added (I removed the spaces between the column names and converted the column names into title case):
UserName String, AwsAccessKey String, EventTime String, EventSource String, EventName String, AwsRegion String, SourceIpAddress String, UserAgent String, ErrorCode tring, Resources String, RequestId String, EventId String, Read-only String, EventType String, RecipientAccountId String, EventCategory String
You might wonder — why didn’t I specify structs for the Resources or other similar columns? Well, that isn’t a permissible operation in the bulk-addition feature. You can, however, go through the struggle to update the fields format using DDL queries (I won’t).
That’s it. Create the table and you’re good to go — here’s a sample query (change the table name accordingly):
SELECT * from "ctl-csv-data"
CloudTrail Log Analysis
CloudTrail practically logs every API call made by any resource, service, or principal in the AWS account.
Let’s first list down all distinct event sources using the query: SELECT DISTINCT eventsource from "ctl-csv-data" .
I’m interested in the Secrets Manager service so let’s filter on it as the eventSource: SELECT * from "ctl-csv-data" WHERE eventsource='secretsmanager.amazonaws.com' and we see just two users attempting calls against the service — i-0cc...3237 (an instance) and cloud_user .
Now the obvious suspect here is the instance itself generating 12 requests (using the configured instance profile) and all those API calls are GetSecretValue. Does this suggest a compromise? Not at all. It could very well be an administrator or a CRON script requesting credentials from the Secrets service — this is where an analyst has to add in “context” to the alarming results.
If we cater in the time to these requests, the pattern does seem suspicious as all requests are generated in a span of 4 minutes. So, where do we pivot from here? We could look at the instance’s activity and see what else has happened at the instance level and if it warrants a closer look.
Six (unique) events in total in our fetched logs — GetSecretValue would’ve been followed by a KMS Decrypt call (as the secret could’ve been encrypted using a service key). There are denied events on the IAM service as well which is interesting.
So, one thing’s for sure — since the Decrypt calls don’t fail, whoever executed the calls does have access to the secrets as well. A chat with the administrator and you get to know that there are ‘access credentials’ for the myadministratoruser inthe secrets! Whoops!
The adversary could’ve logged in using the stolen valid credentials. To pivot, let’s filter the data against the user. Now we see just two calls against GetInstanceProfile. A slip here from the adversary was to configure these credentials on a machine exposing the IP address in the sourceIpAddress field. Had they been configured on the instance, the source IP would’ve pointed to the instance again.
You can further pivot into the activity against the IP address in CloudTrail logs or perhaps the instance’s OS or service (web server) logs to find evidence of the initial compromise.
Other (Potential) Pivots
Secrets Manager, like most services, allows cross-account access to resources created within it. For that, the adversary would have to update the resource policy of the compromised secret and the identity policy of the IAM user(s). Since we saw compromise of a user with access to the IAM service, the adversary could’ve modified both policies and assigned cross-account access for hidden access.
Mind you — this would also require updates to the key policy of the KMS key so the secret could be decrypted.
Why would that be needed if the secret has already been compromised? Secrets Manager allows auto-rotation of credentials of certain secret types using Lambda functions. Persistence in such cases will help the adversary get the updated secrets (or credentials).
If your analysts don’t monitor for activities in unused regions, the Secret Replication feature could also be used wherein the adversary would replicate the secrets in the specified region without raising alarms. Keep your alerts generic to all regions and especially monitor unused regions for suspicious activity.
In case of an adversary attempting to disrupt your services and delete resources (e.g., secrets), the Secrets Manager places a mandatory (minimum) wait period of 7 days after which the key is actually deleted. Regardless, monitoring for such API calls can yield good results.
Here’s a good read from the AWS documentation on the Secrets Manager on Permission Policies outlining some of the API calls from the service and their individual implications.
Handling Compromised Credentials
Great, you’ve detected a secret compromise. What next? Trigger your Incident Response process! As part of it, you can take several actions:
Revoke & Rotate
Classic. Mind you — revoking and rotating credentials isn’t going to do much if the root cause of the compromise is still soundly letting intruders in. Fix the root-cause, closely monitor the intruder, and once you’re certain you’ve tracked all activity and that all open holes have been patched, hit the kill switch.
Revoking credentials is possible for individual users under the Security Credentials tab. You can either delete the key entirely, monitor it for usage, or make it inactive.
If you’d like to go a step further and disable the user entirely, that’s also possible under the same tab. You can also rotate the credentials for the user yourself or force a password reset on the new login.
What if it’s an instance profile, like we saw in our CloudTrail telemetry? Well, you can revoke sessions for roles as well. Under the Revoke Session tab in the role, press Revoke Active Sessions and that’d terminate all open sessions (new sessions can still be created!).
Tune Overly-permissive Policies
Let’s be clear; the blast radius of the compromise could’ve been a lot smaller if only the permissions assigned to the instance profile or the user were limited and instead of a service key, a user-based KMS key would’ve been used with a strict key policy.
Tune the policies, remove all unneeded permissions, and if you must allow access to critical APIs, then filter on the resources as well (e.g., only the secret which is needed by the script or application running on the instance rather than all secrets).
Delete Resources
Identified the users or roles behind the compromise? Time to check what activity or operations they might’ve performed on your AWS cloud and list all such unauthorized resources. Once you’re certain all such resources have been identified, it’s time to delete or terminate the resources so your bill doesn’t surprise you.
You might also wish to isolate some of these resources (e.g. instances) to perform forensics, study the compromise, and improve detection or prevention controls based on your findings.
Acknowledgements
Lina Lau’s articles on Detection and Compromise on the Azure Cloud have been a great source of inspiration for kick-starting this series on Detecting Compromises on the AWS Cloud. Give her content a read here!
Did you like reading this? Leave a comment on what else would you like to read or improve in this article — I’m always open for constructive criticism!
