Who Holds the Keys to the Kingdom?

Take a closer look at sensitive AWS Resources:secret strings and keys used in AWS, learn about the Resources and access control mechanisms relevant to them, delve into the challenges of tracking the permissions granted to them, and see ways in which automating analysis of environment configuration and logs in AWS can help.

What are the keys to the kingdom?

Nearly all cloud environments store sensitive or important data. It requires multiple levels of protection including encryption as well as strict access control, which could be due to regulatory requirements or organizational policies.

In addition to private files and records, some of the most sensitive artifacts stored in the cloud are credentials for both external and in-house services. When an application running on a cloud platform requires a third-party software service or database, it often uses credentials in the form of an API token or access key. These are highly sensitive strings, as they are usually all that’s required to perform actions on behalf of the organization. This means a malicious actor compromising them could perform actions while posing as the application, access sensitive information the application uses, tamper with information or even cause a denial of service incident. On top of their sensitivity, these strings also must be very easily accessible and simple to use programmatically.

Iaas and PaaS vendors have various solutions for this issue. AWS offers services to encrypt data and handle secrets effectively and quite simply. These services work well. In fact, they work so well that even though they literally hold the keys to the kingdom, we tend to forget that the secrets they hold are in fact AWS resources - not unlike S3 buckets or EC2 instances - and access control to them is managed in the exact same way.

In one of our past blog posts we discussed the challenges of managing identities and entitlements of AWS IAM users and roles. The challenge of merely understanding which entity has access to what resource is massive - virtually impossible for an analyst or even a team of analysts relying only on the tools supplied by AWS. It goes without saying that cybersecurity-wise, not all AWS resources are made equal. As their names suggest, “secrets” and encryption keys in our environment are some of the most sensitive resources out there.

By neglecting to control the access to these resources, we might be enabling a backdoor to our most treasured assets, making the cryptography-forged, secured front door moot.

In this post, we will explore the ins and outs of access control for secrets in AWS, and how to significantly reduce the risks.

AWS secret-keeping services

AWS provides three main mechanisms for the management of such sensitive information - key management service (KMS), secrets manager and the parameter store of the simple systems manager (SSM).

We will give a brief overview of how these services are used as background - but this is of course just the tip of the iceberg when it comes to understanding them. We highly recommend you go through the AWS documentation to become familiar with how to use them.

Key Management Service

KMS is the foundation of encryption used in AWS. It allows you to create and manage keys utilized for cryptographic functions. Many AWS services leverage KMS in order to perform cryptographic actions.

The primary resources of AWS KMS are customer master keys (CMKs), which can be used to encrypt or decrypt information. Access to CMKs is configured based on a Key policy, or optionally, on IAM policies.

KMS guarantees that the confidential parts of CMKs required for decryption / signing actions are stored safely and never leave KMS in plaintext form. This means that only an entity with access to the CMKs based on its key policy and / or IAM policy can access the secrets encrypted by it. That’s why access to these resources is so sensitive.

It’s important to distinguish between two types of CMKs:

• Customer managed CMKs - They are (as stated in the AWS docs) the “CMKs in your AWS account that you create, own, and manage.” That is, CMKs you can create for performing cryptographic functions for any kind of proprietary needs you may have (e.g. - encrypting files). You have complete control over them and you, specifically, manage their Key Policy.
• AWS managed CMKs - According to AWS, these are “CMKs in your account that are created, managed, and used on your behalf by an AWS service that is integrated with AWS KMS.” This means they are keys created and used automatically when you use AWS services that require cryptographic actions for their operation. (Secrets Manager described in the next section is a good example). Since these CMKs are managed by AWS, you can’t configure their key policy.

Secrets Manager

Previously, we spoke about storing sensitive values which need to be accessed at runtime by applications, such as credentials for various services and assets such as databases. Since this is a very common requirement for applications everywhere, AWS has a service called Secrets Manager which enables you to “store, distribute and rotate credentials securely”.

Using Secrets Manager allows you to create an AWS resource called a “Secret” - a sensitive string, stored in an encrypted format. AWS allows you to configure the secret’s rotation process (changing the value of the secret and having it still function properly as credentials) using a lambda function and can even create a lambda function that does it for you if the secret in question is credentials for a service it knows (e.g. an RDS database or a Redshift cluster).

Systems manager’s parameter store

The systems manager’s parameter store is a service older than Secrets Manager that allows managing resources called “Parameters,” which are essentially key-value pairs available for use by applications. It wasn’t designed only to hold secrets such as API keys or DB credentials, but also holds non-sensitive values such as configuration information that require no encryption.

When creating a sensitive parameter requiring encryption, you can simply make it a “SecureString” which configures AWS to encrypt it with a KMS key of your choice (either a dedicated AWS managed CMK or one of the CMKs managed by you).

It’s not as convenient as Secrets Manager, specifically if the credentials are for a service where Secrets Manager can provide you with an out-of-the-box lambda function for rotation. However, mostly due to cost considerations, it’s still a popular alternative to Secrets Manager.

It’s all about the resources

Before we move forward, let’s focus on the fact that CMKs, secrets managed by Secrets Manager, and Parameters in the SSM parameter store are all AWS resources. Therefore, access to them is managed in a similar way to how access to any AWS resource is managed.

In the next sections, we’ll take you through the basics of access control for these specific resources to allow you to more easily assess who has access to them.

Reviewing access entitlements for secrets and keys

In this section, we will explore the specific things to look out for when it comes to managing access to the resources described in the previous section. We will review how it’s specified in IAM policies, resource-based policies and key policies / grants for KMS CMKs.

It should be noted that many other aspects of access control which apply to AWS resources, such as service control policies and permission boundaries, also apply here; however, as they don’t apply uniquely to the resources this article focuses on, we’ll come back to them later. Keep in mind this is not a complete list of locations in which to look, but it’s a good start.

First, let’s note the sensitive actions to look for when it comes to each of the resources we described previously. Out of the actions available for each of the resources, the ones we consider as sensitive actions are the ones that allow you to access secured information. In the case of a KMS CMK, it might be decrypting ciphertext encrypted by it, or performing other sensitive cryptographic functions like signing.

Table 1 - Sensitive Actions for Secrets / Parameters / KMS CMKs

A closer look at IAM policies

Like most resources, access control to the resources we previously described can be managed using IAM policies.

To locate principals who have been granted access to these actions, you need to find the customer managed AWS policies you’ve created with these permissions, and check the policy usage to see where they are attached. This enables you to find users, roles and groups which have access to resources through the policy.

In addition, you should similarly review AWS managed policies which allow unrestricted access to secrets - for example, AdministratorAccess and SecretsManagerReadWrite.

The latter allows users full access to your Secrets Manager secrets and is, ironically, the one you should look out for as it’s more prone to excessive access. While most administrators of AWS environments realize AdministratorAccess is a policy to be assigned with extreme caution, “SecretsManagerReadWrite” is sometimes perceived as more innocent and much less harmful. Though it is in many ways, it nevertheless allows access to strings which could be used to read and impact some of the most sensitive assets an enterprise has, as it allows full access to all Secrets Manager actions for all resources (taken from the current version 3 of the policy):

Another AWS managed policy, AmazonConnectFullAccess (in its current version 2), allows kms:CreateGrant to "*".

Other AWS managed policies in their current versions allow access to SecuredString values in the parameter store via ssm:GetParameter and / or ssm:GetParameters, for example:

• AmazonSSMFullAccess (version 4)
• AmazonSSMReadOnlyAccess (version 1)
• AmazonSSMMaintenanceWindowRole (version 3)
• AmazonSSMManagedInstanceCore (version 2)
• AmazonSSMServiceRolePolicy (version 9)

It should be noted that AWS managed policies are frequently changed, and it’s a best practice not to use them as a long-term permission-granting mechanism. Please note we’ve only given some examples here. Even if we did list all the AWS managed policies which currently provide access to these resources - AWS can always change other policies to do so as well - so make sure you’re alert!

Next, we will explain how to keep a closer eye on resource-based policies in which you should also look for the sensitive actions we previously described. Inline policies are obviously also significant to track; however, as this is a much more difficult task, they are referred to in the next section.

Key takeaway: Review AWS managed policies and customer managed policies (and do your best to review inline policies) allowing the specified actions from table 1.

KMS access control

KMS CMKs have their own resource-based access control mechanisms.

Key policy basics

A KMS CMK has a “key policy” which is basically KMS’s version of a resource-based policy. Unlike a regular resource-based policy, it’s mandatory for the CMK (the resource) to have one, and in order for IAM policies to be effective for the CMK, the key policy must include a declaration such as this:

The KMS console allows you to entitle key administrators and key users. Both will have the ability to grant other principals with access to the key (to perform any function) using kms:CreateGrant, while key users will also be able to use kms:Decrypt in order to do so without an additional grant.

In addition, key policies can allow access to external entities (most commonly - other AWS Accounts), which we will review in more detail later.

For AWS-managed CMKs, it’s better to focus your effort on the permissions granted to the services utilizing these keys, which is outside the scope of this article. For customer-managed CMKs, however, as you control their configuration, it’s up to you to make sure it’s not configured in a way which allows excessive permissions.

Key takeaway: Review the key policy for each KMS CMK looking for the actions from table 1 and / or a statement enabling IAM policies on it and / or access to outside accounts.

Keeping tabs on grants

You can also enable access to KMS CMKs by creating a grant for a principal allowing it to perform any action on the CMK. A grant is a resource-based access control mechanism (in addition to the key policy) which can be easily granted and revoked programmatically to delegate the use of a KMS CMK. As the AWS docs state: “Because grants can be very specific, and are easy to create and revoke, they are often used to provide temporary permissions or more granular permissions.” As we know, however, mistakes tend to happen and what’s given temporarily and not revoked properly (either due to faulty programming or a runtime exception) might still be in effect.

Fortunately, AWS provides a simple CLI function called list-grants which you can use to list all the current grants for a key. Its usage is pretty straightforward, and it allows you to detect all the principals and the actions they are currently allowed to perform on the KMS CMK.

Key takeaway: Use the list-grants CLI command to keep track of the grants given to a KMS CMK.

Monitor external access to KMS CMKs

You should also note that the KMS console allows you to configure the key policy to enable access for outside accounts, which simply turns the root user of those accounts into a key user for the key (with the entitlements mentioned above). Once this trust has been established, sensitive permissions can be delegated by the root user of any of these outside accounts to other principals within the outside account.

With Access Analyzer, AWS allows you to monitor outside access granted to KMS. Access Analyzer lets you know when external access is granted to resources of certain supported types (more information is available here) and luckily, KMS CMKs are among them. You can manually review Access Analyzer periodically, or alternatively, if you’re using CloudWatch to monitor various events in your environment, you can configure EventBridge to monitor Access Analyzer and have events about KMS keys go to a specific log group in CloudWatch.

The way to do it is as such: after enabling Access Analyzer, you create a log group in CloudWatch (e.g. aws/events/secretsmonitoring), and then create a rule in EventBridge using the following pattern:

The EventBridge console should look like this when configuring the rule:

Figure 1 - Creating a Rule in EventBridge to send events from Access Analyzer to CloudWatch

Under “Select Targets,” choose a target to be “CloudWatch log group” and the log group to be the new one we created for this purpose:

Figure 2 - Selecting CloudWatch Log Group as the Target for the EventBridge Rule

Once you create and enable this rule, it will track external access to any KMS CMK and will alert about it to the log group in CloudWatch.

Key takeaway: Keep a close eye on outside accounts that have access to a KMS CMK. More preferably, configure Access Analyzer, EventBridge and CloudWatch to monitor events of such access being granted.

Secrets Manager resource-based policies

Secrets Manager secrets are resources for which resource-based-policies are enabled. That means that access for any principal from any account can be granted to them using resource-based policies.

Unlike KMS CMKs, external access to Secrets Manager secrets is currently not tracked by Access Analyzer so the best you can do is review their resource-based policies, either manually or by the use of a script / analysis technology, to figure out who exactly has access to them.

Key takeaway: Review access granted to Secrets Manager secrets using their resource based policy.

Review roles trust relationships

Finally, one significant move that can help with the mapping of entities that can gain access to your sensitive resources is to first map the trust relationship on the roles you found which are assigned a policy allowing access to them / granted access to them via a resource-based mechanism.

If the trust relationship of a role created in an account allows the root user of an account to assume it, any user or role in that account would be able to assume it if it is assigned an IAM policy which delegates the sts:AssumeRole Action to it on that role. This can be limited by changing the trust policy by either setting a more specific principal (or principals) other than the root and / or setting conditions that limit which principals can assume it. In order for you to fully understand who can assume the roles with access to your sensitive resources, review the trust relationships on said roles.

You should note that the trust relationship of a role may allow various entities to assume it, including an outside AWS account, an identity provider or an AWS Service within the account. This could open a path for entities who could be granted access to the resources through various scenarios, for example, EC2 as an AWS service is allowed to assume a role and an IAM User is granted access to EC2 (this is a very common scenario when it comes to developers).

Key takeaway: Review the trust relationships of the roles allowed access to the resources to track which principals can assume them.

Putting it together

To clarify, here’s a diagram of the objects we’ve described and the relationships we’ve reviewed between them:

Figure 3 - Relationship map of the access control objects we’ve reviewed

“Identity Provider” and “AWS Account” are marked red since they allow access via assumption of a role to external entities. “IAM Policy” is a generalization for AWS-managed, customer-managed and inline policies.

Note that resource based policies are indicated in the diagram in rectangles. When it comes to assuming a role, an IAM policy can only delegate what the role’s trust relationships has allowed.

Any combination of relationships could allow an entity to gain access to the resource - so if you’re looking to review it manually, follow all possible paths that could eventually lead to a resource. Finally, as we said before, remember what we’ve presented in this article is not a complete description of how access to these resources is managed. So the situation might even get more complex.

Where do we go from here?

If what you’ve read here has made you think this process is complicated, I’m sad to inform you that the reality is much more grim.

However, there’s no reason to lose hope. Next, we’ll look at the difficulties in implementing this process and how you can use state of the art automated analysis technology to mitigate them.

As we mentioned, in figure 3 above, “Identity provider” and “AWS Account” are marked red since they allow external entities access via assumption of a role. “IAM policy” is a generalization for AWS-managed, customer-managed and inline policies. Note that resource-based policies are indicated in the diagram in rectangles. When it comes to assuming a role, an IAM policy can only delegate what the role’s trust relationships allows.

Any combination of relationships leading to a resource could allow an entity to gain access to it, so if you’re looking to review it manually, you must follow all possible paths that could eventually lead to a resource, which could be a staggering number. Also remember that what we presented here and in the last post is not a complete description of how access to the resources is managed, so the reality could get a lot more complicated.

If you were tempted to think it’s simple to review your AWS environment to control access to the resources we discussed, we hope that by now you’ve realized that’s hardly the case. Even though we can define how it theoretically can be done, real life poses many challenges which are extremely difficult to mitigate. Let’s take a closer look at these challenges and present a way in which automated analysis of configuration and logs could help you mitigate them.

Manual access control auditing limitations

role assumption tracking

Let’s say that while reviewing an IAM policy for an SSM parameter, or a resource policy for a Secrets Manager secret, you realize it grants an IAM role access to the resource in question. If the trust relationship is not defined in a very specific way, many other IAM roles and IAM users could potentially assume that role, and by doing so, use the permissions assigned to it.

This, in fact, can create a major gap in your analysis. To bridge it, you would have to (among other things) review IAM and resource-based policies on many principals looking for the “sts:AssumeRole” action for said role and look for all principals that can pass the role using “iam:PassRole” to an AWS service included in the trust relationship of the role. This is nearly impossible to do manually and extremely difficult to manage if done programmatically.

Inline policy tracking

As we mentioned before, inline policies defined for IAM principals are extremely difficult to track and they can grant permissions just as easily and effectively as IAM policies.

Digest the information and respond to it

The amount of relevant information that should be compiled from the review activity we described in this article could potentially be mind-boggling. You have to gather information about all the KMS CMKs, Secrets Manager secrets and SSM secured parameters. For each of those, you also need information about the IAM principals that have access. In the case they are groups or roles, then you have to also consider the information about the IAM principals that are included in them or can assume them, respectively.

While collecting the information is a difficult task in its own right, it is not possible to manually organize it in a way to generate meaningful insights and respond accordingly.

Real-time auditing to rightsize permissions

Another task that simply can’t be completed using manual tools and / or by solely using the AWS console, is auditing the actual usage of permissions granted to use the secrets. It’s very hard to determine whether a permission granted to a user or a role is justifiably granted and the principal is required to use it on a regular basis. To do so, you must audit the usage of the resources and make sure that any permission granted to a principal is, in fact, in use.

Ensuring full coverage

Finally, no matter how much time and effort you spend manually combing through configurations and logs, you will never have the assurance that you’ve covered everything and made all the relevant considerations. Even if you did, how will you know it won’t be changed the next day by someone else?

The assurance of governing control to your secrets is, in fact, quite elusive, and this assurance is the main reason why we chose to go on this journey. Isn’t it?

The case for automatic analysis

Luckily, software for automatic analysis of cloud environment configurations and logs can mitigate the limitations of manual analysis we described. Specifically, Tenable Cloud Security can help you gain visibility to the access that entities have to your secrets, assess risks which apply to said secrets, and even remediate these risks to significantly improve your security posture, all with very little effort.

First, let’s look at a simulated example of how complicated access control can actually get:

Figure 4 - Bird’s-eye view of an access graph for an example KMS key

Here we see a graph illustrating all the entities who have access to a KMS key. On the right, we see an indication of where the resource (the KMS CMK) is. To the left, we see the IAM policies, roles and the users that are allowed access to it. They are connected to one another so we can understand the paths that lead principals to gain access to the CMK (e.g. a federated user can assume a role which is assigned a policy that allows using the key for decryption). Just imagine having to manually comb through policies and resource configurations to eventually compile this graph; it’s a paralyzing thought. In fact, even going over this simple diagram might take a while, so let’s filter it based on the permissions we really care about. We want to focus on “kms:Decrypt”, “kms:CreateGrant” and “kms:Sign”:

Figure 5 - Filtering the access graph to track only the most sensitive actions

After selecting specific permissions, the graph is automatically filtered and now looks like this:

Figure 6 - The filtered access graph

As this graph is much easier to examine, we easily notice an indication of two IAM roles with excessive access to the “AdministratorAccess” policy:

Figure 7 - Two IAM roles have excessive permissions to the “AdministratorAccess” policy

Zooming in allows us to see that it’s an external account with two roles with an “AdministrativeAccess” policy attached to them, one of which is the instance profile for an Elastic Beanstalk Environment. The external account is allowed access via the key policy of the key. This is obviously excessive. If a malicious actor were to leverage a vulnerability that allows it to assume this role, the KMS CMK could be unnecessarily compromised (and in this case - much more than the KMS CMK).

Figure 8 - Zooming in on the IAM roles with the “AdministratorAccess” policy attached

Additionally, Tenable also does the legwork for you by automatically querying and presenting the grants currently available for each KMS CMK:

Figure 9 - Information on a KMS key including current grants

Finally, Tenable goes the extra mile of auditing the logs to review the actual access and usage of resources by entities so it can provide a more exact analysis of which of the permissions are, in fact, excessive.

When we choose a specific role or user, or a group of users / roles which are clustered on the graph, we highlight the path through which they have access to the resource and can better understand whether all, some or none the permissions are excessive (with a red, yellow or black line making the connection, respectively).

For example, if we look at a group of four IAM users who are clustered together and assigned to the “AdministratorAccess” policy, we can see it provides permissions to the KMS CMK, all of which are excessive permissions.

Figure 10 - 4 IAM users with all excessive permissions to the KMS CMK

Right next to it, there’s a clustered group of IAM roles which also have access to the KMS CMK through the exact same policy. If we select it, we can see that its permissions are marked with yellow, which implies that only some of the permissions granted by the policy are, in fact, excessive.

Figure 11 - IAM roles with some excessive permissions to the KMS CMK

This distinction between different principals accessing the exact same policy is possible because Tenable is able to track all the activity of the principals and assess exactly which permissions are needed and which are not. We will see a more detailed demonstration of this ability in our next example.

When we look at a principal in Tenable, we also see and understand risks the system is exposed to due to the excessive permissions assigned to it. In the example below, we see an IAM role called “ProjectsSecretsReader” which Tenable indicates has permissions to 10 services while it only uses two of them. Not only that, but within those two services it only uses some of the permissions it has to them. Let’s see how deeply we can dive into this with Tenable.


Figure 12 - Tenable indicating a verbal plain language risk for a role

In Figure 11, we can examine the risk and analyze logs showing activity by principals assuming this role. We see that out of the 20 secrets the “SecretsManagerReadWrite” policy (AWS managed policy) has granted them access to, they’ve only accessed four secrets, and out of the 19 permissions they have for each, they only use five. All the rest are excessive.

Figure 13 - Specific indication of excessive permissions for secrets

Tenable provides details of exactly which secrets can be accessed and which permissions should be removed. But that’s not the most interesting part. Tenable takes you to the natural next step of remediating this situation and provides a step-by-step guide explaining how to do it.

Figure 14 - Step-by-step guide for remediating an excessive permissions scenario

As you may have noticed, the first step instructing you to create a new policy to replace “SecretsManagerReadWrite” has action buttons on the right. These buttons give you access to a readymade policy to fulfill the step using a simple copy-paste into the AWS console. You also have the option to get a Terraform or CloudFormation script to do it even faster if you practice managing your infrastructure as code.

Figure 15 - Example of an Tenable policy suggestion to right-size permissions for a role

Figure 15 shows an example of an auto-generated document for creating a policy which provides access to the four secrets the role consumes. For each secret, we see only the actions the role applies to it based on the log analysis. It’s that granular.

In addition, you can also view the differences between the current policy and the one recommended to you (see Figure 16):

Figure 16 - The difference between an existing and a recommended policy

Finally, automate the remediation procedure with Tenable’s step-by-step wizard by simply pressing the “Automate” button in the remediation panel:

Figure 17 - Automating the remediation process directly from the Tenable console

In addition to indicating issues, Tenable sets you on the path to remediate them with as little effort as possible and as much confidence as possible, to achieve the best result possible.

Ready to find out more?

Hopefully this document has helped you understand how important and challenging access control for sensitive resources is in AWS. If you want to learn more about how Tenable’s solution can help your organization mitigate this issue, please contact us for a demo.