Keep Your IAM Users Close, Keep Your Third Parties Even Closer

Technical and legal controls that you take for granted when managing cybersecurity policy within your organization are usually out of reach when a third party is compromised.

Many scenarios call for providing access to your AWS environment by services and users outside your organization. This is the case for entities not represented by IAM users or IAM Roles Anywhere in an AWS account that your organization controls and aren’t managed through federation by an identity provider such as Okta or OneLogin. Typically, these entities are software services requiring access ranging from read-only to granularly defined actions – and even administrative IAM permissions to specific or considerable parts of the environment.

AWS provides two main mechanisms for handling such a situation:

• Programmatic access, for authenticating an IAM user using an AWS IAM access key. This approach enables external access to any entity with which these secrets are shared.
• An alternative mechanism is the AWS cross account role, which allows external accounts to assume (and delegate the ability to assume) a role within an AWS account and use the permissions granted that role.

From a security perspective, allowing access to third parties opens up AWS accounts to significant risks if not managed properly. A very cool table-top exercise recently tweeted by Matt Fuller illustrates how an attacker can use a mechanism allowing third-party access to create a beachhead to your environment.

In this post we will review how third-party access is configured and controlled, and how it can “go sideways” by creating undue risk to your organization. We’ll also explore best practices for protecting against third-party access risk and how automated analysis can help.

Third-party access: How is it done?

The most straightforward way to give an entity – including a third party – access to perform actions and retrieve information from your AWS account is to do so programmatically. You do so by creating an IAM user and enabling the use of an access key:

Figure 1 - Configuring an access key for a dedicated or third-party IAM user

These credentials can be used for a variety of clients and use cases by just about anyone. Some, but not all, third parties support this method. In any case, AWS advocates an alternative approach: letting the third party assume an IAM role that considers the third party’s AWS account to be a trusted entity. You configure this fairly easily by creating a new role and specifying that it be trusted by an external AWS account:

Figure 2 - Configuring an IAM role for external access

Note that in Figure 2 we’ve also checked the External ID option; this is a string coordinated between the AWS and third-party account that restricts access to those using the role. External ID is a very important security feature that we will delve into later. Find more information about giving access to AWS accounts owned by third parties in the AWS documentation.

Creating an IAM role for external access eventually configures a trust relationship that looks like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<3RD_PARTY_ACCOUNT_ID>:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "<EXTERNAL_ID>"
        }
      }
    }
  ]
}

Listing “Principal” as the root user is a little misleading: one might be lured into thinking that only the root user can assume the role and use the IAM permissions granted it. However, assigning the principal as root user allows the account to delegate the assumption of the role to any other principal within the external account.

Using the AWS Security Token Service (STS), the mechanism works by allowing the trusted entity to receive temporary credentials that the third party will use to perform actions. The fact that the credentials are temporary is a security benefit. In addition, unlike access and secret keys, which can be held insecurely and fall into the wrong hands, no secret sharing with the third party takes place.

To work with the role, the third party is required to get: the role’s ARN, the External ID (if used, as recommended) and, of course, the AWS Account ID. None of these values is considered a secret, although they shouldn’t be advertised anywhere public. That secrets are not needed to configure the access to the AWS account is itself a security issue that we will also look at later.

Why third-party access is sensitive

By definition, allowing an external entity to access resources is an extremely sensitive action. Granting access to a third party in the context of an AWS account is no different. Let’s review some reasons why.

Delegating control

As mentioned, when you are setting up a trust relationship with an account, the root user is not the only user allowed to assume the role that grants ]IAM permissions. Allowing the root user to assume the role means that an IAM policy can be configured in that external account to allow any user or role to assume the role. This can happen with no further allowance from the AWS account that allowed the delegation to the root user of the external account.

A much better practice is to allow only a specific user or role to assume the role. That said, the control still remains in the hands of the person managing the external account. Even if only a specific role in the external account can assume the role in your AWS account, the person managing the external account can allow anyone within the external account to assume it, thus giving them access to it. Even if you only allow a specific user to have access, programmatic credentials enable access that can be shared with anyone, as noted earlier.

Third parties are (also) prone to being compromised

The SolarWinds breach showed how devastating a compromise of a vendor can be to a whole roster of clients. While that hack has mostly on-prem implications, we’ve shown how it can also impact an affected organization’s cloud infrastructure.

The point is that whether it’s an external bad actor that gains control of a third party or an insider threat, third parties are, like your organization, prone to getting hacked. The real trouble is that, in the case of third parties, the tools at your disposal to prevent attack via a third party – whether detecting it or handling it once you are aware of it – are much more limited. Technical and legal controls that you take for granted when managing cybersecurity policy within your organization are usually out of reach when a third party is the source of the compromise. For this reason, being extra careful with the AWS IAM permissions granted third parties in the first place is especially important.

Best practices when allowing third-party access

Build in least privilege

One of the most crucial steps in allowing third-party access to your environment is the configuration of the policy that the third-party will use and the permissions that the policy will grant it. When configuring policy, always use extreme caution and never take the vendor’s recommended policy for granted without examining it thoroughly, and understanding the extent of permissions it grants and exactly why each part is necessary. Even if just part of a policy is extreme yet necessary for using the service and you can’t think of an obvious way to remove it and still use the service, at least be aware of the situation so you can perhaps mitigate it using one of the other controls discussed here.

Ideally, you want to provide a third party only the permission it absolutely needs to perform its job, and nothing more. A good example is when configuring permissions for services performing specific actions on your system, such as Skeddly. Skeddly enables you to schedule actions triggered externally, such as invoking a Lambda function or starting an EC2 instance at a specific time or periodically. By nature, Skeddly requires that you configure only the action it performs; consequently, you can easily tailor a policy that is effective for these actions only. Skeddly even offers an IAM policy generator for this very reason.

Unfortunately, on third-party platforms that perform actions that are more extensive and not confined to specific services, it is much harder to define such a specific policy. And be aware that some vendors have hidden caveats in their “recommended policy.” For example, below is an actual policy, among many, that is generated by a CloudFormation script that a certain vendor provides. The automatic process at which the access is configured is quite impressive, yet looking closely at the policy reveals that the third- party is using a role that receives access to “kms:Decrypt” and “kms:CreateGrant” on “*”. This means that the role, unless restricted otherwise, can gain access to the plain text of all the data encrypted using customer managed keys in KMS – access that is extremely sensitive.

{
   "Statement": [
       {
           "Effect": "Allow",
           "Action": [
               "kms:Decrypt",
               "kms:ListKeyPolicies",
               "kms:GenerateRandom",
               "kms:ListRetirableGrants",
               "kms:GetKeyPolicy",
               "kms:GenerateDataKeyWithoutPlaintext",
               "kms:ListResourceTags",
               "kms:ReEncryptFrom",
               "kms:ListGrants",
               "kms:ListKeys",
               "kms:Encrypt",
               "kms:ListAliases",
               "kms:GenerateDataKey",
               "kms:CreateAlias",
               "kms:ReEncryptTo",
               "kms:DescribeKey",
               "kms:DeleteAlias",
               "kms:CreateGrant",
               "kms:RevokeGrant"
           ],
           "Resource": "*"
       }
   ]
}

It is not so rare for a “recommended” policy from a vendor to include such extensive permissions. For example, below is a policy from another vendor that instructs configuration that grants the role access to the client’s account. The policy also includes “kms:CreateGrant” on “*” – a permission that can be just as destructive and even more permissive than “kms:Decrypt” as it allows creating grants that can allow additional cryptographic operations, such as signing. The policy even includes “ec2:StopInstances” on “*”. You can easily imagine how that permission might be abused in the wrong hands.

{
   "Version": "2012-10-17",
   "Id": "Excessive3rdPartyRecommendedPolicyAsOf2020-04-17",
   "Statement": [
       {
           "Sid": "Excessive3rdPartyManagement",
           "Effect": "Allow",
           "Action": [
               "autoscaling:Describe*",
               "autoscaling:ResumeProcesses",
               "autoscaling:SuspendProcesses",
               "autoscaling:UpdateAutoScalingGroup",
               "ce:Describe*",
               "ce:Get*",
               "ce:List*",
               "ec2:Describe*",
               "ec2:ModifyInstanceAttribute",
               "ec2:StartInstances",
               "ec2:StopInstances",
               "ecs:Describe*",
               "ecs:List*",
               "ecs:UpdateService",
               "eks:Describe*",
               "eks:List*",
               "eks:UpdateNodegroupConfig",
               "iam:GetUser",
               "redshift:Describe*",
               "redshift:PauseCluster",
               "redshift:ResizeCluster",
               "redshift:ResumeCluster",
               "rds:Describe*",
               "rds:ListTagsForResource",
               "rds:ModifyDBInstance",
               "rds:StartDBCluster",
               "rds:StartDBInstance",
               "rds:StopDBCluster",
               "rds:StopDBInstance",
               "savingsplans:Describe*"
           ],
           "Resource": "*"
       },
       {
           "Sid": "Excessive3rdPartyStartInstanceWithEncryptedBoot",
           "Effect": "Allow",
           "Action": "kms:CreateGrant",
           "Resource": "*"
       },
       {
           "Sid": "Excessive3rdPartyCloudWatchAccess",
           "Effect": "Allow",
           "Action": [
               "cloudwatch:GetMetricStatistics",
               "cloudwatch:ListMetrics"
           ],
           "Resource": "*",
           "Condition": {
               "Bool": {
                   "aws:SecureTransport": "true"
               }
           }
       }
   ]
}

Even if you don’t feel confident altering the vendor’s recommended policy, you may still have controls available to mitigate the possible risks from such policies. However, you must first be aware of the problem before solving it. If you implement a vendor’s recommended policy without reviewing it, you are attaching it “as is” to the IAM role and may be setting yourself up for a world of trouble.

Use a permission boundary

One of the most effective ways to limit what an IAM user or IAM role can do is a permission boundary. Even if you feel compelled to install the vendor’s IAM policy as is, you are fully justified in taking steps to protect what matters to you in your account.

A permission boundary is a set of permissions you can define for a principal such as an IAM user or IAM role that creates a limit (or, as suggested by its name, a boundary) on the actions that the principal can perform. It doesn’t grant permissions to a principal; rather, it limits the potential of what a principal can do. This means that even if you’ve configured an IAM policy, even extremely overprivileged, as requested by the vendor, you have the power to determine which actions that role cannot perform and, more importantly, which resources it cannot access. If policies attached to the principal grant it permissions for something that are not included in the permissions boundary, those actions will not be allowed. Learn more on using an AWS permissions boundary here.

Let’s look at an example: let’s say we want to use services from the vendor that instructed us to install the IAM policy shown above. It could be pretty complex to figure out why the vendor needs “ec2:StopInstance” on “*” but, to get the most bang for the buck we are paying the vendor, we prefer not to remove it. However, some EC2 instances are extremely sensitive, and we don’t want the role the vendor uses to be able to perform any action on them – let alone stop them from running. So we decide to place this permission boundary on the role:

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "ServiceBoundaries",
           "Effect": "Allow",
           "Action":"*",
           "Resource": "*"
       },
       {
           "Sid": "DenyEC2Production",
           "Effect": "Deny",
           "Action": "ec2:*",
           "Resource": "arn:aws:ec2:*:*:instance/i-<INSTANCE_ID>"
       }
   ]
}

This permission boundary means the maximum permissions set that the role can have access to is all the permissions of AWS (as specified in the first statement) minus the ability to perform any “ec2” action on the EC2 with the arn: arn:aws:ec2:*:*:instance/i-<INSTANCE_ID>". This permission boundary does not give any permissions to the principal, but it prevents any policy attached to the principal (including the policy you attach at the instruction of the vendor) from providing it with permissions for that EC2. You can of course configure this permission boundary to also limit access to any sensitive resources and actions you wish.

Require an External ID on third-party roles - the right way

An important mechanism we mentioned earlier is setting an External ID string on the role that the third party assumes through coordination between the AWS account administrator and the third party. The string is not considered a secret but is an important means for dealing with AWS’s “confused deputy” problem.

Simply put, a confused deputy situation is when a malicious entity takes advantage of the fact that it doesn’t need access to values which are considered (and protected as) “secret” to make a third-party perform actions in your AWS account. For example, if all that’s needed to perform such actions is your Account ID (which is not secret but should not be exposed without reason) and the role’s ARN, a bad actor can fairly easily set up a user profile with the third-party vendor and have it perform actions and/or retrieve sensitive information from your AWS account.

An attack of this sort would go like this:

• An attacker sets up a new profile with the third-party vendor that the victim uses, using the victim's Account ID and ARN Role.
• The attacker can now log into this profile with the third-party vendor and use the third-party service posing as the victim.

For example, if the third party is a service that optimizes billing or creates reports about usage in AWS, the attacker might be able to access all the information presented by the vendor and, if possible, even perform changes to the environment.

Alternatively, when an External ID – and specifically one generated by the vendor for each account – is in use and the attacker tries to set up an account with a third-party vendor the victim uses, the attacker will receive a new External ID from the third party that is different from the one the victim received. The attacker will therefore not be able to trigger actions on the victim’s environment unless the new External ID is configured on the IAM role (which it is not).

This use case helps illustrate why using a coordinated, hard-to-guess string provides an important additional layer of security. Learn more about this mechanism and AWS’s confused deputy problem here: AWS documentation.

That said, simply using the External ID is not enough; you have to do it right (and make sure your vendors do their part right, as well). Security researcher Kesten Broughton published a highly recommended, fascinating article about a sample of vendors and just how well and securely they are using the External ID mechanism. For the vendors (who are indeed third parties), Broughton’s article offers the following five recommendations:

• Generate a cryptographic Universally Unique Identifier (UUID) for the External ID
• Do not allow the customer to modify the UUID in the UI
• Do not include the External ID in any PUT or POST requests which may accidently be consumed and used to modify the External ID on the server side
• Treat the External ID as you would a private key. Do not log it in clear text, or display it to the user after the initial connection test succeeds
• Ensure that the assume-role call fails when no External ID is supplied

For you, as a client, it can be telling to see how well your third-party vendor implements External ID. For example, if the vendor lets you modify the External ID and/or provides you with an External ID that is not a lengthy UUID, that’s probably a bad sign. If you can set up the connection without an External ID at all – and you should of course avoid doing so – consider it bad practice on the vendor’s side.

Track third-party activity

As the title of this article suggests, we highly recommend monitoring the activity of all entities with access to your AWS environment – and that of your third-party entities with even more vigor.

An easy-to-close security gap which many times eludes us is inactive third parties. This phenomenon happens quite a bit; for example, you use a third party for a trial period, choose not to make the purchase and forget about the IAM role you created for it – and it stays in your account forever. If the vendor is somehow compromised or has a serious malfunction you may be affected by this inactive role for no good reason. This access risk can be even more destructive when the access granted involves access keys to an IAM user that are not limited to use from a specific external AWS account.

You can close this gap easily straight from the AWS IAM roles console. Sort the roles based on “Last Activity” in descending order from “None” (never used) to “Today” (used today). You can easily see which are third-party roles by looking under “Trusted Entities”. Determine a threshold from which the existence of such a role seems suspicious (we recommend 90 days) and look into all third-party roles that meet it. In fact, while you’re at it: look into all roles that cross that threshold. The AWS IAM users console has a similar feature.

Figure 3 - Sorting roles in the IAM Console based on “Last activity” to find roles not used at all or not used over a certain period of time

Additionally, you can also use the AWS Access Analyzer to review current roles that have a trust relationship outside your organization. Access Analyzer can take you a step further: as described in our recent post about managing access to secrets, you can configure Access Analyzer along with EventBridge and CloudWatch to get an alert when new access is configured, such as if anyone configures a new third-party access. The process is similar to the one described in the post about managing access to KMS keys. The only difference is that when configuring the EventBridge pattern, under the value of “resourceType”, you need to swipe the value "AWS::KMS::Key" with "AWS::IAM::Role" as you’re tracking access to a different type of resource. Learn more about Access Analyzer filter keys here.

Finally, to keep a close watch on what third parties are up to in your account, we recommend using CloudTrail, CloudWatch or Athena filtering to review the actions they’ve performed. We recognize that this is easier said than done because such granular monitoring can be rigorous and time consuming, and feel like looking for a needle in a haystack. Also, using Athena can be expensive. However – especially when you have reason to closely review a third party’s activity, such as upon a breach – a granular review of actions really is the best (and possibly only) way to get a clue as to the impact of third-party access on your organization.

In addition, reviewing the actions the third party performs can allow you to further rightsize the policy that grants it access by removing any permission not utilized. Doing so is of course extremely difficult without the means for automated analysis.

Rotate keys regularly

When it comes to best practices related to access keys and secrets for IAM Users, especially those provided to third parties, it’s extremely important to rotate them regularly, at least every three months.

Third parties (also) make mistakes

Finally, even if we’re not talking about actions originating from malice by the third party or someone seeking to harm it, third parties can make mistakes like everyone else, as well as design and operate systems that malfunction. The difference in intent doesn’t make the resulting damage less significant.

For example, if you provide a third party with permission to spin up EC2 instances and the third party uses it by mistake to generate massive amounts of computing resources, your bill will increase just as if done by an attacker looking to mine bitcoin. Alternatively, if you provide a third party with access to terminate EC2 instances and by mistake they shut down some which are mission critical at just the wrong time, the impact to your business will be just as significant as that of a hack designed to perform this on purpose.

The case for Tenable Cloud Security's third-party automated analysis

Performing all or even some of the best practices suggested above can be extremely beneficial in reducing the attack surface your organization is exposed to from third parties. However, doing so using only the out-of-the-box tools provided by AWS can be tedious—and is prone to human error.

Let’s review how a platform for managing cloud-infrastructure entitlements such as Tenable Cloud Security can help you perform these actions with greater ease and accuracy. Because the Tenable platform monitors your environment continuously, regularly analyzing its configuration and activity logs, it does much of the legwork described above for you.

The visual output gives you visibility into all third-party roles configured in your environment:

Figure 4 - List of third-party IAM roles ordered, as per the Findings column, by severity of risks detected for each .

The Tenable platform has a dedicated panel that lists third-party roles that it detects as exposed to certain risks, such as roles that are inactive or over privileged. You can review each role, get detailed information about its access permissions and mitigate the detected risks as you see fit.

Figure 5 - Risks panel for third-party roles

Let’s see how remediating risk plays out in Tenable. The “inactive role for more than 90 days” example is too simple since its obvious remedy is to remove the role. Let’s look at a more interesting example, such as the vendor recommended policy featured earlier in the post. That policy allowed the use of "kms:CreateGrant" on "*", creating a grossly over permissive and unnecessary risk.

In the Tenable platform you would see the risk and open its associated Findings page, which would look something like this:

Figure 6 - Findings page associated with the detected excessive risk

Using the Findings page, you can review an access graph of the policy that the role uses. The graph shows clearly and intuitively which services the role has access to, which permissions it has and, based on an automatic analysis of the actual activity the role performs, information about which permissions are completely or partially excessive:

Figure 7 - Access graph on a Findings page

Below is the remediation panel from which you can perform actions to rightsize the policies attached to the role.

Figure 8 - Remediation panel

Before taking any action, you can view the difference between the existing policies and the suggested one. In this case, you can see the exact statement that allows the over permissive action that you are about to remove.

Figure 9 - Difference between existing and suggested policies

The Tenable platform also provides an easy to use auditing console for tracking the activity that a third party has performed through the role that gave it access:

Figure 10 - Activity log

Among the risks that the Tenable platform reviews are keys that aren’t rotated. Looking for such keys manually is nearly impossible without a huge investment of time. The automated review sends you notification of the risk, enabling you to start the process of rotating the keys and updating the third party.

Figure 11 - Unrotated keys on a Findings page

Conclusion

Allowing third parties to access your AWS environment is unavoidable but poses ongoing security challenges. We hope you now have a clearer idea of the risks associated with third party access and the many ways to reduce these risks, including through security best practices like least privilege. Automated security tools can do much to help you control this extremely sensitive aspect of AWS identity and access management.

Where to go from here?

We hope this article has helped you understand ways that you can practice least privilege amid the sensitive yet unavoidable need to allow third parties to access your AWS environment.

If you’d like to know more about how Tenable Cloud Security can help you mitigate third-party access risk and achieve least privilege, we invite you to contact us for a demo.

***

Have you heard about Tenable's Access Undenied open-source tool that parses AWS AccessDenied CloudTrail events, explains the reasons for them and offers actionable fixes? Ready to take it for a spin? Try "Access Undenied" here, or read more about it here.