Auditing iam:PassRole: A Problematic Privilege Escalation Permission

How to determine which identities need iam:PassRole to help enforce “use it or lose it” least privilege.

iam:PassRole is an AWS permission that enables critical privilege escalation; many supposedly low-privilege identities tend to have it. It’s hard to tell which identity and access management (IAM) users and roles need the permission. We have mapped out a list of AWS actions where it is likely that iam:PassRole is required and the names of parameters that pass roles

In this post, we explain how to monitor your CloudTrail to determine which identities actually need iam:PassRole, and for which roles and services, to help you enforce “use it or lose it” least privilege

Why should we care about iam:PassRole?

By now we all know how important IAM governance is. IAM acts both as the perimeter of a cloud deployment and as one of the main mitigators of damage when a node in the cloud deployment becomes (inevitably) jeopardized. Unfortunately, even in some of the best-built cloud environments, between all the oft-rotated credentials, granular-permission policies and requirements for mult-factor authentication (MFA)-protected access, one little permission with massive hidden privileges tends to get stuffed — like cramming things into a full closet in an otherwise tidy room. If this scenario resonates with you in any way:

1. I feel your pain; and
2. It’s time for some cloud security KonMari.

The AWS Security community is painfully aware of this problem, and we can be sure attackers are (gleefully) aware of it as well.

What is iam:PassRole?

The basic idea of iam:PassRole is simple: whenever a principal (which can be a user or a role, a human, code or a service) uses a service that needs to perform other actions, the AWS architecture often has that service assume an AWS role to perform the actions. When that happens, the service performing the actions is “passed” a role by the calling principal and implicitly (without performing sts:AssumeRole) assumes that role to perform the actions. The privileges associated with the role are different from —- and can be greater than — those of the principal calling the action.

Perhaps the most well-known example of this is when launching an EC2 instance with a certain IAM Instance profile. The instance profile is resolved to an IAM role whose permissions determine what the instance can and can’t do. Whenever behavior like this happens, AWS checks, behind the scenes, if the calling principal has the permission iam:PassRole to pass the role to the service. If it does, and if the role is unrestricted by IAM policies, the attacker is able to grant elevated permissions and escalated privilege to the EC2 instance. This scenario and others are described in a post by Rhino Security Labs. Often, in such cases, the attacker's ability to escalate privilege is only bounded by the highest-privilege role they can pass. You know all those unused high-privileged roles and identities you may have lying around (and unfortunately, the majority of customer environments we encounter have a ton of these)? iam:PassRole is their opportunity to “shine,” as they are now enablers of unfettered privilege escalation.

The fact that iam:PassRole is both a facilitator of critical privilege escalation and a permission that is remarkably difficult to monitor, control and create policies presents a problem for security teams.

Monitoring PassRole to achieve least privilege

At Ermetic (now Tenable Cloud Security), we ran into a problem trying to audit PassRole permissions. By nature, PassRole’s privilege escalation vector makes the question of which roles an identity is allowed to pass just as important as the question of whether the identity should have permission to pass any role at all.

We needed an effective way, with resource level permission granularity, to know where iam:PassRole is being used, and exactly with which roles.

To determine which PassRole permissions are actually being used, we decided to leverage CloudTrail and other activity logs. Close examination of the logs helps us understand the overprivileged policies in place and strip them down to least privilege without breaking usability.

Let’s use a policy like this as an example:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeImages",
                "ec2:DescribeSubnets",
                "ec2:RequestSpotInstances",
                "ec2:TerminateInstances",
                "ec2:DescribeInstanceStatus",
                "ec2:CreateTags",
                "ec2:RunInstances"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "iam:PassedToService": [
                        "ec2.amazonaws.com"
                    ]
                }
            }
        }
    ]
}

We look at the CloudTrail logs and find events that have logged actions for which PassRole was required. If, for example, we find that the only role is:

role/EC2EmptyRole

We narrow the policy down to something like:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeImages",
                "ec2:DescribeSubnets",
                "ec2:RequestSpotInstances",
                "ec2:TerminateInstances",
                "ec2:DescribeInstanceStatus",
                "ec2:CreateTags",
                "ec2:RunInstances"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource":  "arn:aws:iam::123456789012:role/EC2EmptyRole",
            "Condition": {
                "StringEquals": {
                    "iam:PassedToService": [
                        "ec2.amazonaws.com"
                    ]
                }
            }
        }
    ]
}

And, voila! We have a much safer policy that doesn’t allow user privilege escalation to the highest privilege instance profile.

iam:PassRole is a pain to audit

Let’s drill down into what makes PassRole so difficult to audit.

Not logged

The first problem is that because iam:PassRole is only a permission and not an API action, “uses” of PassRole (or, rather, checks of the permission iam:PassRole) are by definition not logged to CloudTrail. In fact, even Access Advisor doesn’t account for iam:PassRole. That is, if you have an identity that actually uses its PassRole permission all the time but doesn’t do anything else on IAM, Access Advisor will indicate that you haven’t used your IAM permissions. This could cause you to remove PassRole and negatively impact usability. Try this for yourselves: Using the AWS CLI, create a lambda that uses an existing role (using an identity that hasn’t used IAM) and check out Access Advisor for that identity:

Insufficiently documented

As mentioned in the recent article by Dustin Whited of ScaleSec, actions dependent on iam:PassRole are, ostensibly, documented in the AWS Actions, Resources, and Condition Keys reference documents. Unfortunately, this documentation is highly insufficient. We sifted through the docs looking for actions dependent on iam:PassRole and found reference to only 58 actions in 14 services. It was immediately apparent that even some of the most infamously problematic actions that depend on PassRole were absent, for example: Lambda’s CreateFunction/UpdateFunction and EC2’s RunInstances. By the end of our own PassRole investigation, we found more than 300 actions in more than 90 services dependent on iam:PassRole: roughly six times the amount documented.

Hard to pin down

Since iam:PassRole is not explicitly logged or comprehensively documented, we have to find it between the lines and spaces, using the behavior, documentation and parameters of other API actions to identify the tracks of a role being passed and infer that the iam:PassRole permission is likely to be required for that role.

Lack of standardization

Most IAM actions are logged to CloudTrail. In the CloudTrail log, the parameters of the API action are (usually) logged under a key called “requestParameters”. These request parameters are our main source of information for inferring passed roles. Since iam:PassRole is not logged to CloudTrail, if we want to audit pass-role at resource-level granularity (and we do!), we have to deduce the role that iam:PassRole passes from each event’s request parameters. This is no cake walk, as parameters for PassRole vary wildly. Some are relatively straightforward, some are more complex.

Here’s a relatively simple example:

{
  "awsRegion": "us-east-1",
  "eventID": "57a1eefb-2c4d-4881-99e3-9a137e5db024",
  "eventName": "CreateFunction20150331",
  "eventSource": "lambda.amazonaws.com",
  "eventTime": "2021-01-05T14:53:49Z",
  "eventType": "AwsApiCall",
  [... User identity information, source IP, etc ...]
  "requestParameters": {
    "functionName": "my-function",
    "runtime": "nodejs12.x",
    "role": "arn:aws:iam::123456789012:role/lambda-ex",
    "handler": "index.handler",
    "code": {},
    "publish": false,
    "environment": {}
  },
  [... Response elements and some more event data ...]
}

False positives

Of course, even where roles are mentioned in the request parameters, iam:PassRole is not necessarily involved. If we’re not careful, this could lead to actions being mistakenly identified as requiring iam:PassRole. For example: iam:GetRole, which requires a role as the target of the action and lists that role in its “requestParameters” field, does not require the PassRole permission.

Parameters can appear in the request without being logged by CloudTrail

CloudTrail events aren’t exact representations of the actions they log; sometimes certain request parameters just aren’t logged for certain actions. Take a look at this CloudTrail event for elasticbeanstalk:AssociateEnvironmentOperationsRole (some details redacted):

{
    "eventVersion": "1.05",
    "userIdentity": {
        "type": "IAMUser",
        "principalId": <principalId>,
        "arn": "arn:aws:iam::123456789012:user/<user>",
        "accountId": "123456789012",
        "accessKeyId": <accessKeyId>,
        "userName": <user>
    },
    "eventTime": "2020-12-14T14:39:53Z",
    "eventSource": "elasticbeanstalk.amazonaws.com",
    "eventName": "AssociateEnvironmentOperationsRole",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "xxx.xxx.xxx.xxx",
    "userAgent": "aws-cli/1.18.185 Python/3.9.0+ Linux/4.4.0-19041-Microsoft botocore/1.19.25",
    "requestParameters": null,
    "responseElements": null,
    "requestID": <GUID>,
    "eventID": <GUID>,
    "eventType": "AwsApiCall",
    "recipientAccountId": "123456789012"
}

The action request includes the parameters “EnvironmentName” and “OperationsRole” but Cloud Trail has not logged them. It’s therefore impossible to tell which role was passed from the CloudTrail log.

The “matryoshka” problem

Another way roles can be passed is via other parameters that contain a reference to a role within them. The most well-known example of this is iamInstanceProfile, the parameter that defines the IAM role that the EC2 instance will use. When it runs, the instance obtains temporary security credentials that correspond to the role the instance profile contains and passes these to the application. LaunchTemplates and LaunchConfigurations are also examples of this and can contain instance profiles that define the roles these instances will use.

Image source: Tenable, 2021

In many of these cases, iam:PassRole is required to perform the action.

We nicknamed this the matryoshka problem, after the famous Russian nesting dolls, as it can lend itself to multiple layers of nesting, e.g. a role within an iamInstanceProfile within a LaunchTemplate, further obfuscating understanding permissions scope.

Implicitly passed roles?

In some cases in which an action is performed with a service, a service-linked role is used (and created if needed) by default — even when a parameter isn’t explicitly stated. We must be aware when defaults are used in lieu of an explicitly stated parameter.

AWS API vs. CloudTrail vs. IAM

To top it all off, we also have to solve the classic problem of name differences across CloudTrail, the AWS API and IAM permissions. Scott Piper and others have written about this before; you can help by upvoting these open Github issues: 1, 2.

Characterizing the problem

Now that we’ve understood the problem, this is what we need to do to solve it for our use case:

• Find role parameters
• Sift through them to identify which require iam:PassRole, filtering out false positives
• Find all the matryoshki (nested role parameters)
• Find implicit passes of a role

Once we complete these steps, we are able to expose where a role is in fact passed.

Automated approaches to mapping PassRole requirements

Since AWS has over 8,500 permissions (and counting), a fully manual approach doesn’t really scale for this problem, so we had to find ways to automate at least part of the work.

Finding PassRole Candidates

Scanning the API

Boto3 is a python AWS SDK that contains, among other things, an updated accounting of the AWS API in JSON files in the botocore project. We used a python script to scan these JSON files for every action in which “role” is mentioned and found 392 API actions.

(Note: We also handled nested parameters at this stage.)

It is clear that some of the API actions do not pass a role. After sifting out read-only actions, we were down to 379 actions. But we must remember the matryoshki — the “containers,” as it were. We started by adding the three most well-known examples: InstanceProfile, LaunchTemplate and LaunchConfiguration. (To be on the safe side, we went a step further and added everything with the word “launch” in it.) We were up to 408 actions. We also took a deep dive into other AWS Compute services — which tend to be the riskiest in terms of privilege escalation — and found additional role-containing parameters.

Scanning CloudTrail

Working from the API side is inherently dependent on the good graces of AWS parameter naming which, unfortunately, occasionally disappoints. To cope with the problem, we scanned our customers’ CloudTrail data and looked for cases in which either a role or the containers we identified appeared within the request parameters. This approach helped find parameters we would not find from the API, such as those given by “key”: <key>, “value”: <value> entries in parameter dictionaries, as in this example from datapipeline:PutPipelineDefintion:

"pipelineObjects": [
       {
           "fields": [
               {
                   "key": "role",
                   "stringValue": "DataPipelineDefaultRole"
               }
           ]
       }
   ]

As you can see, the passed role is stored in the BSON “array of documents” fashion: in a dictionary inside a list inside a dictionary inside a list that is the value of the parameter key “pipelineObjects”.

Manual approaches to mapping PassRole requirements

Ultimately, automated approaches can only provide hints to uses of iam:PassRole. Some decisions cannot be made without actually trying to perform the actions. We manually checked some of the actions identified by the automatic API and CloudTrail scans. We focused on actions in AWS Compute services first as these tend to be the easiest for attackers to exploit.

Examining special cases

We created the IAM user Melo to serve as a principal for manually checking if PassRole is required to perform an action. These are Melo’s IAM policies:

AdministratorAccess

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        }
    ]
}

MeloCantPass

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Deny",
            "Action": "iam:PassRole",
            "Resource": "*"
        }
    ]                                                                                                                                                                                                                                                                                           }

(Note: We affectionately named user/Melo after future basketball hall-of-famer and New York legend Carmelo Anthony, who -- much like user/Melo -- sometimes seemed able to do anything except pass.)

We opted to use an IAM user since we were using a completely standalone account so the entity bore no special risks. Using an IAM role is preferable, especially if your sandbox account is connected to your organization in any way. If you do choose to use an IAM user, we strongly advise you to:

• use SCP guardrails;
• make the API keys inactive when not in use;
• and rotate keys frequently.

Using Melo’s credentials, we perform an action. If the action requires PassRole, we get this result:

user@MACHINE:~$ aws autoscaling create-launch-configuration  --launch-configuration-name rsc-test-launch-config --iam-instance-profile arn:aws:iam::123456789012:instance-profile/<Ec2RoleName>

An error occurred (AccessDenied) when calling the CreateLaunchConfiguration operation: User: arn:aws:iam::123456789012:user/Melo is not authorized to perform: iam:PassRole on resource: arn:aws:iam::123456789012:role/<Ec2RoleName> with an explicit deny

Here, we did the manual work of checking actions identified by our hints for potential uses of iam:PassRole, focusing on AWS Compute services first to be most granular where it matters most.

Which types of actions require IAM PassRole?

The general rule is that PassRole is required during the setup and creation of resources, as the AWS user manual states: 

[...] You only have to pass the role to the service once during set-up, and not every time that the service assumes the role.

However, this rule doesn’t apply consistently — sometimes PassRole could be required for two stages of the same process. One interesting example is CreateLaunchConfiguration and CreateLaunchTemplate. You would think these two actions are largely similar and should behave in the same way. However, CreateLaunchTemplate is something Melo can do and does not require PassRole, whereas CreateLaunchConfiguration gives an AccessDenied error message!

Another example can be found in ECS. If you want to create and run a task, ecs:RegisterTaskDefinition and ecs:RunTask both require iam:PassRole so, in the chain of creating and running a task, you actually may need to have the PassRole permission for two actions.

Service-linked roles requiring iam:PassRole

The AWS user manual has this to say: 

When you create a service-linked role, you must also have permission to pass that role to the service. Some services automatically create a service-linked role in your account when you perform an action in that service. For example, Amazon EC2 Auto Scaling creates the AWSServiceRoleForAutoScaling service-linked role for you the first time that you create an Auto Scaling group. If you try to create an Auto Scaling group without the PassRole permission, you receive an error.

This is not a very fun paragraph to read. Implicit requirements for iam:PassRole in 61 different services? Sounds like a lot of trouble. However, we sent Melo to check a few sample services (EC2 Auto Scaling, for one) and found no issue at all. It seems that the association of the AWS service role does not check PassRole. You can try this! Create a role with Melo’s permissions and try to use one of the 61 AWS services that support a service-linked role. If you find an instance where an action fails, we’d love to hear from you.

[cloudshell-user@ip-xxxxx ~]$ aws autoscaling create-auto-scaling-group --auto-scaling-group-name <asg-name> --launch-template "LaunchTemplateName=<lt-name>,Version=1" --min-size 1 --max-size 3 --vpc-zone-identifier "<subnet-1>,<subnet-2>,<subnet-3>"
[cloudshell-user@ip-xxxxxx ~]$

(Note: CreateAutoScalingGroup did not fail, even when PassRole was denied and AWSServiceRoleForAutoScaling did not exist.)

Whatever its cause, this discrepancy does make auditing PassRole and configuring IAM policies easier.

Conclusion

To audit IAM permissions, we searched customer CloudTrail logs and the AWS API for all references to role. We filtered the identified actions for non-read-only actions, identified role-containing parameters, iterated the process with the role-containing parameters and then checked actions manually in high-risk services for requiring PassRole. Note that for actions not checked manually, we erred on the side of over-permission when compiling this general-use list. We suggest that you use this list not as a be-all-end-all solution but, rather, as a starting point for exploring PassRole least-privilege.

The PassRole permission requirement is ubiquitous. In our list, it extends to more than 300 actions in more than 90 services and, in some cases, is obscured by parameters that indirectly contain the role being passed. Comprehensively auditing it can be a handful but is absolutely worth it as not doing so means leaving relatively easy avenues for privilege escalation wide open. If you’re constructing your cloud deployment, try to resist the pressure to facilitate usability at all costs: avoid allowing PassRole on resource “*”.

We’re releasing our best-estimate, true-as-of-today list of PassRole API actions. You can use this list to determine if an identity really needs iam:PassRole and what roles it needs it for, as follows:

1. Look for CloudTrail events corresponding to a PassRole-requiring action performed by the identity in question
2. Find the parameter that describes the role that was passed
3. This identity might indeed need iam:PassRole for that role and/or service — or others like it

After you’ve affirmed what permissions are actually needed, remove any that aren’t.

We offer this as a broad starting point to addressing the problem, with a focus on AWS Compute services. Everyone knows different services well -- and a community effort can go a long way — so please be sure to add your updates and corrections to the GitHub gist.

Thoughts? Contact me!
X (formerly Twitter): @NoamDahan