How Cymulate Discovered an Abuse Risk in Google Cloud Platform (GCP)

As a Vulnerability Researcher at Cymulate, I spent the last few months honing my skills in cloud environments vulnerability detection. That is a logical evolution for a former mobile security researcher doubling up as a CTF enthusiast member of the CamelRiders team.

Hunting cloud environment vulnerabilities naturally led me to set up my own GCP environment, where I recently uncovered an abuse risk in GCP's 'google-guest-agent'.

Here is the roadmap to that discovery.

The Two Layers of GCP

After gaining a pretty good knowledge of GCP attack scenarios, I started checking a lot of attack paths.

The way I saw it, there are two layers where I can search for vulnerabilities:

• Linux layer - the Linux image with all its default services/binaries, etc.
• GCP layer - the layer Google has added that includes services, files, etc.
  The GCP layer is new relative to the Linux layer, and this layer includes all the concepts of cloud environments (service- accounts, metadata server, cloud networking, etc.)

My goal was to find a vulnerability in the GCP layer.

Checking Attack Paths

I tried techniques like MITM, ARP poisoning, SSRF, Misconfigurations, etc.

But nothing was successful.

Though I did not find vulnerabilities, I found elements that could be used for attack scenarios.

So, after 5 months of research, I started to weaponize the knowledge accumulated during my fruitless research by developing attack scenarios.

GCP Metadata Server

Short explanation about GCP metadata:

One of the attack paths that I was pursuing was to change the values of defined properties in the metadata. As mentioned above the Metadata is stored as key: value pairs.

Let's take, for example, a metadata property:

As we can see, the key in the metadata is ‘instance/name’, and the value is Cymulate.

Attempting Key Manipulation

So, I tried to change the value of the instance name but unsuccessfully.

The user only controls the keys and values under the attribute’s property which can be ‘instance/attributes/’ or ‘project/attributes’.

When I was trying to manipulate existing keys in the metadata, I found a regex validation.

Developing Attack Scenarios

The Cymulate Exposure Management and Security Validation platform includes an advanced purple teaming framework called ‘Advanced Scenarios’ that contains executions and templates of attack scenarios. With that, I started to create attack scenarios for GCP in a variety of categories (Lateral Movement, Privilege Escalation, Persistence, etc.)

While writing the attacks, I understood GCP concepts more deeply, and I came up with more attack scenarios but nothing that led to vulnerability or an abuse risk.

Uncovering the Abuse Risk

And then, the trigger came…

One of the attacks that I implemented was to add ssh-keys at the project level, which enabled expanding our foothold on every instance under the project (Lateral Movement).

When I wrote this attack, I came up with an idea similar to the regex validation attempt mentioned above and checked if there is any validation of regex on the username to which we are adding ssh-key.

The answer was NO.

There is no validation on the username string, which means that we can write anything!!!

My first POC was a quick win. Check what happens if I add the prefix of ‘../’ to the username.

1. Here we can see the root directory before I edit the metadata.

2. After I edited the metadata, a new directory named attacker was created at the root directory!

3. Inside the attacker directory we can see our ‘publickey’ inside ‘.ssh/authorized_keys’.

It works, I just write to the root directory with a weak user! When I saw this, I realized something here is wrong. So, what the hell happened here?

Background Process

What is happening in the background is basically this scenario:

• The GCP layer brings with her a process called ‘google-guest-agent’.
• This process checks whether the username we submit exists.
• If the username exists, the process creates in its home directory file named ‘.ssh/authorized_keys’ which the content will be the public key we submit into the metadata. (Don’t forget the format: ‘username:publickey’)
• If the username doesn’t exist, the ‘google-guest-agent’ will create a new user with the home directory “/home/username” and inside this home directory the file ‘.ssh/authorized_keys’ will be created.

Path Traversal Abuse Risk

Do you realize what we did?

Because our username ‘../attacker’ doesn’t exist in the VM, the ‘google-guest-agent’ created a home directory for that username at ‘/home/{username}’ -> ‘/home/../attacker’ -> ‘/attacker’ !!!

The amazing thing here is that the ‘google-guest-agent’ is running as a root, so what we are gaining here Is Path Traversal at the entire file system at root privileges!

Abuse Risk Limitations

But there are some limitations. We can create a file and fully control its content, but we are not controlling his name (‘authorized_keys’), and, even worse, this file will be created inside a directory called ‘.ssh’

How to Exploit the Abuse Risk

So, I started wondering how I can exploit this abuse risk…I came up with some ideas and am sharing the most interesting ones here.

Linux Sudoers File

The first thing I could though about was the Linux sudoers file. Basically, the thought was that every file in the ‘/etc/sudoers.d’ will be added to the sudoers file. And If I can add users and groups to the sudoers file I will be able to do anything I want.

Here we can see the sudoers file, in the last line, we can see @includedir to ‘/etc/sudoers.d’.

I could manage to write to the ‘/etc/sudoers.d’ but there was a limitation. As I mentioned above, my file was created inside a directory called ‘.ssh’.

The @includedir we saw above in the sudoers file doesn’t mean include sub-directories ☹

It means that the content of the file inside the ‘.ssh’ directory isn’t added to the sudoers file.

Linux Crontabs

My second thought was to look for a process that doesn’t validate file names but just reads the content of files in a specific path and executes them, so I started to play with crontab.
This kind of process would be great because I could plant my malicious file in the relevant path and get code execution of my malicious payload.

There is one problem, though. The crontabs will run the execution of our malicious file on behalf of the user that is the owner of the file. This is a problem, because at the end of the creation of the files, the ‘google-guest-agent’ which is the process that creates the files, executes the ‘chown’ command to the new user that just created.

Kubernetes Privilege Container Escaping

At this moment, my team leader Roy Haimof joined me to think about other ways of exploiting this abuse risk.

Our first thought was to create a scenario presupposing an existing initial foothold in a Kubernetes Node that is running a privileged container. In that privileged container, the ‘PermitRootLogin’ property in the ‘sshd_config’ file located on the ‘/etc/ssh’ is enabled which means that a ssh connection to the root user is enabled.

The proposed attack scenario supposed that
an adversary:

• Gains an initial foothold through a Kubernetes Node weak user, as explained above.
• The Kubernetes Node contains a privileged container.
• Exploits the Path traversal in order to override the ‘authorized_keys’ of the root user and adds an ssh-key.
• Connects as a root to the privileged container
• From the privileged container, mounts the Node file system.
• Gains root access on the Kubernetes Node -> Container escaping -> Privilege escalation.

This idea almost worked but just one thing was missing…

If you remember, I mentioned above that, at the end of the process, the ‘google-guest-agent’ executes the ‘chown’ command on the ‘authorized_keys’ file. This is a problem because there is a check of the ssh service that verifies whether the owner of the ‘authorized_keys’ file is equal to the UID we want to connect to. And when the root UID (0) doesn’t match our new user id that has just been created, we get an error - denied public key from the ssh service.

This POC failed but we could gain a DOS attack by exploiting the Path Traversal abuse risk. We basically can override every user’s ‘authorized_keys’, therefore, we can execute denial of the ssh service to the targeted users.

Bash Injection

Our second thought was to dive into the source code of the ‘google-guest-agent’ and check if the lack of validation on the username strings can lead to bash injection. After some checks, we found out that the username passed as a string and cannot translate into a bash command.

Command Injection Leads to Privilege Escalation

From then on, we dived into the source code of the ‘google-guest-agent’. We were feeling close to finding an exploit of that abuse risk. And finally, after a couple of hours, we found a way to exploit the abuse risk. Here is the full paper that was sent to Google, which describes the abuse risk and how to exploit it with a fully documented POC.

Description

Lack of validation of the username string when adding ssh-keys to the metadata leads to privilege escalation to root through command injection.

The ‘google-guest-agent’ adds a new user when triggered by a change of the ssh-keys property in the metadata.

The username string is not validated, and due to incorrect parsing of the command by the ‘google-guest-agent’, an attacker can extend the command ‘useradd’ being executed on trigger.

An adversary could exploit this abuse risk by maliciously triggering the execution of the ‘useradd’ command with an arbitrary payload in the username string.

Figure 1: The function formatting the username controlled by the user into the ‘useradd’ command

The command is extended by adding spaces to the username (that is user controlled), since the function that formats the command splits the arguments using spaces by calling strings. Fields (Golang).

Figure 2: Documentation of the strings. Fields() function in Golang

Splitting the string causes the newly split strings to be parsed as additional arguments, therefore allowing an adversary to add flags to the command.
The command is then executed by the ‘google-guest-agent’.

Figure 3: The function that is responsible for the execution of the altered command

By creating an extension for the ‘useradd’ command, through lack of validation, an adversary could alter the command and add a new user to any group, for example, sudo group (27).

Adding an extra “-p” argument to the command allows the adversary to override the previous value passed to the “-p” flag by the ‘google-guest-agent’, therefore an adversary also controls the password of the user created, which is necessary in order to log in to the new user and escalate to root privileges.

Figure 4: The command extension, format and the final payload to be executed by the google-guest-agent

Attack Scenario Walkthrough

An adversary:

• Gains access to a service account with permission to edit the instance metadata.
• Injects a command extension to the username string under the ssh-keys metadata property.
• Executes a gcloud API that updates the metadata and triggers the ‘google-guest-agent’.
• The ‘google-guest-agent’ then executes the adversary-controlled malicious command payload.
• A new privileged user has been created in the sudo group by the ‘google-guest-agent’ which the adversary has full control over.
• The adversary has gained access to a user in the sudo group and therefore, has root privileges.

Proof of Concept

1. The adversary does not have sudo privileges on the victim's machine.

2. Adversary exploits command injection in order to create a new user with a custom password that will be part of the sudo group(27) by adding a malicious value to ssh-keys property in the metadata, from a file.

3. A new user called weakuser has been created, and he is part of the sudo group(27), which has the privilege to execute sudo commands.

4. The adversary is able to access the new user and therefore has escalated privileges to root.

Mitigation Measures

Firstly, ensure that only service accounts you trust have permission to edit the metadata. Editing metadata is a high privilege. The next mitigation is to enable the OS-Login property in the metadata by adding "enable-oslogin=TRUE" to the metadata.

What is OS-Login?

OS-Login is a feature that is used to manage SSH access to instances using IAM without having to create and manage individual SSH keys. OS-Login maintains a consistent Linux user identity across VM instances and is the recommended way to manage many users across multiple instances or projects. When OS-Login is enabled, all the SSH connections will be made by authenticated users from the IAM.

How does enabling OS-Login protects you from attacks?

When OS-Login is enabled, the ‘google-guest-agent’ will not maintain connections to the instance via SSH keys. It will also remove the value of the ‘ssh-keys’ property in the metadata, which means that no matter which malicious username the attacker will use, it is not relevant due to a deletion of the ‘ssh-keys’ value by the ‘google-guest-agent’.

You may ask yourself that if an attacker can edit the metadata, why wouldn’t he just disable OS-Login?

Yes, he could as long we didn’t use the OS-Login organization policy ?

To ensure that all new projects, and the VM instances created in these new projects, have OS-Login enabled, you can set up the following OS-Login constraint in your organization:

“gcloud beta resource-manager org-policies enable-enforce compute.requireOsLogin --organization=organization-id”

To read the documentation for managing OS-Login, see - Manage OS Login in an organization

Conclusion

I wish I could claim to be a genius and the only one capable of uncovering an abuse risk missed by the SecDevOps of as prestigious an entity as Google, but I am afraid I am not.

The truth is that only security researchers and committed cyber-attackers have the time and resources to look for such vulnerabilities and abuse risks. There is a constant race between security Researchers and threat actors to uncover them, but vulnerabilities are there in any and all environments - and uncovered they will be, either by your team or by cyber attackers.

The best protection is to ensure that your own environment security controls are optimally configured so that, even if a supplier – GCP or other – contains exploitable vulnerabilities and abuse risks, your own security controls will compensate and minimize its reach in your environment.

Cymulate’s BAS and Purple Teaming Enablement validate that your security controls are optimally configured and ready to stop, or at least spot, a cyber attacker hitching a ride on a supplier’s exploited vulnerability and abuse risks.