Researchers have discovered that hundreds of thousands of servers running open source monitoring software Prometheus on the open web expose passwords, tokens, denial of service (DoS) opportunities, and remote code execution.
As a leader among open source monitoring tools, Prometheus is widely used by organizations to monitor the performance of their applications and cloud infrastructure. But it comes with a problem: as stated in its documentation, “It's supposed to Untrusted users can access the Prometheus HTTP endpoint And records. They have access to all the time series information in the database, as well as a variety of operational/corrective information.”
Apparently, a large number of users are either unaware of the ways in which Prometheus is exposed by default, or are unaware of the value of the data that is exposed along the way. Using Shodan, researchers from Aqua Nautilus discovered more than 40,000 exposed Prometheus servers, and More than 296 thousand exposed “sources”. Which the software uses to collect data from monitored endpoints. The researchers found sensitive data in those servers and sources, and opportunities for “repojacking” and DoS attacks.
What Prometheus reveals
At first glance, the data Prometheus collects may seem rather bland: application performance metrics, metrics associated with specific cloud tools, and CPU, memory, and disk usage, for example.
“We think these are just statistics, they're just information about the health of the system. That's the problem,” says Assaf Morag, director of threat intelligence at Aqua Nautilus. Examining data from the attacker's perspective reveals all kinds of information that can help facilitate cyberattacks.
“We noticed that we could actually see passwords, plain-text tokens, and API addresses for internal sites that should remain hidden,” says Morag. For example, an exposed, unauthenticated version of Prometheus belonging to Skoda Auto, a Czech car manufacturer, was found, which exposed some of the company's subdomains, Docker registries, and images.
Besides exposing secrets, open Web Prometheus servers and sources also carry DoS risks. There's the '/debug/pprof' endpoint, for example, which helps users create a profile for remote hosts, and is enabled by default by most Prometheus components. In their testing, researchers showed that they can overload an endpoint to disrupt connections or completely crash Amazon Web Services Elastic Compute Cloud (AWS EC2) instances or Kubernetes pods.
“The result was conclusive: we ended up stopping the virtual machines every time we ran our script,” Morag said. Explaining the importance of such an attack scenario, he joked: “I read somewhere that Kubernetes clusters are running in fighter planes. I don't think they're vulnerable to the Internet, but (it shows) we're running Kubernetes in a lot of places today.”
Opportunities for re-kidnapping in Prometheus
Users can protect Prometheus servers and issuers by taking them offline, or at least adding a layer of authentication to keep prying eyes out. Of course, there are tools designed to mitigate DoS risks.
There is a third problem with the platform that is less easily solved: many issuers have been found to be vulnerable Repost attacks.
A re-exploit can occur when a developer changes or deletes their GitHub account and does not check out the namespace. Simply put, the attacker registers the developer's old username, then plants malware under the same address as the developer's old legitimate projects. Then, any projects pointing to this repository but not updating with the correct redirect link could end up ingesting the malicious version.
Prometheus' official documentation cited multiple issuers linked to freely claimable usernames, meaning an attacker could have jumped in and profited from remote code execution. Aqua Nautilus reported the issue to Prometheus, and it has since been addressed.
Morag stresses that the opportunities for re-hijacking are likely to be far more widespread than has been realized, so organizations need to monitor any discrepancies between the projects they rely on and the links they follow to access them. “It's not that difficult,” he says. “But if you're doing this for millions of open source projects, that's where the problem starts. If you use an automated (scanning tool), you might be safe.”
