Authors
In February 2021, we had the opportunity to assess the HyperFlex HX platform from Cisco during a routine customer engagement. This resulted in the detection of three significant vulnerabilities. In this article we discuss our findings and will explain why they exist in the platform, how they can be exploited and the significance of these vulnerabilities.
The vulnerabilities discussed have been assigned CVE ID’s and considered in Cisco’s subsequent Security Advisories (1, 2). These are:
- CVE-2021-1497
Cisco HyperFlex HX Installer Virtual Machine Command Injection Vulnerability (CVSS Base Score: 9.8); - CVE-2021-1498
Cisco HyperFlex HX Data Platform Command Injection Vulnerability (CVSS Base Score: 7.3); - CVE-2021-1499
the Cisco HyperFlex the HX the Data Platform the Upload the File Vulnerability (CVSS Base Score: 5.3)
Background
Cisco HyperFlex HX is a set of systems that combine various networks and computing resources into a single platform. One of the key features of the Cisco HyperFlex HX Data Platform(software-defined storage) is that it allows the end user to work with various storage devices and virtualize all elements and processes. This allows the user to easily back up data, allocate resources or clone resources. This concept is called Hyperconverged Infrastructure (HCI) . You read more about this on the Cisco website “Hyperconverged Infrastructure (the HCI): HyperFlex” and “Cisco HyperFlex the HX-the Series“.
Cisco HyperFlex HX comes with a web interface, which allows for easy configuration. The version we tested is the Cisco HyperFlex HX Data Platform v4.5.1a-39020. This can be seen below:
The HyperFlex platform is deployed as an image on the Ubuntu operating system. Our initial inspection showed that nginx 1.8.1 is used as the front-end web server. Knowing this, we decided to look at the nginx configuration files to see what else we could learn. The nginx configuration for “springpath” project are located in the /usr/share/springpath/storfs-misc/ directory. Springpath developed a distributed file system for hyperconvergence, which Cisco acquired in 2017.
Our priority was to gain access to the system management without any authentication. So we carried out a detailed examination of each route (location) in the configuration file. After a thorough investigation of the configuration file, we were able to prioritize areas to research further which may allow us to do so.
Findings
CVE -2021-1497: RCE through the password input field
Authentication is the process of verifying that a user is who they say they are. This process is frequently achieved by passing a username and a password to the application. Authorization is the process of granting access or denying access to a particular resource. Authentication and authorization are closely linked processes which determine who and what can be accessed by a user or application.
During our testing we noted that the process of authentication is handled by a third-party service. This is shown in the configuration file below:
By looking at the content of this configuration section, you can see that authentication process is handled by the binary file /opt/springpath/auth/auth. This service is a 64-bit ELF application. We noted that its size is larger than standard applications.. This could indicate a large amount of debugging information in the binary or a big compiled Golang project. The latter was quickly confirmed after reading section headers with the readelf command.
The auth binary handles several URL requests:
/auth/auth/change/auth/logout/auth/verify/auth/sessionInfo
Most of these requests do not take user input, however the URL /auth and /auth/change allow user input through the parameter’s username, password and newPassword. The /auth page handles authentication. When a user enters their username and password, the HTTP request is sent as follows:
Analysis of the authentication application showed that the credentials, are retrieved in the main_loginHandler function through the standard functions net/http.(*Request).ParseForm. Next, the login and password are passed to the main_validateLogin function. This function retrieves the value from the username parameter and the corresponding user hash from the /etc/shadow file. If the user exists, then a further process is executed which checks the password entered through the main_validatePassword function, using the main_checkHash function.
The hash value is calculated by calling a one-line Python script via os/exec.Command:
python -c "import crypt; print(crypt.crypt(\"OUR_PASS\", \"$6$$\"));"Then the resulting hash value is extracted and compared with the value from /etc/shadow.
The is a big problem with this method of executing commands from Python is that allows for command injection. This is a significant vulnerability; there is no input validation, and any user input is passed to os/exec.Command as it was entered. Additionally, commands are executed with the privileges of the application, in this case root. It’s therefore trivial to execute systems commands with malicious intention. For example we entered the following into the password field, causing a reboot of the system:
123", "$6$$"));import os;os.system("reboot");print(crypt.crypt("
This vulnerability allows a malicious user to call a remote reverse shell with root privileges using only one HTTP request:
password parameterThe other URL that handles user input, /auth/change, also presents a way to execute arbitrary code.
The password change is handled by the main_changeHandler function. This works much the same as the login process /auth. The existence of the user is checked using the same processes and the password hash is calculated using the same function main_checkHash. In the value of the new password, newPassword we were able to pass the same input, causing a system reboot:
123", "$6$$"));import os;os.system("reboot");print(crypt.crypt("
newPassword parameterWe found two ways to trigger the remote execution of arbitrary code, using the /auth and /auth/change endpoints. However, as both the password and newPassword parameters use the same function, main_checkHash to execute external commands, the vendor only issued one CVE. A more secure way to execute external commands in python is to use the sub-process module and to validate the arguments taken from user input before execution.
CVE-2021-1498: Cisco HyperFlex HX Data Platform Command Injection Vulnerability
We analyzed the nginx configuration file and noticed that the /storfs-asup endpoint redirects all requests to the local Apache Tomcat server at TCP port 8000.
We then looked at the Apache Tomcat configuration file, web.xml, we found:
From this file it is clear that the /storfs-asup URL is processed by the StorfsAsup class, located at /var/lib/tomcat8/webapps/ROOT/WEB-INF/classes/com/storvisor/sysmgmt/service/StorfsAsup.class.
public class StorfsAsup extends HttpServlet {
...
protected void processRequest(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
String action = request.getParameter("action");
if (action == null) {
String msg = "Action for the servlet need be specified.";
writeErrorResponse(response, msg);
return;
}
try {
String token = request.getParameter("token");
StringBuilder cmd = new StringBuilder();
cmd.append("exec /bin/storfs-asup ");
cmd.append(token);
String mode = request.getParameter("mode");
cmd.append(" ");
cmd.append(mode);
cmd.append(" > /dev/null");
logger.info("storfs-asup cmd to run : " + cmd);
ProcessBuilder pb = new ProcessBuilder(new String[] { "/bin/bash", "-c", cmd.toString() });
logger.info("Starting the storfs-asup now: ");
long startTime = System.currentTimeMillis();
Process p = pb.start();
...
}
...
}
}When analyzing this class, we noticed that the parameters received from the user are not filtered in any way or validated in anyway. They are passed to a string, which is subsequently executed as an operating system command. Based on this information, we can form a malicious GET request, that will be executed as an OS command.
GET /storfs-asup/?Action=asd&token=%60[any_OS_command]%60 HTTP/1.1
Host: 192.168.31.76
Connection: close
This results in the execution of arbitrary commands on the server from an unauthenticated user.
It is worth noting that the web path /storfs-asup is only available if port 80 is accessible externally. To exploit the vulnerability through port 443, the request needs to be modified to use the path /crossdomain.xml/..;/storfs-asup/. This works because the nginx configuration file specifies that all requests starting with /crossdomain.xml are proxied to Tomcat and using the well-known directory traversal tomcat technique “..;/“, we can access any servlet on the tomcat web server.
CVE-2021-1499: Cisco HyperFlex HX Data Platform File Upload Vulnerability
Closer inspection of the nginx configuration file showed us the following location for file uploads:
To request this URL, no authorization is required and the path is accessible externally. As is the vulnerability CVE-2021-1498, this is setup in a similar way. A request to the proxy application which is listening on port 8000 for incoming connections.
As an experiment, we sent a multipart request for directory traversal and it was accepted.
As a result, the file with the name passwd9 was created for the user “tomcat8” in the specified directory:
The complete lack of authentication means that we are able to download any arbitrary files to any location on the file system with “tomcat8” user privileges. This is a significant oversight of the developer’s part.
During the process of publishing this paper we gained a broader understanding of the vulnerability allowing us to execute arbitrary code. The vulnerability seems a lot less harmless now, than it did before. The details are available at the following link.
Not every mistake is a mistake
The default route in the nginx configuration file also brought our attention. This route handles all HTTP requests that do not meet any of the other described rules in the configuration file. These requests are redirected to port 8002, which is only available internally.
As with the auth binary, this route is handled by the installer 64-bit ELF application and is also written in Golang.
Assessment showed that this application is a compiled 64-bit Golang project. This application was made for handling the /api/* requests. To work with the API interface, it is necessary to have an authorization token. The installer binary handles the following endpoints:
/api/run/api/orgs/api/poll/api/about/api/proxy/api/reset/api/config/api/fields/api/upload/api/restart/api/servers/api/query_hfp/api/hypervisor/api/datacenters/api/logOnServer/api/add_ip_block/api/job/{job_id}/api/tech_support/api/write_config/api/validate_ucsm/api/update_catalog/api/upload_catalog/api/validate_login
Though the initial requirement for this research was to find vulnerabilities that don’t require prerequisites or authentication, this finding requires a user to be logged into the Cisco HyperFlex web interface. We analyzed the endpoint handlers and found two requests that were working with the file system. The /api/update_catalog and /api/upload routes allowed us to upload arbitrary files to a specific directory. The handlers responsible for working with the URL data are main_uploadCatalogHandler and main_uploadHandler.
In the first case, the files we transferred were written to the /opt/springpath/packages/ directory. Using a simple path traversal attack, we were able to write a file outside of this directory in an arbitrary location on the system.
As a result, we are able to write files to any place on the system as these requests are made with root privileges.
The second example of requests causes a file written to the /var/www/localhost/images/ directory, from the web interface. The works in a similar way to the previous request by changing the file name in an HTTP multipart POST request. This allows a malicious user to create a file anywhere on the file system.
Cisco does not consider these as vulnerabilities, assuming that if the attacker knows customer credentials, it would be possible to log in via enabled SSH server. However, we still consider this code to be poorly implemented.
Conclusion
This research project started as an opportunity during a routine customer engagement. What we found is three significant vulnerabilities. These vulnerabilities are the result of a lack of input validation, improper management of authentication and authorization, and reliance on third party code. These can be mitigated by following secure coding best practices and ensuring that security testing is an integral part of the development process.
Command injection vulnerabilities remain a significant issue in industry, despite the development of best practices such as SSDLC (Secure Software Development Lifecycle). This could be solved in two parts. First if there is appetite in the industry to make best practices a requirement through standards. Second, if external testing is implemented to assess if standards are adhered two.
Finally, it should be noted that third-party products are often not up to the same rigorous security standards that are implemented in existing product lines. The acquisition and integration of third-party products is a difficult path to manage. Every acquisition should involve thorough review of coding practices and security testing. In some cases, they may benefit from a complete overall to ensure that data is handled consistently between components and in a secure manner.