On 1 July 2019, Huawei’s Product Security Incident Response Team (PSIRT) published a response to Finite State’s Supply Chain Analysis of Huawei. After measured consideration, we are wholly disappointed with their response, which continues to demonstrate Huawei’s lacking commitment to security principles.
- Huawei’s security posture has not materially improved over time.
- Huawei has underinvested in root cause analysis and systematic fixes.
- Huawei’s evaluation of risks is insufficient, as is their understanding of the value of layered controls.
- Huawei has prioritized performance over security in ways that other critical hardware manufacturers have not.
Furthermore, this response contains language that is counter to the language offered by Huawei’s US Chief Security Officer, Andy Purdy, on Fox Business News. When asked about the Finite State SCA-1 report, he stated, “The fact is, this is exactly the kind of report that’s generated when a product goes to a customer… This is exactly what you find… The good news is: this is exactly what’s necessary to make America safer in communications and 5G — independent verification of everybody’s products.”
Despite the statements from their CSO, Huawei attacked our credibility, criticized our analysis as flawed, and in particular stated that “we cannot determine whether Finite State obtained the software from legitimate channels or guarantee its integrity.” In fact, we conducted our analysis on the actual firmware images that Huawei distributes to its customers — more than 95% of which were the latest versions available at the time of the analysis. Huawei argues that the source code analysis conducted by the UK government is superior to the Finite State approach. We believe the two approaches are complementary, and the findings are consistent between both approaches.
Since 2010, Huawei has partnered with the government of the United Kingdom to mitigate perceived and actual risks arising from inclusion of Huawei devices in critical infrastructure through the Huawei Cyber Security Evaluation Center (HCSEC). HCSEC analyzes source code voluntarily provided to them by Huawei. However, after eight years of work through this evaluation center, they have been able to achieve binary equivalence for only one device. In other words, they cannot guarantee that the source code tested exactly matched what was used to build the software inside Huawei’s devices. The UK has concluded “that any assurance provided by the overall risk management strategy, and therefore the Oversight Board, is currently limited.”
Finite State does not suffer from that problem. We analyzed only what is in the binaries distributed to their customers, and all firmware used in the study was available to their customers as of April 2019.
In their response, Huawei criticizes the public nature of the SCA-1 report. However, Finite State takes responsible disclosure very seriously. We follow an approved internal process, modeled after established standards, designed to give the manufacturer of a product time to mitigate a vulnerability and inform the affected users prior to public disclosure. Our team is currently engaged with several manufacturers to mitigate 0-day vulnerabilities discovered in their products. Huawei’s concerns about our disclosure policies are not relevant to this discussion because Finite State did not disclose any 0-day vulnerabilities in the SCA-1 report. Rather, we articulated risks and quantified the potential for vulnerabilities. All disclosed vulnerabilities were for known issues that are tracked in the National Vulnerability Database. We look forward to engaging with the Huawei PSIRT team to responsibly disclose all 0-day vulnerabilities discovered as part of our ongoing research into Huawei products.
Huawei also denies that the backdoor vulnerabilities discovered are relevant due to the nature of their operating system. In fact, several of the backdoor credentials discovered and detailed in the report appear in their VRP system. For those that are located in the Linux portion of their devices, Huawei’s response demonstrates a lack of understanding of the importance of layered security controls. Huawei argues that hard-coded credentials and cryptographic keys are not an issue because there may be other controls in place that mitigate some of the risks associated with these practices. It is true that Huawei’s implementation lowers the risk of some of these vulnerabilities, but it does not mean that they should be ignored. There is no reason that these credentials or crypto keys should be present. Given that these vulnerabilities allow for potential privilege escalation into lower layers of their architecture, they should be remediated quickly. Huawei’s response calls into question their risk assessment methodologies.
To Huawei’s credit, they do mention that they are taking actions in response to our report, including the removal of embedded crypto keys in at least one device. Huawei does not comment on whether they will be analyzing all firmware images to remove these keys to mitigate risk across their portfolio. Of additional concern, Huawei does not comment on how they will be proactively preventing these situations from occurring again. A strong commitment to security is demonstrated not just by responding to the incident, but by analyzing the root cause of how this incident occurred and what can be done to prevent it or similar issues from ever occurring again. Finite State welcomes the opportunity to work with Huawei PSIRT to verify the remediation of these security issues.
Huawei claims that Finite State’s analysis of firmware security over time was based upon an aggregate analysis of publicly reported Huawei CVEs over time. Huawei explains that Cisco and Microsoft also have an increased number of vulnerabilities over time, which we agree with. However, Finite State’s analysis of Huawei firmware security weakening over time was not based upon aggregate analysis of CVEs. In the report, it is clearly shown that the newer version of firmware for the Huawei CE6851 network switch was of higher risk across most of the nine different dimensions – such as unsafe function calls, default credentials, and known CVEs. Further, Finite State’s analysis of known CVEs is of significantly broader scope than Huawei’s. Finite State considers all known CVEs of the device and third-party code contained within. There is no plausible reason that Huawei is not mitigating the vulnerabilities in the components they use.
Huawei’s culture of risk management is called into question by their response to the lack of implementation of compiler-level protections. Huawei argues that implementing these protections cannot be accomplished because of the potential performance impact of enabling them. Huawei engages in ad hominem attacks based on this. They say Finite State lacks “maturity and competence” and that “Huawei would be happy to teach Finite State the basics of imbedded [sic] systems and global telecommunications operations that cover the globe.” But, Huawei is not the only hardware manufacturer that must balance the competing needs of secure computing with performance. A number of x86 processors used throughout the world are vulnerable to speculative execution attacks. After the initial set of vulnerabilities were disclosed in 2018, Intel released firmware upgrades that suffered an approximate 15% performance penalty, which was roughly 5x worse than Intel’s primary competitor, AMD. These fixes were deployed worldwide and directly impacted the performance of computing workloads in Amazon Web Services, Microsoft Azure, home PC’s, and a variety of embedded devices. Intel acted as a responsible hardware manufacturer. Intel released these fixes despite the performance impact and potential brand impact to ensure that Intel’s customers were properly protected from these vulnerabilities. Huawei, however, argues that they should not implement well known security fixes because this may impact the performance of their embedded devices.
In conclusion, we feel that Huawei’s response demonstrates a number of problems that help illustrate that their approach to security is insufficient. Based on 8 years of analysis of HCSEC reports, along with the recent Finite States analysis, we can clearly see that Huawei’s security posture has not materially improved over time. Huawei appears to have prioritized reactionary security and direct incident response rather than proactively improving their security engineering processes. Further, the maturity of Huawei’s evaluation of risks is in question due to their lack of understanding of the value of layered controls. Lastly, we can see that Huawei’s products are unsafe at any speed; they have prioritized performance over security in ways that other critical hardware manufacturers have not. An organization-wide culture change and shift in priority is necessary for Huawei to live up to the marketing promises made with respect to security. Huawei cannot deny that, now, multiple organizations have independently found similar, substantial security vulnerabilities in their products. We hope they act on these findings to ensure the security of their customers.
Read Assessment
