Fuzz Testing vs Penetration Testing in Automotive Cybersecurity

Introduction

As automobiles and the automotive industry in general become more intelligent and interconnected, security weaknesses and threats become more prominent. Due to the complexity in the integration of these cyber-physical systems, modern vehicles are susceptible to a wide array of threats from remote code execution to data breaches and inside actors. The security of these systems is not merely a matter of protecting personal data or preventing financial losses; it is a critical safety concern with people’s lives on the line.

Performing security testing and mitigating any risks identified during the development process is very important before vehicles are released to the public. Cybersecurity testing helps in safeguarding the integrity, safety and trustworthiness of these systems. By proactively identifying and addressing security weaknesses through comprehensive testing methodologies like fuzzing testing and penetration testing, automotive manufacturers can instill confidence in their products to ensure the safety of drivers, passengers, and pedestrians alike.

Understanding Fuzz Testing

Fuzz Testing, also known as Fuzzing, is a type of security test that validates the security robustness of a system / device by feeding random and / or structured data to it and monitoring the response. This is similar to the validation process regarding a crash test to monitor the safety robustness of a vehicle. The tool used for fuzz testing is generally called a fuzzer and they come in many different types, for instance; random fuzzers, mutation based fuzzers, and generation based fuzzers.

Random Fuzzing

Random fuzzing is the oldest form of fuzzing where a randomized data is sent to the system / device. There are more advanced techniques like generation-based fuzzing and mutation-based fuzzing which are detailed further below.

A good example of a CAN fuzzer is the popular caringcaribou automotive security exploration tool. This tool has a lot of helpful functionalities like identifying UDS servers, UDS services and interacting with ECUs directly. Additionally, the CAN fuzzer tool makes it easy to quickly replay previous fuzzing activities.


Mutation Based Fuzzing

Mutation based fuzzing is when there’s a known template structure of the data to be sent and this gets mutated at each time when the data is sent. This is also known as a “dumb” fuzzer due to the reason that it just changes the structure of the data blindly. The simplest definition can be found at fuzzingbook.org – “introducing small changes to existing inputs that may still keep the input valid, yet exercise new behavior.” A good example of a mutation based fuzzer is radamsa, which is “used to test how well a program can withstand malformed and potentially malicious inputs.”

Generation Based Fuzzing

Generation fuzzers or “smart” fuzzers are capable of generating test cases from predefined templates or models rather than mutating existing inputs. Taking the example of sending random bytes – this can be done much smarter with a generation based fuzzer by knowing the best data values and combining them together to create an even better value that has more probability of making the system / device behave unexpectedly. AFL++ and Peach fuzzer are good examples of generation based fuzzers, they also have mutation based fuzzing capabilities too!

Types of Fuzz Testing

Black Box Testing

Testers have no knowledge of the internal workings of the software being tested. They treat the software as unknown and focus solely on providing inputs and observing outputs.

White Box Testing

Testers have access to the internal workings of the software, including the source code, architecture, and design. This knowledge is used to guide the fuzzing process, tailor test inputs, and analyze the results.

Advantages of Fuzz Testing

Automation: Fuzzing can be automated which makes it easier to test a complex system / device.
Scalability: Fuzzing can be done in large volumes and can be done on multiple system(s) / device(s) at once. The time to complete a fuzz test depends on the number of data values in the dataset and the number of vectors for input.
Uncovering Vulnerabilities: Since fuzzing is majorly an automated process, it can uncover various types of vulnerabilities, including buffer overflows, memory leaks, and input validation errors.

Disadvantages of Fuzz Testing

False Positives: Fuzzing can generate a large number of test cases, leading to false positives where potential vulnerabilities identified by the fuzzer are not actual security issues, requiring additional time and resources for validation.
Complexity of Inputs: Fuzzing may struggle to generate complex inputs that accurately reflect real-world usage scenarios, especially for software systems with highly structured and proprietary protocols implemented.
Resource Intensive: Fuzzing can be resource-intensive, requiring significant computational power and storage capacity to run large-scale fuzzers.

Understanding Penetration Testing

Penetration Testing is a proactive cybersecurity assessment methodology used to evaluate the security posture of a system / device. Security professionals imitate the tactics and techniques of that of an attacker to discover weaknesses and report them to the concerned authorities. The objectives tested generally contain applications, communication networks and security-critical systems. Penetration testing should not be considered as a one and done process, it should be a continuously repeated process which is to be done during and after the development lifecycle. Penetration tests help to identify and mitigate security weaknesses before threat actors identify and leverage these weaknesses for their benefit.

There are 5 core stages of a penetration test:

  1. Recon: The information regarding the target and the environment is researched and identified. This can be public knowledge or the information provided by the vendors themselves.
  2. Scanning: Automated and manual scanning include in depth technical analysis of the system through network mapping, port scanning and dynamic analysis.
  3. Vulnerability Assessment: Identify the potential entry points of the system and their associated weaknesses.
  4. Exploitation: The attempted exploitation of the identified weaknesses to provide a proof of concept of how the vulnerability could be used in the wild.
  5. Reporting: The detailed and step-by-step explanation of the test cases executed and any associated findings. Findings should include Critical, High, Medium, Low and Informational ratings to assist in understanding a priority for remediation.

Types of Penetration Tests

Black Box Testing

Tester has no prior knowledge of the internal workings of the system being tested. They approach the assessment from the perspective of an external attacker with limited information about the target.

White Box Testing

Tester has full knowledge of the internal workings of the system being tested, including access to source code, network diagrams, and other documentation. They assess the system with an understanding of its architecture, design and complete access.

Grey Box Testing

Combines elements of both black box and white box testing. Testers have some knowledge of the internal workings of the system, but not to the extent provided in white box testing. This partial knowledge allows testers to focus their efforts while still simulating aspects of an external attack.

Advantages of Penetration Testing

Real-world Simulation: Provides a realistic assessment of an organization’s security posture and enables organizations to understand their vulnerabilities in a practical context.
Targeted Approach: Can be tailored to focus on specific areas of concern within a specific module or vehicle allowing organizations to prioritize their security efforts effectively.
Actionable Recommendations: For improving security controls and mitigating identified vulnerabilities, Penetration Tests assist organizations in addressing security issues proactively.

Disadvantages of Penetration Testing

Point-in-Time Assessment: Organizations should not get comfortable because they’ve conducted a penetration assessment as it’s only a snapshot of the security posture of a specific module or a vehicle at a specific point in time. As security threats and vulnerabilities are constantly evolving and changing, periodic assessments shall be conducted to ensure the robustness of the Device under Test (DUT).
Impact on Operations: Can potentially disrupt normal operations or cause downtime if not conducted carefully; requiring coordination with IT and security teams to minimize the impact on business operations.
Authorization Requirements: Requires authorization from relevant stakeholders and may involve legal and regulatory considerations, which can sometimes be time consuming.

Block Harbor Vehicle Cybersecurity Lab (VCL)

Service Offerings

Block Harbor Automotive Cybersecurity offers a comprehensive suite of security services tailored to meet your specific needs. From fuzz testing and penetration testing to a range of other specialized security solutions, we’re dedicated to safeguarding your automotive systems against evolving cyber threats.

References

  1. https://www.code-intelligence.com/blog/roadmap-to-successful-fuzz-testing
  2. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9736493/pdf/sensors-22-09211.pdf
  3. https://peachtech.gitlab.io/peach-fuzzer-community/
  4. https://www.sciencedirect.com/topics/computer-science/fuzzing
  5. https://www.infosecinstitute.com/resources/hacking/fuzzing-mutation-vs-generation/
  6. https://www.fuzzingbook.org/html/MutationFuzzer.html
  7. https://github.com/CaringCaribou/caringcaribou
  8. https://www.cloudflare.com/learning/security/glossary/what-is-penetration-testing/
  9. https://insights.integrity360.com/what-are-the-5-stages-of-penetration-testing
  10. https://www.redscan.com/news/types-of-pen-testing-white-box-black-box-and-everything-in-between/