Using CMake, GoogleTests and gcovr in a C project
This article presents a simple way to setup a C project using CMake, GoogleTest and gcovr. Setting all together and making them work can be a bit tricky, so I will try to explain the steps to make it work.
Project Hierarchy
Our project will only contain one module and one function for the sake of simplicity. The hierarchy of the project will be as follows:
hello.git/
├── CMakeLists.txt
├── LICENSE.md
├── README.md
├── include/
│ └── utils.h
├── src/
| ├── CMakeLists.txt
│ ├── hello.c
│ └── utils.c
└── tests/
├── CMakeLists.txt
├── hello_test.c
└── utils_test.c
Our main program will be in src/hello.c
and the corresponding test in
tests/hello_test.c
. The utils.c
and utils.h
files will contain some
utility functions that will be tested in tests/utils_test.c
.
The Code
We provide two simple functions in utils.c
and utils.h
: max()
and
print_hello()
. Then, hello.c
is the main program that run on console.
The content of hello.c
is the following:
#include <stdio.h>
#include <stdlib.h>
#include <utils.h>
int main (int argc, char *argv[]) {
if (argc == 1) {
printf ("Hello World!\n");
return EXIT_SUCCESS;
}
print_hello (atoi (argv[1]));
return EXIT_SUCCESS;
}
The content of utils.c
is the following:
#include "utils.h"
#include <stdio.h>
int max (int a, int b)
{
if (a > b)
return a;
return b;
}
void print_hello (int count)
{
for (int i = 0; i < count; i++)
printf ("Hello World!\n");
}
And, the content of utils.h
is the following:
#ifndef UTILS_H
#define UTILS_H
/* Returns the max of 'a' and 'b' */
int max (int a, int b);
/* Prints "Hello, world!" 'count' times on stdout */
void print_hello (int count);
#endif
Build-System (CMake)
Set up the CMakeLists.txt files requires to have a CMakeLists.txt
file in the
root directory of the project and one in src/
in charge of building the main program.
The main CMakeLists.txt
file is the following:
# Minimum version of CMake required to build this project
cmake_minimum_required(VERSION 3.15)
# Name of the project
project(hello LANGUAGES C)
# Set the C standard to C11
set(CMAKE_C_STANDARD 11)
# Add subdirectories
add_subdirectory(src)
The src/CMakeLists.txt
file is the following:
# Include directories
include_directories(${CMAKE_SOURCE_DIR}/include)
# Move executables to the root of the build directory
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR})
# Add target and source files
add_executable(hello hello.c utils.c)
Now you can build the project by running the following commands:
#> mkdir build
#> cd build
#> cmake ..
#> make
The point of having the binary files in a build/
directory is to keep the
source directory clean. It also allows to manage several build directories for
different configurations (e.g., debug, release, etc.). CMake has several default
build types such as Debug
and Release
. You can specify the build type by
running cmake -DCMAKE_BUILD_TYPE=Debug ..
or cmake
-DCMAKE_BUILD_TYPE=Release ..
. So, for example, you could do:
#> mkdir release
#> cd release
#> cmake -DCMAKE_BUILD_TYPE=Release ..
#> make
#> cd ../
#> mkdir debug
#> cd debug
#> cmake -DCMAKE_BUILD_TYPE=Debug ..
#> make
Thus, you can have two different build directories for a build in debug mode and another in release mode.
Finally, note also that we usually create the build/
directory at the root of
the sources but it could be anywhere else.
Testing (GoogleTest)
Lets now add the GoogleTest framework to our project. We will use CMake to install it locally to our project and avoid any system-wide installation.
One may notice that the GoogleTest framework is in C++ and we are using C. This is not a problem because we can easily mix C and C++ code in the test files. Yet, this requires to have a C++ compiler installed on the system and to learn a bit C++.
Lets first get the GoogleTest framework installed in our project through CMake
(to be added in the main CMakeLists.txt
file):
...
# Name of the project
project(hello LANGUAGES C CXX) # We require C++ for GoogleTest
# Set the C standard to C11
set(CMAKE_C_STANDARD 11)
# Set the C++ standard to C++17
set(CMAKE_CXX_STANDARD 17)
# Fetch GoogleTest framework and build it
include(FetchContent)
FetchContent_Declare(
googletest
URL https://github.com/google/googletest/releases/download/v1.15.2/googletest-1.15.2.tar.gz
PREFIX ${CMAKE_CURRENT_BINARY_DIR}/gtest
DOWNLOAD_EXTRACT_TIMESTAMP TRUE
DOWNLOAD_NO_PROGRESS TRUE
)
set(gtest_force_shared_crt ON CACHE BOOL "" FORCE)
FetchContent_MakeAvailable(googletest)
...
So, the FetchContent_Declare()
function is used to download the GoogleTest
from github and the FetchContent_MakeAvailable()
function is used to build it.
Now, lets add some tests to our project in the tests/
directory. First, we
test utils
functions in tests/utils_test.c
:
#include <gtest/gtest.h>
extern "C" { /* Because our code is in C */
#include <utils.h>
}
/* Tests on utils max() function */
TEST (UtilsTest, MaxTest) {
EXPECT_EQ (max (1, 2), 2);
EXPECT_EQ (max (2, 1), 2);
EXPECT_EQ (max (1, 1), 1);
}
/* Tests on utils print_hello() function */
TEST (UtilsTest, PrintHelloTest) {
testing::internal::CaptureStdout ();
print_hello (0);
EXPECT_STREQ (testing::internal::GetCapturedStdout ().c_str (), "");
testing::internal::CaptureStdout ();
print_hello (1);
EXPECT_STREQ (testing::internal::GetCapturedStdout ().c_str (),
"Hello World!\n");
testing::internal::CaptureStdout ();
print_hello (2);
EXPECT_STREQ (testing::internal::GetCapturedStdout ().c_str (),
"Hello World!\nHello World!\n");
testing::internal::CaptureStdout ();
print_hello (-1);
EXPECT_STREQ (testing::internal::GetCapturedStdout ().c_str (), "");
}
And, then, we test the hello
program in tests/hello_test.c
:
#include <gtest/gtest.h>
#define BUF_SIZE 128
/* Helper function to execute a command and capture its output */
std::string execute_command(const std::string& command) {
std::ostringstream output;
std::array<char, BUF_SIZE> buffer;
FILE* pipe = popen(command.c_str(), "r");
if (!pipe)
throw std::runtime_error("popen() failed!");
while (fgets(buffer.data(), buffer.size(), pipe) != nullptr)
output << buffer.data();
pclose(pipe);
return output.str();
}
/* Tests on main program */
TEST (HelloTest, MainTest) {
std::string bindir = BINDIR;
std::string command = bindir + std::string("/hello");
EXPECT_STREQ (execute_command(command).c_str(), "Hello World!\n");
command = bindir + std::string("/hello 2");
EXPECT_STREQ(execute_command(command).c_str(), "Hello World!\nHello World!\n");
command = bindir + std::string("/hello A");
EXPECT_STREQ(execute_command(command).c_str(), "");
}
Here, we define an helper function execute_command()
to run the hello
program and check how it behaves on various inputs. Note also that to find the
path to the executable hello
, we use a BINDIR
variable that we will define
in the CMakeLists.txt
file in the tests/
directory.
And, here is the content of the tests/CMakeLists.txt
file:
# Include subdirectories
include_directories(${GTEST_INCLUDE_DIRS} ${CMAKE_SOURCE_DIR}/include)
# hello_tests (tests on the 'hello' binary)
## Set the binary directory as a macro
add_compile_definitions(BINDIR="${CMAKE_BINARY_DIR}")
## Add the tests
add_executable(hello_tests hello_tests.cc)
target_link_libraries(hello_tests GTest::gtest_main)
add_test(NAME hello_tests COMMAND hello_tests)
# utils_tests (tests on the 'utils' module)
## Add the tests
add_executable(utils_tests utils_tests.cc ${CMAKE_SOURCE_DIR}/src/utils.c)
target_link_libraries(utils_tests GTest::gtest_main)
add_test(NAME utils_tests COMMAND utils_tests)
A test requires to build an executable and to link it with the GoogleTest. Then,
we add the test to the list of tests to run with the add_test()
function.
Note that we chose to name the test and the executable the same way. This is not mandatory but it makes things simpler.
Then we need to add to the main CMakeLists.txt
file the following lines:
# Tests
include(CTest)
include(GoogleTest)
enable_testing()
# Add subdirectories
add_subdirectory(src)
add_subdirectory(tests)
Now, you can run the tests by running the following commands inside the build directory:
#> ctest
#> ctest --verbose
#> make test
Code Coverage (gcovr)
It is now time to add code coverage to our project. We use the gcovr
framework
which is a Python module that uses the GNU gcov
tool to produce code coverage reports.
The coverage is achieved by instrumenting the code with the --coverage
flag
when compiling and linking the program. The gcovr
script is then used to
generate the report.
First, we need to add a custom target to the main CMakeLists.txt
file to run
the gcovr
script:
...
# Add a custom target for generating coverage reports
if (BUILD_TYPE STREQUAL "Coverage") # Check if the build type is 'Coverage'
find_package(Python3 REQUIRED COMPONENTS Interpreter) # Required for gcovr
add_custom_target(coverage
COMMAND ${CMAKE_COMMAND} -E make_directory coverage
COMMAND ${Python3_EXECUTABLE} -m gcovr -r ${CMAKE_SOURCE_DIR} --html --html-nested -o coverage/index.html
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
COMMENT "Generating HTML coverage report..."
)
endif()
...
Note that we add this target only if the build type is Coverage
. This is to avoid to keep the coverage instrumentation in the other build types. Coverage is only needed in development and the instrumentation embedded in the executable may slow it down.
Then, to add the compilation flag --coverage
to the project, we need to modify
the src/CMakeLists.txt
file as follows:
# Enable coverage flags for the Coverage build type on GCC/Clang
if(CMAKE_C_COMPILER_ID MATCHES "GNU|Clang")
set(CMAKE_C_FLAGS_COVERAGE "${CMAKE_C_FLAGS_DEBUG} --coverage")
set(CMAKE_EXE_LINKER_FLAGS_COVERAGE "${CMAKE_EXE_LINKER_FLAGS_DEBUG} --coverage")
set(CMAKE_SHARED_LINKER_FLAGS_COVERAGE "${CMAKE_SHARED_LINKER_FLAGS_DEBUG} --coverage")
set(CMAKE_MODULE_LINKER_FLAGS_COVERAGE "${CMAKE_MODULE_LINKER_FLAGS_DEBUG} --coverage")
endif()
Then, to have a complete coverage report, we need to run the following commands:
#> mkdir coverage
#> cd coverage
#> cmake -DCMAKE_BUILD_TYPE=Coverage ..
#> make
#> make test
#> make coverage
#> firefox coverage/index.html
The make coverage
command will generate the coverage report in the coverage/
directory. You can then open the index.html
file in a web browser to see the
coverage report.
Gathering all together
We modified quite a lot the CMakelists files to add GoogleTest and gcovr to our project, here is the final version of the main CMakeLists.txt
file:
# Minimum version of CMake required to build this project
cmake_minimum_required(VERSION 3.15)
# Name of the project
project(hello LANGUAGES C CXX)
# Set the C standard to C11
set(CMAKE_C_STANDARD 11)
# Set the C++ standard to C++17
set(CMAKE_CXX_STANDARD 17)
# Fetch GoogleTest framework and build it
include(FetchContent)
FetchContent_Declare(
googletest
URL https://github.com/google/googletest/releases/download/v1.15.2/googletest-1.15.2.tar.gz
PREFIX ${CMAKE_CURRENT_BINARY_DIR}/gtest
DOWNLOAD_EXTRACT_TIMESTAMP TRUE
DOWNLOAD_NO_PROGRESS TRUE
)
set(gtest_force_shared_crt ON CACHE BOOL "" FORCE)
FetchContent_MakeAvailable(googletest)
# Tests
include(CTest)
include(GoogleTest)
enable_testing()
# Add a custom target for generating coverage reports
if (BUILD_TYPE STREQUAL "Coverage") # Check if the build type is 'Coverage'
find_package(Python3 REQUIRED COMPONENTS Interpreter) # Required for gcovr
add_custom_target(coverage
COMMAND ${CMAKE_COMMAND} -E make_directory coverage
COMMAND ${Python3_EXECUTABLE} -m gcovr -r ${CMAKE_SOURCE_DIR} --html --html-nested -o coverage/index.html
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
COMMENT "Generating HTML coverage report..."
)
endif()
# Add subdirectories
add_subdirectory(src)
add_subdirectory(tests)
And, here is the final version of the src/CMakeLists.txt
file:
# Set the compiler flags on the Coverage build-type
if(CMAKE_C_COMPILER_ID MATCHES "GNU|Clang")
set(CMAKE_C_FLAGS_COVERAGE "${CMAKE_C_FLAGS_DEBUG} --coverage")
set(CMAKE_EXE_LINKER_FLAGS_COVERAGE "${CMAKE_EXE_LINKER_FLAGS_DEBUG} --coverage")
set(CMAKE_SHARED_LINKER_FLAGS_COVERAGE "${CMAKE_SHARED_LINKER_FLAGS_DEBUG} --coverage")
set(CMAKE_MODULE_LINKER_FLAGS_COVERAGE "${CMAKE_MODULE_LINKER_FLAGS_DEBUG} --coverage")
endif()
# Include directories
include_directories(${CMAKE_SOURCE_DIR}/include)
# Move executables to the root of the build directory
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR})
# Add target and source files
add_executable(hello hello.c utils.c)
And, finally, here is the final version of the tests/CMakeLists.txt
file:
# Include subdirectories
include_directories(${GTEST_INCLUDE_DIRS} ${CMAKE_SOURCE_DIR}/include)
# Set the compiler flags on the Coverage build-type
if(CMAKE_C_COMPILER_ID MATCHES "GNU|Clang")
set(CMAKE_C_FLAGS_COVERAGE "${CMAKE_C_FLAGS_DEBUG} --coverage")
set(CMAKE_EXE_LINKER_FLAGS_COVERAGE "${CMAKE_EXE_LINKER_FLAGS_DEBUG} --coverage")
set(CMAKE_SHARED_LINKER_FLAGS_COVERAGE "${CMAKE_SHARED_LINKER_FLAGS_DEBUG} --coverage")
set(CMAKE_MODULE_LINKER_FLAGS_COVERAGE "${CMAKE_MODULE_LINKER_FLAGS_DEBUG} --coverage")
endif()
# hello_tests (tests on the 'hello' binary)
## Set the binary directory as a macro
add_compile_definitions(BINDIR="${CMAKE_BINARY_DIR}")
## Add the tests
add_executable(hello_tests hello_tests.cc)
target_link_libraries(hello_tests GTest::gtest_main)
add_test(NAME hello_tests COMMAND hello_tests)
# utils_tests (tests on the 'utils' module)
## Add the tests
add_executable(utils_tests utils_tests.cc ${CMAKE_SOURCE_DIR}/src/utils.c)
target_link_libraries(utils_tests GTest::gtest_main)
add_test(NAME utils_tests COMMAND utils_tests)
Coding style with clang-format
Now, we can add a coding style check to our project using clang-format
. We
first have to select a base style among: LLVM, GNU, Google, Chromium, Microsoft, Mozilla and WebKit. Then, create a .clang-format
file at the root of the project with the following content:
---
Language: Cpp
BasedOnStyle: GNU
...
Then, we can run the following command to conform to the style of the code:
#> clang-format -i $(find . -name '*.[ch]')
If you want to tweak the base style, you can run the following command to dump a
more detailed configuration in the .clang-format
file (see here for the full list of options):
#> clang-format --style=GNU --dump-config > .clang-format
Note that once the .clang-format
file is created, the clang-format
command
will use it by default. Moreover, you can configure your editor to use it and
all the other developers will have the same coding style. That is why it is
considered a good practice to have a .clang-format
file at the root directory
of the project.
Static analysis with clang-tidy
Another useful tool to add to the project is clang-tidy
. It is a static
analysis tool based on the clang
compiler that checks the code for potential
bugs and stylistic issues. To use it, you need to have a .clang-tidy
file at
the root of the project with the following content (see
here for the main categories of
checks and here for
the full list of available checks):
---
Checks: '-*,clang-analyzer-*,readability-identifier-length'
...
Basically, the Checks
field is a list of checks to run. The -*
check
disables all checks and clang-analyzer-*
enables all the checks from the
clang-analyzer
module.
Then, you can run the following command to check the code:
#> clang-tidy $(find . -name '*.[ch]') -- -std=c11 -Iinclude
[1/3] Processing file include/utils.h.
[2/3] Processing file src/hello.c.
[3/3] Processing file src/utils.c.
16 warnings generated.
include/utils.h:5:13: warning: parameter name 'a' is too short, expected at least 3 characters [readability-identifier-length]
5 | int max(int a, int b);
| ^
include/utils.h:5:20: warning: parameter name 'b' is too short, expected at least 3 characters [readability-identifier-length]
5 | int max(int a, int b);
| ^
src/utils.c:5:13: warning: parameter name 'a' is too short, expected at least 3 characters [readability-identifier-length]
5 | int max(int a, int b) {
| ^
src/utils.c:5:20: warning: parameter name 'b' is too short, expected at least 3 characters [readability-identifier-length]
5 | int max(int a, int b) {
| ^
Suppressed 12 warnings (12 in non-user code).
Use -header-filter=.* to display errors from all non-system headers. Use -system-headers to display errors from system headers as well.
The ‘--
‘ option is used to mark the start of the options passed to the
compiler. Here, we pass the ‘-std=c11
‘ option to specify the C standard and
the ‘-Iinclude
‘ option to specify the include directory.
Of course, if you remove the readability-identifier-length
check from the
.clang-tidy
file, you will not have the warnings about the parameter names
being too short. And you can locally disable a check by adding a comment in the code:
int max(int a, int b) { // NOLINT(readability-identifier-length)
...
}
Finally, if you want to run clang-tidy
with all the checks enabled, you can
simply replace your .clang-tidy
file with:
---
Checks: '*'
...
But, you will need to sort out the warnings and disable the ones that are not relevant to your project. This is why it is better to start with a minimal set of checks and then add the ones that are relevant to your project little by little.
Fuzzing
Another way to reduce the number of bugs inside a C code is to try to fuzz it with some advanced fuzzer such as AFL++ (American Fuzzy Lop). I will not explain here the principle of a code coverage guided fuzzer. We will just learn how to do the fuzzing through CMake in an easy way.
First of all, we need to create a build directory with specific build-options in order to instrument our binary to let AFL++ know about the execution paths taken inside our binary:
#> mkdir build-afl/
#> cd build-afl/
#> cmake -DCMAKE_C_COMPILER=afl-gcc -DCMAKE_C_FLAGS="-fsanitize=address,undefined -fno-sanitize-recover=all -g" -DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address,undefined -fno-sanitize-recover=all" -DCMAKE_MODULE_LINKER_FLAGS="-fsanitize=address,undefined -fno-sanitize-recover=all" -DCMAKE_BUILD_TYPE=Debug,ASAN,UBSAN -DBUILD_SHARED_LIBS=OFF ..
#> make
Note that we also added compilation flags to enable a memory sanitizer, this helps to force a crash when a memory violation is encountered. Indeed, a fuzzer can only detect crashes, so adding the sanitizer helps us to capture more bad behaviours in our program.
Then, we need to create an input/
and an output/
directories to
give to the fuzzer a collection of typical inputs that could be fitted
to the stdin
of our program and to collect the outputs of the fuzzer.
#> mkdir input output
#> echo -e "aaa\naaa\n" > input/sample.txt
Now, we are ready to start. We will first try to fuzz the stdin
of our
program. Here is the way to do it:
#> afl-fuzz -i input/ -o output/ -- ./hello
Note that you may be asked by AFL++ to set your kernel in some specific modes in order to get the best of it during the fuzzing phase. Usually, AFL++ issue quite explicit messages about it and give the needed command to execute. One that is almost always needed to execute is the following (yes, you’d better to be root on the fuzzing machine):
#> cd /sys/devices/system/cpu
#> echo performance | sudo tee cpu*/cpufreq/scaling_governor
Once all is set and that you managed to start the fuzzing, you should see the following panel:
Congratulation, you got your first AFL++ session running!
For now, you just fuzz the stdin of your software. But, if you want to fuzz some input files given on the command line, you can do:
#> afl-fuzz -i input/ -o output/ -- ./hello --file @@
Here, the @@
is a placeholder for AFL++ to fit the fuzzed file and
feed your program with.
Finally, you will find all the crashes encountered in the
output/default/crashes/
directory. Try it on your software to see if
you encounter a real bug and solve it.
Of course, AFL++ can do much more but this is another story…