unit tests: support code coverage instrumentation and reports

Currently, make check runs the unit tests and reports pass/fail,
but we have no way to guage how much of the code is covered by
these tests.  gcov provides those statistics on a per source
file basis, but requires special CFLAGS and LDFLAGS.  Here, we
add the --enable-gcov configure option to setup those options
correctly.  We also add a make target called check-coverage,
which runs the unit tests, runs gcov and uploads the data to
the codecov.io cloud service for display.

Finally, we include a Dockerfile-coverage which creates a
container image in alpine linux to run the tests.  To create
the image:

$ docker build \
	--build-arg commit=`git rev-parse HEAD` \
	--build-arg token=<upload token from codecov.io> \
	-t frr-gcov:latest \
	-f docker/alpine/Dockerfile-coverage .

and to create and upload the report:

$ docker run -it --rm frr-gcov:latest

Testing done:

Created and uploaded a report from my fork using alpine linux 3.7.
Non-coverage alpine 3.7 build still works.

Issue: https://github.com/FRRouting/frr/issues/2442
Signed-off-by: Arthur Jones <arthur.jones@riverbed.com>
This commit is contained in:
Arthur Jones 2018-06-14 06:44:38 -07:00
parent 732c8da348
commit 8328420909
4 changed files with 56 additions and 1 deletions

View file

@ -115,3 +115,23 @@ noinst_HEADERS += defaults.h
indent: indent:
tools/indent.py `find sharpd bgpd eigrpd include isisd lib nhrpd ospf6d ospfd pimd qpb ripd vtysh zebra -name '*.[ch]' | grep -v include/linux` tools/indent.py `find sharpd bgpd eigrpd include isisd lib nhrpd ospf6d ospfd pimd qpb ripd vtysh zebra -name '*.[ch]' | grep -v include/linux`
if HAVE_GCOV
coverage: check
@ find . -name '*.o' -exec gcov {} \;
yorn:
@ echo "OK to upload coverage to https://coverage.io [y/N]:"
@ read yn; test "$$yn" = "y"
upload-check-coverage:
@ if [ "x${COMMIT}" = "x" ]; then echo "COMMIT required"; exit 1; fi
@ if [ "x${TOKEN}" = "x" ]; then echo "TOKEN required"; exit 1; fi
curl -s https://codecov.io/bash | bash -s - -C ${COMMIT} -t ${TOKEN}
force-check-coverage: coverage upload-check-coverage
check-coverage: coverage yorn upload-check-coverage
endif

View file

@ -226,7 +226,14 @@ AC_ARG_ENABLE([memory-sanitizer], AS_HELP_STRING([--enable-memory-sanitizer], \
]) ])
dnl if the user has specified any CFLAGS, override our settings dnl if the user has specified any CFLAGS, override our settings
if test "x${enable_dev_build}" = "xyes"; then if test "x${enable_gcov}" = "xyes"; then
if test "z$orig_cflags" = "z"; then
AC_C_FLAG([-coverage])
AC_C_FLAG([-O0])
fi
LDFLAGS="${LDFLAGS} -lgcov"
elif test "x${enable_dev_build}" = "xyes"; then
AC_DEFINE(DEV_BUILD,,Build for development) AC_DEFINE(DEV_BUILD,,Build for development)
if test "z$orig_cflags" = "z"; then if test "z$orig_cflags" = "z"; then
AC_C_FLAG([-g3]) AC_C_FLAG([-g3])
@ -441,6 +448,8 @@ AC_ARG_ENABLE([clippy-only],
AS_HELP_STRING([--enable-clippy-only], [Only build clippy])) AS_HELP_STRING([--enable-clippy-only], [Only build clippy]))
AC_ARG_ENABLE([numeric_version], AC_ARG_ENABLE([numeric_version],
AS_HELP_STRING([--enable-numeric-version], [Only numeric digits allowed in version (for Alpine)])) AS_HELP_STRING([--enable-numeric-version], [Only numeric digits allowed in version (for Alpine)]))
AC_ARG_ENABLE([gcov],
AS_HELP_STRING([--enable-gcov], [Add code coverage information]))
AS_IF([test "${enable_clippy_only}" != "yes"], [ AS_IF([test "${enable_clippy_only}" != "yes"], [
AC_CHECK_HEADERS(json-c/json.h) AC_CHECK_HEADERS(json-c/json.h)
@ -692,6 +701,11 @@ AC_DEFINE_UNQUOTED(MULTIPATH_NUM, $MPATH_NUM, Maximum number of paths for a rout
AC_DEFINE_UNQUOTED(VTYSH_PAGER, "$VTYSH_PAGER", [What pager to use]) AC_DEFINE_UNQUOTED(VTYSH_PAGER, "$VTYSH_PAGER", [What pager to use])
dnl --------------------
dnl Enable code coverage
dnl --------------------
AM_CONDITIONAL([HAVE_GCOV],[test '!' "$enable_gcov" = no])
dnl ------------------------------------ dnl ------------------------------------
dnl Alpine only accepts numeric versions dnl Alpine only accepts numeric versions
dnl ------------------------------------ dnl ------------------------------------

View file

@ -205,6 +205,15 @@ options from the list below.
hardcoded arrays that FRR builds towards, so we need to know how big to hardcoded arrays that FRR builds towards, so we need to know how big to
make these arrays at build time. make these arrays at build time.
.. option:: --enable-gcov
Code coverage reports from gcov require adjustments to the C and LD flags.
With this option, gcov instrumentation is added to the build and coverage
reports are created during execution. The check-coverage make target is
also created to ease report uploading to codecov.io. The upload requires
the COMMIT (git hash) and TOKEN (codecov upload token) environment variables
be set.
You may specify any combination of the above options to the configure You may specify any combination of the above options to the configure
script. By default, the executables are placed in :file:`/usr/local/sbin` script. By default, the executables are placed in :file:`/usr/local/sbin`
and the configuration files in :file:`/usr/local/etc`. The :file:`/usr/local/` and the configuration files in :file:`/usr/local/etc`. The :file:`/usr/local/`

View file

@ -0,0 +1,12 @@
FROM alpine:3.7
ARG commit
ARG token
ENV COMMIT=${commit}
ENV TOKEN=${token}
ADD . /src
RUN cd /src && \
source alpine/APKBUILD.in && \
apk add --no-cache alpine-sdk $makedepends $checkdepends && \
./bootstrap.sh && \
./configure --enable-gcov
ENTRYPOINT [ "/bin/sh", "-c", "cd /src && make && make -j 1 check-coverage" ]