c++ – Dilemma over authenticity of gcov generated code coverage percentage where unit tests are not technically correct

When I joined my company as a new comer and I was exploring the unit test suite of the product code. It is using gtest framework. But when I checked all the tests, they were testing the whole functionality by calling real functions and asserting expected output. Below is one such test case as an example:

TEST(nle_26, UriExt1)
    int threadid = 1;
    std::shared_ptr<LSEng> e = std::make_shared<aseng:: LSEng >(threadid, "./daemon.conf");
    std::shared_ptr<LSAttrib> attr = e->initDefaultLSAttrib();
    e->setLSAttrib( attr );
    std::shared_ptr<DBOwner> ndb = e->initDatabase(datafile,e->getLogger());

    std::shared_ptr<NewMessage> m = std::make_shared<NewMessage>(e->getLogger());
    ASSERT_TRUE(m != nullptr);
    ASSERT_TRUE(e != nullptr);
    e->scanMsg(m, &scan_callBack_26, NULL);
    std::map<std::string, std::vector<std::string>> Parts = e->verboseInfo.eventParts;
    std::vector<std::string> uris = Parts("prt.uri");
    ASSERT_EQ(uris.size(), 2 );
    ASSERT_EQ(uris(0) , "mailto:www.us_megalotoliveclaim@hotmail.com");
    ASSERT_EQ(uris(1) , "hotmail.com");

I found all the tests in the unit test directory having the same pattern like:

  1. Creating and initialising actual object
  2. Calling actual function
  3. Starting actual daemon
  4. Loading actual database of size around 45MB
  5. Sending actual mail for parsing to daemon by calling actual scanMsg function, etc.

So, all the tests appear more of as functional tests, rather than unit tests.

But, the critical part is, on their official intranet site, they have projected the code coverage percentage of this product as 73%, computed using gcov.

Now, code profiling tools like gcov computes coverage on the following params:

  1. How often each line of code executes
  2. What lines of code are actually executed
  3. How much computing time each section of code uses.

As, these tests are running actual daemon, loading real database and calling actual functions to scan the message, of course, above 3 params will play some role in it, so I doubt it will be completely zero.

But my bothering questions are:

  1. Black box testing also does functional testing just as this, so what’s the difference between above and functional test?. In blackbox, testers unaware of the inside code, writes test cases to test the functionalities specific to requirements. How above such kind of tests are different than that? So does gcov generated coverage on this test suite, can be trusted or misleading?

  2. Apparently, gcov code coverage data is based on test suite with all technically incorrect unit tests, does it mean the actual code coverage may be even zero?

  3. In unit test, we mock function calls using google mock-like framework rather than calling actual calls Purpose of unit test is to test the code itself, by smallest unit wise. But above tests, seemingly more like functional tests, can gcov generate reliable code coverage data based on it??

This is haunting me for last two days. So thought to serve on the table for experts.

Awaiting wonderful insights 🙂