For me, the main objective for a unit test framework is to have some syntactic sugar for assertions and, less important, for grouping tests into test suites and stuff.
Do we need assertions? It is surely helpful, if an equality assertion fails, to have expected and actual value presented on a golden plate. On the other hand, getting those values with a debugger isn't much more complex, and you typically fire up a debugger anyway to investigate. In a language such as Java or C++, you can have some higher-level assertions such as "assert-throws" or "assert-container-contains", which save quite a lot of typing. In C, the highest level you could get is something like "assert-string-equals", but the gain compared to plain 'assert(strcmp(...) == 0)' is vanishingly small.
Do we need to group tests? People delight in numbers, "I have 937 unit tests, and of those, 936 pass, look how awesome I am!". But, seriously, even if one test fails, the product is broken and someone needs to investigate, so, for me, unit tests produce a "ok"/"nok" result, so I just need a binary that runs or crashes.
Now, what I don't expect in a unit test framework proper is the whole area of mocking stuff. In languages like Java, how to mock something is pretty straightforward: you use abstract interfaces, factories, etc., and to mock a file system access, you pass a mock FileSystem object. In C, how you can mock - if you mock at all - depends on your codebase. Do you have the equivalent of virtual interfaces? Do you want to play linker tricks? Do you want to play preprocessor tricks? That decides what mocking framework, if any, you can use.
Personally, when I unit-test C, I'm testing computations and logic, which usually don't need mocking. Things like: put in this input, expect this output, from my computational kernel.
I mostly agree with you. Indeed I think it's not important which framework you use, but if you use and how you arrange unit testing at first.
The big issue with C embedded projects is how you organize your build system to (maybe it can be an entire discussion of a new post):
- build Release/Debug for multiple embedded targets and models of final product
- build the same as above, but for host machine (simulator)
- build and run tests on the host
- (eventually) build and run tests on the targets
This is more difficult for embedded because we use different cross-compilers for multiple targets, multiple IDEs from silicon vendors and so on.
This is my main reason why migrate to a Makefile and abandon automatic build system of IDEs from silicon vendors. Starting from a custom and ubiquitous (even if cryptic and ancient) Makefile, I can customize the entire build system for production on real targets, simulators, tests and so on.
However this is very difficult for me and starting from a working and non trivial example is a big bonus.
I found this article very very interesting in this topic. It uses Cpputest in examples (even if the author clearly says that the framework is not important). The author gives a pro to frameworks for code coverage report. I know almost anything on this, so I don't know if code coverage can be reported without a unit test framework.
Mocking is another argument that I will certainly explore. I will start testing simple units that don't have many dependecies and where I can use stubs or fakes, not real mocks.
This will be my starting point too, but I think it is a small part of a non trivial project. Critical and obscure bugs are hidden in complex units that have many dependencies that can't be replaced by simple fakes and stubs during tests, you need full mocks.