That is a different sort of function - here you are writing code where the pointer is sometimes null, and sometimes non-null. An "assert" is therefore inappropriate for testing, because both values are valid. It depends on the code whether you want to have the test inside "probe_subtree" or inside the caller, but either way the test is part of the algorithm, not a check for argument validity.
There should not be a grey area here. Arguments are either within the specification, or they are not. It's okay to treat different argument values differently, and return special codes in some cases - that is part of the specification of your function.
You have two choices here. One is to specify that the function will take two values that can be summed to an eight digit number, and it will return that sum. The other is to specify that the function will take two values, and will either return their sum if it fits within 8 digits, or FAIL if it exceeds 8 digits.
That's your functionality specified. You can use asserts to check the input specifications if you want.
What you can't do is claim the function can take any eight digit inputs, then crash or return surprise values because their sum takes more than eight digits.