Why is my CAN-FD setup failing to handshake?

Before you touch any clock or timing setup, go look at the transceiver chip between the STM32 and the bus. A lot of the common CAN transceivers have a standby or shutdown pin, and on several of them it floats high by default, which puts the chip to sleep. The microcontroller can transmit all day long and nothing ever reaches the wire. I lost a whole weekend to exactly that once. Check the datasheet for what that pin needs to be in normal mode, and tie it there, not left floating.

Second thing, and this one matters: the pull-up and pull-down you put on the differential lines. CAN does not work like I2C. CANH and CANL do not get individual resistors to a rail. What the bus wants is a single 120 ohm resistor across the two lines, and only at the two physical ends of the network, nowhere else. If you've got pull-ups or pull-downs hanging on those lines, they can acutally fight the transceiver and distort the signal. Go back and check what you actually placed there.

Third, the bit timing. CAN-FD runs at one speed to start a message and can switch to a much faster speed for the payload. If your starting rate or sample point doesn't match the rest of the network, you'll get silence back even though a scope shows nice clean edges. Get the exact settings from whoever runs the network, or put a scope on a frame and measure the width of the first bits to match it. One more trick: most of these chips have a loopback mode where the controller talks to itself without the pins. Turn that on in firmware. If that works, the problem is downstream of the controller, which points right back at the transceiver or how you wired it.

Solid rails rule out the boring stuff, so it's almost certainly one of the three above. My money's on the standby pin.

I think you should check the termination impedance even if the resistors are correct. I once spent hours trying to get my CAN bus working but realized the bus capacitance was dragging the signal too low. It took me using a dedicated oscilloscope probe and seeing the ringing on the lines to realize the issue. Even with the 120 ohm resistor in place, if the traces are too long or the termination is messy, the chip won't send valid data packets. Did you happen to calculate the total cable and PCB trace length? That might be where the real trouble is.

I completly agree about checking the transceiver standby pin first. That is honestly the quickest way to lose time on these projects. When I ran into this issue myself on a diferent board, the chip wasn't just floating, it needed a definite pull-up to power to even recognize the bus. My fix was just adding a 10k pull-up resistor on that specific shutdown pin, and communication started working immediately. It is so easy to forget that one critical pin when you are focused on the data lines and clock signals.

I dealt with a problem like this once where the voltage looked fine but the chip still wouldn't talk. It turned out the power was just too noisy. There was alot of ripple coming thru my ground and power wires, which messed up the signal. I fixed it by adding a big capacitor right next to the brain. If the power is clean enough, does the placement of the capacitors really matter that much on a small circuit?

I ran into this exact same silence issue on a project a while back. Everything seemed totally fine with the chips, but the data wasn't hitting the host. I realized my traces for the differential lines were running way too long. This caused signal reflection right back into the driver, meaning the board was transmitting a bad signal into the bus. I had to keep those traces as short as possable and make sure they were routed closely togather to minimize any crosstalk. Even a few extra millimeters of length seemed to make a big difference in stability.