I questioned this too, knowing that every so many years (I think they say about 7 years), every cell in a human's body is replaced. So without repeated exposure, it would diminish, and eventually be gone.
In addition, Selenium is actually required by the human body in small amounts. Animals particularly livestock need higher amounts. Of course this needs to be ingested, not inhaled from being burned.
Of all the old electronic gear I worked on when I was much younger, I never had a selenium rectifier burn up. I probably still have a dozen or so of them in my old parts boxes. I used to save all that stuff, as long as it was still in good shape.