One need not fully understand relativity, though, to play with it as a plot device. I can't claim to understand it completely, but I do understand what it does if people travel at certain speeds.
So far I have not seen the term "Hard SF" in this thread, and that's the type of SF that concerns itself with scientific accuracy, and that strain appeared after the genre was established, in the golden age.