A good indication for me to DNF a book is if it feels like homework. If I'm just reading it to finish and be done and I'm not getting anything out of the book, then I may as well just move on. Shadowland by Peter Straub -- which some have recommended as his best work -- was like that for me recently.