Computer-assisted studies of fiction tend to count words and trace themes. It's harder to measure the things that really keep readers turning pages: mystery, suspense, and surprise.
@TedUnderwood the range of “divergence from summaries” in the plot vs name_cloze accuracy is .095 to .13. How should I interpret this- is it a big difference or a small one?
@Virginicus Well, you can compare it to the divergence within a book in the earlier figure. The variance between books’ mean difficulties is comparable to the variance between the hardest/easiest passages in any single book.
@Virginicus yes, although the range between outliers in Hound may also be partly just having more data points — I had to do four passes on it versus one in the other illustration
If we got this method to work, the payoff might be that we could measure the creation of mystery, and distant-read things like the history of the cliffhanger. Here's a toy example with Conan Doyle's Hound of the Baskervilles.
I find that uncertainty is higher at the end of serial installments than at other chapter breaks. (The "tune in next week" effect.)
Very cool. I think there's lot of promise in having LLMs "read" texts like this and using it to analyze the text itself.
Can I ask why you went with the distance in the embedding space? The model itself already produces probabilities, so you could check the probability of the truth under the model distribution (if this is available in the GPT API), or just the entropy of the predicted part (alhtough NNs are notoriously bad an producing well-calibrated uncertainty).
Add comment