Instead of enabling Evil mode globally, I now have it turned on only in buffers with major modes derived from prog-mode or text-mode.
Since Evil is disabled in non-editing buffers like Magit and Dired (where it was a nuisance), I got rid of Evil collection and cut my startup time in half.
