The reason for the comment count being off has been found. In Settings you need to have 'Undetermined' in your language selection, or any comment where language wasn't specified becomes invisible (incl. old comments before language was a property on posts).