@simonbs creating a dummy account and browsing around for a while (depends how frequently you see the issue) should give you an indication if it’s Safari or potentially some issues with your config. Could be an interplay of extensions, corrupt pref files, something iCloud is doing etc etc
Using a dummy account will be almost the equivalent of a clean install.