The purpose of swap is to balance memory pressure between the page cache and anonymous pages, so when memory pressure occurs the kernel has the ability to evict not-recently-used anon memory instead of just the page cache.
The page cache intentionally keeps a hold of file pages until reclaim so we can avoid disk I/O as much as possible, therefore naturally RAM usage grows over time.
Without swap the kernel has to ALWAYS evict page cache pages meaning that you might end up having to do a bunch of disk I/O that you might not otherwise have to have done.
TL;DR: Swap lets the kernel use memory for what is most in-demand, whether that is anonymous memory or cached file memory. Turning it off means that stale anonymous memory is ALWAYS preferred over cached file memory even if that doesn't make sense.
EDIT: Didn't realise Chris was on fedi - @cdown - incredible article, love it!
ljrk