Now that I've seen the price of Elastic Cloud (and no I don't want to deal with Java software on my servers), I really want to build a sane Open Source log storage platform. Store raw data on a S3 or on a NAS, treat indices are disposable, and split in shards to scale. I need more time.
@galdor@bahmanm Fully agree, the whole log storage solution spectrum is highly overcommercialized and overpriced. Their UIs are mostly bloated and slow. And I don't want to learn yet another "for-purposes-of-lock-in-invented" *QL language to query the logs.
I moved our log storage needs to plain old Postgres and some shell scripts which deprecate old log entries (I don't need long-term storage). It might not be the most efficient solution but it is easy to manage and can be self-run on some cheap VMs.
Datadog et al. are clearly more suitable for "enterprise" clients who have excess money to burn.
@galdor Btw. MySQL/MariaDB have a "CSV" and "ARCHIVE" table engine and MariaDB also supports native S3 storage for table data. Might be a solution for you?
@louis@bahmanm I learned years ago that a company cannot make decent money on a product with "just" log storage/search; pricing is dominated by storage cost, it's just not interesting. Companies doing just that end up sold and integrated to some kind of platform (e.g. Logmatic bought by Datadog). This is also why Elastic is so much more than just logs.
Relational databases work for tiny volumes, but it's not the right tool for the job (good luck storing gigabytes of logs a day with months of history, and yes this is a small number). And you still have to write a decent web interface.
Add comment