orbitals backend is getting big enough to warrant monitoring so i am investigating that sorta plumbing.
a trillion years ago, when i rode my triceratops to work at the rock smashing gravel pit (websense and then anonymizer) in like 2005, i was quite happy monitoring stuff with cacti. it did the job well and was pretty lightweight.
but now i see .. grafana and prometheus and elastic search and docker images for all those and .. just..
ugh.
do i have to invent the universe just for a graph?
though it is pretty rewarding to throw massive massive orgs into it like cern and disney and watch an absolutely dumptruck of a report come out the other side.
@bensonk i dont want to have to install, configure and maintain three separate pieces of software to get a cpu graph. that setup is incredibly bloated, topheavy and needy for a handful of machines, not to metion the bountiful attack surfaces they all expose
I personally prefer client-push systems like ganglia, graphite, influxdb; but if you can architect your system so Prometheus can pull everything, it’s obnoxiously easy.
@josephholsten@Viss Having seen both in use at scale at work, and having moved from a pull model to a push model, I... am pretty happy with the pull, actually. Prom is much closer to the system I wish we had.
@bensonk@Viss I wonder how much is that I just am terrified of having network acl that allow inbound traffic from a rotating set of IPs in the monitoring cluster. Being received by a lightly maintained support service that usually has at least full read only access to internal service state.
@josephholsten@bensonk i did a training class in oslo last week, and one of the things i showcased was misconfigured publicly accessible prometheus instances leaking all sorts of internal data to anybody who curled the right way
@Viss@josephholsten There is a certain amount of configuration that is absolutely a key piece of any monitoring configuration. If there are systems that let you perform that kind of monitoring without requiring nuanced configuration, I'd love to read about how they work.
@bensonk@josephholsten well for starters, any system that has a default config of listening on every interface with no creds is .. bad. like redis. and prometheus. and docker.
they need extra care to make them un-bad.
and i dont feel like picking up the developers slack for them
@morb im looking into netdata now, i hadnt heard about it before, but their demo site has pretty sexy graphs and it seems to integrate with redis, celery and rabbitmq out of the box, and we rely heavily on those
Add comment