Jeremiah, (edited )
@Jeremiah@alpaca.gold avatar

I need Mastodon and Kubernetes help:

I am trying to deploy Mastodon to a GCP GKE Autopilot cluster. The Helm chart is failing due to container sysctl calls somewhere getting blocked because GKE does not allow them.

I don't know where to look to investigate where these calls are being made to see if I might avoid them from being made. Any ideas?

https://github.com/mastodon/mastodon/discussions/25653

Jeremiah,
@Jeremiah@alpaca.gold avatar

Is anyone aware of a Mastodon dev ops / server admin community I can reach out to?

The GitHub Discussions for the project seem to be more unanswered questions than answers.

grrywlsn,

@Jeremiah the official helm chart needs some serious love

Jeremiah,
@Jeremiah@alpaca.gold avatar

Update: the issue is the Elasticsearch Bitnami chart dependency, not Mastodon itself. I found a managed service offering to use instead.

However, removing all of the Bitnami dependencies then unveils a transitive dependency that is not yet fully fixed: https://github.com/mastodon/chart/pull/60

I think my only option is to maintain a fixed fork temporarily. I have nearly bled to death from the papercuts at this point. 🤣

kfekete,

@Jeremiah AFAIK you cannot run sysctl and privileged containers if you are using Autopilot.

Jeremiah,
@Jeremiah@alpaca.gold avatar

@kfekete That part is confirmed by GCP. I’m trying to find out why/where Mastodon is making them because they don’t seem like they should be necessary. The logs don’t point me anywhere unfortunately.

kfekete,

@Jeremiah Sounds like a perfect use-case for eBPF! You can drop execsnoop into the cluster and you could catch what process is trying to execute that. It's quite simple to do with our (solo.io) open-source project called BumbleBee, however I don't think we pushed an execsnoop example already to our repository. How urgent is this? :)

Jeremiah,
@Jeremiah@alpaca.gold avatar

@kfekete I'm up for trying anything at this point! I would like to get this running by July 7.

alvin,
mhamzahkhan,
@mhamzahkhan@intahnet.co.uk avatar

@Jeremiah are you using the chart to deploy redis as well? Or have you done that separately?

mhamzahkhan,
@mhamzahkhan@intahnet.co.uk avatar

@Jeremiah or Elasticsearch?

IIRC the Mastodon helm chart uses the bitnami Redis/Elasticsearch charts. I think those use sysctl

Jeremiah,
@Jeremiah@alpaca.gold avatar

@mhamzahkhan Thanks for the reply! I am using GCP’s managed Redis and disabled it in the Helm chart. I am deploying Elasticsearch but it fails on the Mastodon containers. I also tried disabling the Elasticsearch container.

mhamzahkhan,
@mhamzahkhan@intahnet.co.uk avatar

@Jeremiah Have you tried running the helm template with your values file? Then you can see through the rendered yaml to see where the sysctl container is being created?

  • All
  • Subscribed
  • Moderated
  • Favorites
  • mastodon
  • ngwrru68w68
  • rosin
  • GTA5RPClips
  • osvaldo12
  • love
  • Youngstown
  • slotface
  • khanakhh
  • everett
  • kavyap
  • mdbf
  • DreamBathrooms
  • thenastyranch
  • magazineikmin
  • megavids
  • InstantRegret
  • normalnudes
  • tacticalgear
  • cubers
  • ethstaker
  • modclub
  • cisconetworking
  • Durango
  • anitta
  • Leos
  • tester
  • provamag3
  • JUstTest
  • All magazines