Should I use one docker-compose.yml for all my services?

czardestructo, 10 months ago

I have a folder that all my docker services are in. Inside the folder is a folder for each discrete service and within that folder is a unique compose file necessary to run the service. Also in the folder is all the storage folders for that service so it’s completely portable, move the folder to any server and run it and you’re golden. I shut down all the services with a script then I can just tar the whole docker folder and every service and its data is backed up and portable.

neon_commie, 10 months ago

Could you share your script?

czardestructo, 10 months ago

Scroll down a bit, I added it later.

neon_commie, 10 months ago

Thanks!

Spazztastik, 10 months ago

This is exactly what I do and could not be happier!

MaggiWuerze, 10 months ago

Exactly my setup and for exactly the reasons you mentioned

MalReynolds, 10 months ago

Does portainer just work?

MalReynolds, 10 months ago

To answer my own question, yes, yes it does. Should’ve done this ages ago…

YonatanAvhar, 10 months ago

Exactly what I do except my master folder is ~

czardestructo, 10 months ago

I do ~/docker so I also have a docker-prototype folder for my sandbox/messing around with non-production stuff and I have a third folder for retired docker services so I keep the recipe and data in case I go back.

dpflug, 10 months ago

@czardestructo I like the tidiness of this.

czardestructo, 10 months ago

In case anyone cares here is my script, I use this for backups or shutting down the server.

<pre style="background-color:#ffffff;">
<span style="color:#323232;">#!/bin/bash
</span><span style="color:#323232;">
</span><span style="color:#323232;">logger "Stopping Docker compose services"
</span><span style="color:#323232;">
</span><span style="color:#323232;">services=(/home/user/docker/*)    # This creates an array of the full paths to all subdirs
</span><span style="color:#323232;">#the last entry in this array is always blank line, hence the minus 1 in the for loop count below
</span><span style="color:#323232;">
</span><span style="color:#323232;">for ((i=0; i<=(${#services[@]}-1); i++))
</span><span style="color:#323232;">do
</span><span style="color:#323232;">    docker compose -f ${services[i]}/docker-compose.yml down &
</span><span style="color:#323232;">done
</span><span style="color:#323232;">
</span><span style="color:#323232;">#wait for all the background commands to finish
</span><span style="color:#323232;">wait 
</span>

dustojnikhummer, 10 months ago

I personally don’t. It is just messier. I only group things that belong together, like a webserver+database, torrentclient+vpn and so on.

Moonrise2473, 10 months ago

No, keep them ungrouped, migration to a new server is much easier, otherwise you need to migrate everything everywhere all at once

You can have the same effect (connect to the named container) if you create a docker network and place everything on the same network

himazawa, 10 months ago

I was thinking about that just today, I have something like 30+ services running on a single compose file and maintenance is slowly becoming hard. Probably moving to multiple compose file.

Auli, 10 months ago (edited 10 months ago)

I’ll be the opposite of everyone I guess I have all my services in one compose file. Never had an issue with it. Why I have no exposed ports and everything is accessed through a reverse proxy, and the big one it’s easy to just go docker compose and have them all come up or down.

AlexKalopsia, 10 months ago

Same for me, it all mostly started from the desire to have a single MariaDB and Postgresql container holding all the databases. Not sure if I could achieve the same result with different compose files, perhaps I can, bit never had the need.

I actually find my setup super comfortable to use

tiwenty, 10 months ago

I have multiple files but a single stack. I use an alias to call compose like this:

docker compose -f file1.yaml -f file2.yaml

Etc.

d4nm3d, 10 months ago

i go so far the other way with this personally… I actually have a seperate LXC for each docker container and a lot of the time i use docker run instead of docker-compose…

I’ve still not had anyone explain to me why compose is better than a single command line…

Awwab, 10 months ago

I always thought the compose file is great for maintenance. You can always save the docker run commands elsewhere so at the end of the day it's more of an orchestration choice.

midas, 10 months ago

I would not. Create an external network and just add those to the compose files.

zbecker, 10 months ago

@midas @bronzing

Out of curiosity why not? this is what I have been doing forever.

midas, 10 months ago

So there’s a million ways to do things and what works for you works for you. For me, putting all services ina single compose file only has downsides.

• Difficult to search, I guess searching for a name and then just editing the file works - but doesn’t it become a mess fairly quickly? I sometimes struggle with just a regular yaml file lmao
• ^also missing an overview of what you’re exactly running - docker ps -a or ctop or whatever works - but with an ls -la I clearly see whats on my system
• How do you update containers? I use a ‘docker compose pull’ to update the images which are tagged with latest.
• I use volume mounts for config and data. A config dir inside the container is mounted like ‘./config:/app/data/config’ - works pretty neatly if each compose file is in its own named directory
• Turning services on/off is just going into the directory and saying docker compose up -d / or down - in a huge compose file don’t you have to comment the service out or something?

theterrasque, 10 months ago

Bingo. Or just bite the bullet and dive into Kubernetes

Auli, 10 months ago

Overkill for home use

vera, 10 months ago

A compose file is meant for different components of a single service but you’re allowed to experiment with whatever you want

juicebox, 10 months ago

You can use an external network if you wish to refer to them all by a name. Just make sure all the containers you wish to refer to are in it.

degrix, 10 months ago

I’ve thought about going that route, but ultimately decided to adopt something like portainer.io. My thought process behind it was that some projects within each category may have overlapping dependencies and so I’d end up with multiple entries for a particular dependency in the same file which I didn’t like.

I don’t expose services to the internet from my home lab, so I generally just add host entries manually to each of my computers so that I don’t have to type in ip and port.

MaggiWuerze, 10 months ago

For simplicity sake alone I would say No. As long as services don’t share infrastructure (eg. a database) you shouldn’t mix them so you have an easier time updating your scripts.

Another point is handling stacks. When you create dockers via compose you are not supposed to touch them individually. Collecting them all, or even just in categories, muddies that concept, since you have unrelated services grouped in a single stack and would need to update/up/down/… them all even if you just needed that for a single one.

Lastly networks. Usually you’d add networks to your stacks to isolate their respective Backend into closed networks, with only the exposing container (eg. a web frontend) being in the publicly available network to increase security and avoid side effects.

wplurker, 10 months ago

So right now I have a single compose file with a file structure like this:

<pre style="background-color:#ffffff;">
<span style="color:#323232;">docker/
</span><span style="color:#323232;">├─ compose/
</span><span style="color:#323232;">│  ├─ docker-compose.yml
</span><span style="color:#323232;">├─ config/
</span><span style="color:#323232;">│  ├─ service1/
</span><span style="color:#323232;">│  ├─ service2/
</span>

Would you in that case use a structure like the following?

<pre style="background-color:#ffffff;">
<span style="color:#323232;">docker/
</span><span style="color:#323232;">├─ service1/
</span><span style="color:#323232;">│  ├─ config/
</span><span style="color:#323232;">│  ├─ docker-compose.yml
</span><span style="color:#323232;">├─ service2/
</span><span style="color:#323232;">│  ├─ config/
</span><span style="color:#323232;">│  ├─ docker-compose.yml
</span><span style="color:#323232;">
</span>

Or a different folder structure?

MaggiWuerze, 10 months ago

The second one is exactly what I have. One folder for each service containing it’s compose file and all persistent data belonging to that stack(unless it’s something like your media files)

einsteinx2, 10 months ago

The second is exactly how I do it. Keeps everything separate so easy to move individual services to another host if needed. Easy to restart a single service without taking them all down. Keeps everything neat and organized (IMO).

troy, 10 months ago

Probably want to keep services with different life cycles in separate docker compose files to allow you to shutdown/restart/reconfigure then separately. If containers depend on each other, then combining into compose file makes sense.

That said, experimenting is part of the fun, nothing wrong with testing it out and seeing if you like it.