Well I think part of the answer comes from having a society that is more interconnected than we currently have.
If there were people that were both part of the gardening group and part of the builder’s group then those people would have the necessary common knowledge to be able to satisfy the needs of both groups.
That is part of why I think a society of anarchists necessarily needs people to be educated in ways that make them a lot more generalist than we are now (hence the emphasis most anarchists have with the idea of self-sufficiency).
Edit: Also in the cases where there isn’t significant overlap between the two groups having a third group that does have knowledge of both of them participate in the decision making would also serve the same function.