There is a maximum number of modern adults that you can regularly get to the same place at the same time. This is not an inconsequential fact! Lots of projects, hobbies, cultural practices, etc. want a certain number of people in the same place at the same time, and it’s a fact that you can do that more easily if the target number is “two” as opposed to “seven.”
The larger point: Don’t build too much of your life around the assumption that X other people don’t have anything else to do but YOUR thing. You might imagine the dream team at your job for a particular project, but some of them have conflicts, and the higher the value for X, the more likely that is.
Keep the groups lean when you can!