I think 6 players max per team is fine, although I would be fine with seeing more. I would rather be able to form a party with a bunch of my friends rather and fight monsters that are redonkulously strong then be limited to say, 4 players like in PSO, and not be able to invite little Jimmy so he can possibly get that Dragon Slayer sword he's always wanted from the level we're playing.

That said, it just really boils down to how difficult the enemies are in comparison to the players for each level range, whether or not that difficulty scales up if there are more people in the party, how well the game can handle multiple character models/animations at once without causing lag/slowdown/drop in frame rate, etc. How well Sega/Sonic Team can handle all that, well... I don't have much faith right now, but I agree that making use of the Hedgehog Engine would open up some possibilities.