I've put babies straight into 4's without problems, but have also had some that have slowed down at first a bit because the viv's been pretty expansive for them to find food in. They were fine once I changed a couple of things in there for them.
There's no such thing as a viv that's "too big", but you can have one that's not set up well for babies. Keeping it simple, with minimal places for crickets to hide behind is crucial, as is thinking about how a baby's instincts work and setting up the viv to work to them. Partitioning it may make feeding easier for you, but there's no reason why you can't put one straight into a larger cage and use "work-arounds" to overcome problems...