Oh, yes, I definitely got the impression that the only 'rebels' were Jack, the Doctor and the three Joneses.

Though the soldiers on the Valiant seem extremely quick to change loyalty as soon as the Doctor's re-empowered - Jack runs straight off to gun down the TARDIS and orders the men who were guarding him moments before to go with him and they do! If I was Jack, I'd be half-expecting them to try and stop me!