As I stated in the following sentence, in most cases like for the Arachnos Soldier for example cannot have “perma-pets” even if they slot for maximum recharge reduction. They still have some downtime with said pets even while haste is running.
Coming from experience, these pets are not very strong to begin with so I would not say that this would create any unbalancing issues. Making them permanent would simply make these classes more fun to play. Correct me if I am wrong but pet based heroes are not exactly top tier builds.