An invisible man only has as much power as he is, and how he thinks of himself. I say this because throughout this novel, the eponymous narrator struggles with his invisibility, seeking power in it. When he is not invisible, however, he becomes part of an oppressive collective of people, losing his power.
The college and the Brotherhood were the two main places in which the narrator strives to be visible, to be recognized as an individual, recognized as a person. However, in both cases, the exact opposite occurs, leaving the narrator used and abuse. The group attempted to assimilate him, but to no avail. As much as the narrator wished to be a part of the group, he never could allow his identity to be crushed.
But when the narrator is left to his, own devices, he had the power to move crowds, to stop fights. Most easily demonstrated in his curbside speech, where he essentially controls the crowd, captivating them when he wishes, yet allowing them to do what they want when he wants them to. This leads to the run in with the brotherhood, etc.
The invisible man's power is there, it always was. What stopped him was himself, and to an, extent, the corporations and groups controlling him. No, he needed to be alone to vinyl his fate.
