Yes, I've always disliked this saying. People have agency. They can always refuse to play the game (and that is precisely what many great people chose to do).

Refusing to play the game just means you lose the game. It isn't a third state, it's a resignation.

You can't resign from a position you never held or a game you never entered into

Oh you're in the game. Your choice is whether to win the game or lose the game. Also if you lose the game you die. It's like Jumanji. Pretending the tiger isn't real is not a viable option.

Life is not 1 game with 2 ends. And everyone dies.

Capital is far more powerful than social ostracization. Consider the middle east situation: most of the money is on the opposite side from most of the people. You get kicked out of most groups for advertising that you support the money side. But the money side still won because it paid off everyone important.