How can I remove 3 players from a specific game but not from the central roster?

You would simply show them as not playing that game (Played checkbox) in the statistics input area (under Games input).