There are two possible reasons you might have received a penalty.
- You've used a cheat. In this case, you won't be able to start the game for some time, until the ban is lifted.
- You left the match before it ended. In this case, you won't be able to join custom games. To lift the ban, you simply need to win one quick game. After this, you'll once again be able to use all the game features.