We should remove the state from our map only if the last button is going up.

Fixes #260

R=chinmaygarde@google.com

Review URL: https://codereview.chromium.org/1215743003.