This makes a lot of code safer.  We only try modifying the object in one
place, after making sure it's safe to do so.  So, do a const_cast<> in
that one place...