Clara said she messed with it so while I want to also go conspiracy theory on this and say “is it still human compatible and I mean they were alluding to the Doctor being part human maybe??”, I’m going to say that he probably did draw the short end of the stick and got the wipe.
But only a sorta-wipe? Like a small moist towelette that’s not nearly big enough to clean your hands when you ordered those BBQ ribs. In this analogy his memories are the BBQ sauce.
He still KNOWS, you know? That is some stubborn BBQ sauce and that wipe is bad. It probably just smeared everything around before ripping.