If they trade memories than what is the point of the swap? It is like it didn't happen.
I've also been thinking of that long long before. I think it's better for one to keep their memories so they could savor the sensation of a different body and life and it's up to time for them to get settled to the new.
Some people like the idea of that, it counts under another fetish category, Hypnosis or Mind Control, which, usually can or will involve stuff like completely becoming the new person, personality, memories, usually by force. Its like mental bondage basically.