I’d suggest reading up on John Locke’s “continuity of consciousness” theory of self. It’s essentially what you’re describing here: he holds that the “self” over time is a construct of the continuity of one’s consciousness through memory. This has some interesting implications: for instance, if someone becomes blackout drunk and has no recollection of the experience, can it be said that that experience belongs to the same person after the fact? Also, as another commenter pointed out this has significant implications in the case of memory loss, especially for someone suffering from dementia.