This is a good question and I’ll be interested to see what others have to say. One thing I’ve noticed is that memories can transform into ‘memories of memories’. What I remember is not the felt impression of something but what I subsequently told or showed myself about it. And it doesn’t even just apply to childhood - I have same thing for big events in my life from last couple of years.

I do find that it’s possible to consciously dig deeper and unearth forgotten memories though.

I’ll also add that I think what constitutes a sense of ‘self’ is more than just memory.