We are our choices
Who we are depends not so much on our environment, or even our selves, but on the choices we make. The difference between one person’s self and another person’s self is not that they have different bodies, lives, or histories. The difference is the choices that they would or would not make when in the same environment.
As a thought experiment, consider the life of John the plumber. John grew up in an average family. John decided to learn a trade rather than go to college, and has been successful so far. Second, consider Jill the research chemist. Jill grew up in an average family. Jill decided to go to college rather than learn a trade, and has been successful so far. The difference between John and Jill is only superficially that one is a plumber and one is a chemist. The true difference is the choice of whether to go to college or to learn a trade. (Of course, this thought experiment is so simplistic that it might even be used as a straw man against choices defining selves, but an entire history for two people would be overly verbose.)
Of course, asserting the point that the defining characteristic of a self is the choices it would or would not make begs the question of why we should (or even can) disregard personal history and environment. The reason we can do so is that the universe is governed by physical laws. All matter and energy, and interactions between the two, follow rules. In Newtonian physics, if you knew the initial state of the universe, you could compute all future states, merely by application of physical laws. Quantum physics asserts that there is non-determinism on a micro-level, and so more than just the physical laws are required, but on a macro-level physical laws again are the rule.
If Jill’s “self” were to have John’s body, environment, and history from birth, what would be the difference between Jill and John? Only the choices that Jill would or would not make that differ from John’s choices. So why, then, do we consider Jill in John’s place, when such a thing is impossible? The reason is that your environment, from your body to your family to your country to your planet, is initially entirely outside of your control. Environmental factors act upon you, and you define yourself by your reaction to them. Your “environment” is an extensive term; it includes not just the ecosphere but also the people around you, your day-to-day life, and even your own body.
As another though experiment, consider replacing John’s self with Jill’s self, and vice-versa. John’s body will still have John’s memories, and Jill’s self will be acting entirely in the context of John. How then can we differentiate between John in John’s body and Jill in John’s body? Only by the choices that whoever happens to be inhabiting the body of John makes.