Note the importance of "permissiveness". Someone who flouts a rule is punished or shunned not because he or she harmed others, but because of contraposition: if the behavior (or avoidance) is allowed to be acceptable, others will follow the example and then people, especially in future generations, may be harmed.
The moral paradoxes in the parables in the New Testament deal with these issues. (My favorite is the Parable of the Talents.) Jesus is dealing with the idea that there is no way for life to be both "free" and "fair" at the same time; it's almost like quantum physics. It's true that in "real life" (as Mother used to call it), people often have responsibilities they don't "choose" under "freedom of contract". People often half to take care of others before or without having their own children. It's true that a lot of aspects of my own life and thinking are problematic, even though it's not so clear how these "problems" add up; they are rather like a final examination where you "answer any four" of maybe six questions. One thing is apparent: if you get a benefit you didn't earn, you incur some responsibility to provide for others, at least down the line. Disability or immutability has nothing to do with it. And if your content or skills benefits "humanity" (even on the level of solving "Enigma"), there needs to be the capacity of individuals in your neighborhood or orbit to matter, just because they are people, even if they need sacrifice. When people don't see this, they think no "rule of law" applies to them.