Thank you! It’s a tough line to delineate, that’s true.
I believe that safe place to start with is John Stuart Mill’s ‘Harm Principle,’ which is often defined as “your right to swing your fist ends where my nose begins.” It’s for a longer but interesting discussion what that exactly entails. I have for example enjoyed Stanford Encyclopedia on the issue: https://plato.stanford.edu/entries/mill-moral-political/#HarPri (although, some of it gets hopelessly philosophical).
But, yes, it’s for a longer discussion. Definitely not an easy question to answer, I don’t want to give the impression that I am making light of the issue. And I haven’t personally properly dealt with all the nuances here either — because there are a lot of them. Perhaps for a blog post in the future :)