The point Epictetus is making is that what irritates you, is your own belief. That irritation may not irritate someone else meaning that, that is your own belief. It is something you think is irritating. An example is when people talk during lecture in class. That is something that can irritate me because I cannot hear the professor and I cannot focus. But maybe someone else doesn’t mind it can continue listening. This is my belief because I am irritated with that, while some people may not be irritated by it.
The stoic point he is making about the behavior of others and expectations is that you should look at the view of the other person. Similar to the saying from To Kill a Mockingbird, “You never really understand a person until you consider