I think this probably speaks to the idea of self determination. Does Israel have the right to self determination, a state of Israel. Does Palestine have the right to self determination. Or do both have that right, which is the two state solution.
I can see why wearing a Palestinian shirt with a map of Israel may be taken as a commentary that Israel does not have a right to self determination. I will say I'm no expert and I'm just trying to make sense of it, but I think that's where the criticism probably comes from. Is that anti-semetic? I wonder if Irvine was even aware.
Fwiw I think denying either Israelies or Palestinians self determination is pretty fucked up.