He never held him at all. He never grabbed his jersey or any other part of his equipment and held him.
Plus, watch it again. He starts off by pushing on the player's back, keeping him on the ground. As the play is still going on, the player starts to get up, so James Harrison pushed his ass...