The piece that looks like a set screw is actually a roll pin-I didn't realize that when I removed my front sight and promptly sheared it off-my sight pusher is fairly strong obviously.:rolleyes:
I think you can tap it thru to the inside of the slide.