I never worried too much about them. I got stung in the face several years ago by about 5 bald faced hornets. It looked like I'd went a round or two with Iron Mike but other than that I was just fine. We'd had these red wasps ( dude, there were hundreds of them) hanging out on the corner of the house by the spigot for a few years and never had any real issues with them. Live and let live and all that. Then one day I was weed eating and I felt a spot on the back of my head get really hot. I figured out pretty quickly why when I saw a couple red wasps flying agitatedly towards me. So I got mad, went in the house looking for wasp spray. The wife informed me I had given my last two cans to her brother across the street, who wasn't home. So I went to the other side of the house to weed eat. My wife heard me fire up the weed eater and then a few minutes later heard it die and when it didn't fire back up again she came outside and found me on the ground in anaphylactic shock next to my trailer.
Anyway, ever since then I carry an Epipen with me and it touched off a war between me and these bastard red wasps. I'd go out early in the morning and later in the evening and soak the crap out of the gap in the aluminum fascia where they were getting into my soffit. I'd kill them off by the hundreds during the day when they would gather on the fascia. After a year or so I decided it was time to rip all that old crap off the house and put all new up. While I was up there tearing everything off I ripped out the old wooden soffit, looking for the queen. I never found her but I suspected that I had removed her and her lair was somewhere in the mess all over my yard now. Fast forward to a month or so ago and there on the corner, crawling under the shingles were a few red wasps. I sprayed up in there for a few days and figured maybe it was just a few of them. Then I get a call from my wife, who had just jabbed herself with an Epipen ( she has been highly allergic since she was a kid) telling me the war was back on.
So I got a bunch of cans of Raid from work and went to war. As of the last few days I haven't seen any, but they aren't going to lull me into a false sense of security again.