If I were you, I'd carry with one in the chamber. When it counts, that couple seconds to rack one into the chamber could be too long. Between the trigger safety, the grip safety, and the sear blocker...you're completely secure carrying condition 0.
If you're worried, do what I did when I first got my XDs - I carried it with a dummy round in the chamber, and a live-fire-loaded magazine. After a day of of normal carry, as well as some more "extreme" carry situations like moving building materials and lots of lift/stretch/twist/crawl activity, I would come home, unholster, point it in a safe direction, and pull the trigger.
Click.
Proof it had not "fired" on its own during my day's activities.
Not once had it self-fired on my hip.
The dummy round now sits in my safe, and I carry daily in condition 0 without a second thought. If it comes down to it, all I have to do is draw, aim, and pull the trigger - that's the nice thing about passive safeties!