I'm with you. I'm the same, and my wife is not at all concerned. Of course we do work in the same office, but still.
I wasn't just referring to you and your wife. Aside from that, personally, I'd just let it go. It's a solo setup, I'd leave it as is, as opposed to going to any lengths for the girls at the office. As a manager with several employees, it's been my experience that such an overture is unnecessarily risky ... professionally speaking. It's something that is almost impossible to see from your vantage point, but it's there. Could be everyone has a lot if fun and that's it. Could be not.