The first step is truly open dialogue.
Now that he opened up, the most important is to develop complicity along that line.
You are here to help and support each other and create synergy (bring your power together to take this to the next step).
The best way to create dialogue is to create a "forum space" for the two of you.
It can sound a bit artificial but believe me, it works.
Find out how deep this is.
Fantasies are alright as long as you don't act on it.
It's valid for any type of fantasies.
In such case, simply sharing might already clear up his desire and make him realize it was just a thought passing by.
On the other hand, if he truly can't stand your touch it might be the sign that something deeper is going on for him.
Find out together what is truly going on and see how it evolves over a period of a month.
This is happening to him, not to you.
Don't take responsibility for his actions feelings or behaviors, no matter what.
You have your own life, individuality and your own foundation.
Make sure you don't let this impact on your work, career, relationship with children, social life, etc.
Regenerate! Stay healthy!
Keep the challenge to its real size and location: it's something happening to him and has to do with emotions and feelings in him that you can't control.
So, invest exactly in proportion with how much influence you have.
At the end, it's truly up to him to decide, act on it or do something about it if it's a problem for him.
Sign up for a sessions with me.
Google "gay forum" and check what others say about these issues.
You can as well check with a sex therapist to get fresh perspectives on this topic.
It's important you get extra support, ideas, feed back, experiences and knowledge on this topic.
This will definitely empower the two of you in dealing with this situation.