I checked manual for racoon and didn't understand how to do this. 

Racoon can support the following authentication_method :
pre_shared_key 
rsasig 
gssapi_krb 

From the manual, seems rassig is for RSA signature, but not RSA-encrypted nonces.

Anyone can help me with this question?
That will be greate help if provide a example.

THanks