When you try to start a VM in XenServer that talks to a StorageLink Gateway server, you get:
1/19/2012 x:xx:xx PM Error: Starting VM 'xxxxxx' - Storage assignment failed
(SSL_ERROR_SSL error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate
verify failed)
in the XenCenter log. You can verify the SSL cert by opening up the following in a browser window and replacing the x’s below with your StorageLink server’s IP address:
https://xxx.xxx.xxx.xxx:21605
You will get a cert error message in your browser. Notice the Citrix CVSM SSL certificate issued on 1/19/2009 has expired today 1/19/2012 at 20:25:53 PM (GMT) which is 2:25 PM Central Standard Time. So basically any VM you try to turn on, reboot, or migrate after the cert expired will not work and return the SSL error above in the XenCenter log. Yeah, big problem.
I was the first to call in about this issue soon after the cert expired apparently. As I was on the phone troubleshooting this with the support engineer, others began calling in with the same problem. We have escalated it to the highest level at Citrix support and have been assured a workaround and a new cert are both being worked on and something should be available tomorrow morning. This is going to impact pretty much all StorageLink customers globally so trust me, they are working on it. Over the past several hours, I have tried numerous workarounds myself but been unable to get a full fix yet. I’ve tried self signed certs using OpenSSL, IIS & SelfSSL, etc. but to no avail so far. The StorageLink Gateway does not use a web server such as lighttpd, Apache, Tomcat, etc. either so I can’t force it to use another set of certs on that end. Apparently it uses API calls. When you restart the services, you will notice it copies the following SSL certs which are the culprits (into memory I’m guessing). I used Process Monitor to verify:
D:\Program Files (x86)\Citrix\StorageLink\Server\cacert.pem
D:\Program Files (x86)\Citrix\StorageLink\Server\server.pem
I actually did manage to get a little further than I thought on the handful of workarounds I tried, but nothing completely successful yet to regain functionality while we wait for a hotfix. If you want to try playing with the certs yourself, just remember to restart the StorageLink services after you swap out the certs each time so it pulls them in. XenCenter should see the SSL cert change and prompt you almost immediately with a warning message.
I will keep this post updated with the latest developments. Please post if you are having the same issue or come up with a temporary fix. In the meantime, call Citrix and open a case so you are in the loop when the fix is released.
UPDATE January, 24th, 2012 – Citrix has published the fix:
http://support.citrix.com/article/CTX131994
Apply the certs using the instructions in the KB. Shouldn’t take long at all.
Chris Markovic
January 19, 2012 at 10:18 PM
Allow me to send my thanks for this information. We’ve just had several customers complain about this very issue. I am sure many people are keeping an eye out for this patch.
Simon
January 20, 2012 at 2:44 AM
Thanks for giving us this information!
We found a temporary fix for this. Send me an e-mail to get the details and maybe you can post the solution for all the others.
Marco Stalder
January 20, 2012 at 3:44 AM
Can you please share the temporary fix for this.
Simon
January 20, 2012 at 4:19 AM
You have to replace the certificates like Jason suggested.
The password for the private key is hardcoded and I am reluctant to post it here.
Send me an e-mail to storagelink-problem_at_dakosy.de and I will give you the password or new certificates.
Koen Vanneste
January 20, 2012 at 5:10 AM
@Simon
Hi Simon,
We have the same problem with storagelink gateway. Can you please send us the new certificates ?
Thank you very much,
Koen
Koen Vanneste
January 20, 2012 at 8:04 AM
@Koen Vanneste
Hi Simon,
Your fix works. Thank you so much. 1000 Kudos ! At least have an extra month to wait for an official fix from Citrix !
I don’t understand why it takes so long for Citrix to find a solution for this issue. They just need to renew the certificates and put them on the support site.
Best regards,
Koen
Nick
January 20, 2012 at 8:49 AM
There will be a KB published soon, in the mean time please go here for the fix.
http://forums.citrix.com/thread.jspa?threadID=300586&start=15&tstart=0
Joel
January 20, 2012 at 8:58 AM
The fix has been released. Support just walked me through replacing the certs.
Koen Vanneste
January 20, 2012 at 9:05 AM
@Koen Vanneste
Hi,
The fix has been released by Citrix. This will be published as CTX 131994.
regards,
Koen
Jason Samuel
January 20, 2012 at 9:08 AM
Thanks guys, we are testing the fix from our Citrix rep this morning as well.
Jason Samuel
January 20, 2012 at 9:56 AM
All our environments are functional again after applying the new certs to the hosts and SLG servers. I’ll update the post with the official KB link once it’s up.
Jason Samuel
January 24, 2012 at 9:47 AM
http://support.citrix.com/article/CTX131994