When you upgraded a NetScaler from firmware version 10.5 to 11.0, there were many things you needed to be aware of which I covered in my “Citrix NetScaler 10.5 to 11.0 firmware upgrade issues to watch out for” article. I really love 11.1 and all the improvements over the older firmware but one of these issues is still very prominent even when upgrading 11.0 NetScaler Gateway environments to 11.1. You may still have around 10-20% of your users get a blank white page after the upgrade. Yes even if you are already on X1 on 11.0 and keep the same X1 theme on 11.1 these users can be impacted. You’ll notice that base.css and rdx.js don’t want to respond which gives you a blank page:
If you try and type the the URLs for those files directly into a browser:
https://gw.yourdomain.com/vpn/js/rdx.js
or
https://gw.yourdomain.com/logon/themes/Default/css/base.css
you’ll notice it will show a Secure Connection Failed Message like this:
An error occurred during a connection to gw.yourdomain.com. SSL received a record with an incorrect Message Authentication Code. Error code: SSL_ERROR_BAD_MAC_READ
you’ll hit refresh a few times and eventually the page will come up fine:
After a few refreshes of the login page it might work but for some users, but it can also get pretty bad where the whole page is messed up with certain elements loading and much of the .js files giving HTTP 304 codes saying they are “Not Modified”:
As I mentioned in my article before, this is being cached on the client browsers and there are some steps you can take to mitigate this. Some people think if they have a Platinum NetScaler with Integrated Caching you can go in there and flush all objects or even flush the “loginstaticobjects” Cache Content Group but this will have no effect. It’s not the NetScaler caching these objects. The easiest way to fix this is to create a no cache rule and bind it to your NetScaler Gateway vserver for a few days before and after your upgrade. The impacted 10-20% of users will hit it and this will force their browsers to flush their caches.
A default NetScaler Gateway vserver will have 6 Cache policies bound to it like this:
What you need to do is add an extra cache policy with the priority order of 1 so it gets hit first. You go under Optimization > Integrated Caching > Policies and create something called “cache_pol_fix_blank_page”. Set the Expression to true and the action to “NOCACHE”:
Now you bind that as priorty 1 on your NetScaler Gateway vserver like this:
and now the impacted users should be back to normal again with a regular login page! Just don’t forget to unbind this after a few days. Please do post a comment below if this helps you or if you have any questions.
ziggricky
January 26, 2017 at 12:51 PM
Thanks Jason. I have been dealing with cache nightmares on netscaler everytime we update a rewrite policy that modifies html on the page. I will give your suggestion a try and report back.
Jake
April 22, 2017 at 8:08 AM
I’m planning on going from 11.0 to 11.1 soon, but I only have Enterprise licensing. Is there another way to solve this problem or should I add on Integrated Caching licensing prior to upgrading. Great article!
Nathan
November 28, 2017 at 6:08 PM
Unfortunately even with this configuration a lot of the content delivered by the VPN vServer is still delivered with a Cache-Control header of no-cache, which doesn’t actually stop caching, it just makes the browser validate the ETag which can still result in this issue occurring in our experiences on various device upgrades in our Enterprise. You can validate it’s pulling from the cache by looking at the HTTP 304 responses you get on those calls.
We found a more effective method, albeit far more bandwidth intensive, by using rewrite policies/actions that look something like this:
add rewrite action CacheControl-NoCache_replace replace "HTTP.RES.HEADER(\"Cache-Control\")" "\"no-cache, no-store, must-revalidate\""add rewrite policy RWP-SDC-PRS-CacheControl-NoStore "HTTP.RES.HEADER(\"Cache-Control\").EQ(\"no-cache\") CacheControl-NoStore_replace
This replaces the no-cache header and adds the no-store and must-revalidate header which will force it always to re-pull on all subsequent connections.
You can bind this to a VPN vServer, or if you want to work on a AAA forms page you can add another bit to the policy like:
&&(HTTP.REQ.HOSTNAME.EQ("asdf.com"))and then bind it globally.We avoided any reports of blank or white screens on an environment supporting 10K+ AAA/VPN users, compared to a year ago using other methods where we had 20%+ reports.
zwhiterabbit
December 28, 2017 at 9:50 AM
another solution is to unbind the _cachevpnstaticoptions from the default 6 policies , it should work as a charm
C. Uerel
October 9, 2018 at 9:42 AM
@zwhiterabbit
I tried to unbind _cachevpnstaticoptions from the default 6 policies (FW 12.0 59.8) I get “Policy not bound to specified policy label”
Ravi
March 18, 2019 at 12:23 PM
Hi,
we are moving MPX to VPX. Added VPX in HA pair and synced the config. all looks good bu when i promote VPX to Primary, the gateway URL is not openining in browsers, but if we open in incognito or Private it works.
Seems there is a cache policy, but we have this policy in place and it got synced fine as well. any thoughts. ?