Had an interesting morning. Our network team discovered my workstation was making around 6000 requests an hour on RDP port 3389 through our internal firewall to an unroutable network reserved in our server subnet. Basically a SYN flood (DOS/Denial of Service attack) was being executed internally from my workstation. I know my machine is clean and the only thing I had open was XenCenter. It had been running for about a week now, that was the last time I rebooted my workstation. So time to put on the detective hat and figure this one out.
While using TCPView to do a live netstat, I discovered that XenCenter by default will always establish a connection via RDP when you click the Console tab. It tests to see if RDP is available on the VM and then ungreys that “Switch to Remote Desktop” option. Even if you are using a console session, XenCenter wants to see if RDP is an available option to you. You know that little flash of the console you usually see after hitting the Console tab? That’s the XenCenter console checking for RDP and “connecting” to the VM transparently to verify RDP is available. Not so transparent and actually annoying but I never thought too much about it.
The problem is in a PVS environment, you usually have your network split between a streaming traffic NIC and a regular traffic NIC. The streaming traffic NIC is supposed to be the first NIC/device. So Device 0. It will look something like this on all your VMs:
Where the first streaming NIC is only routable within the Blade enclosure or server subnet and the secondary NIC routable and used for regular network traffic.
Well the problem is that when you click the Console tab on one of these VMs, XenCenter will send a SYN request to what IP is at Device 0. So in our case, an unroutable IP in the server subnet.
Not a problem right? Well it never stops. It continuously sends the SYN requests attempting to connect. Even if you click off the Console tab or go to another VM, it continues to try RDP on that IP. Our firewall separating workstation and server subnets was getting hammered. You can verify because your Console will have the “Switch to Remote Desktop” option greyed out during this whole process.
and TCPView will show all those little red SYN_Sent attempts. After a few days of leaving XenCenter up and clicking from console to console, the amount of traffic hitting your firewall will be tremendous. It will look like a SYN flood attack. If you have an IPS or IDPS (Intrusion Prevention or Intrusion Detection & Prevention System), it might even shut down your port.
I called Citrix and submitted a ticket with their development team. I got a call back later and there is a work around. In XenCenter, go to Tools > Options > and click the Console option. Then uncheck “Enable Remote Desktop console scanning”:
When you uncheck this, it will also uncheck “Automatically switch to the Remote Desktop console when it becomes available”. This is fine:
After this you will notice the SYN flood will immediately stop and all your VMs will now have the “Switch to Remote Desktop” option ungreyed from the get go. If you click it, then it will attempt the RDP connection and you will see the SYN_Sent again:
In my opinion, Citrix should fix this but stopping the SYN requests after you click away from the Console tab. This is not an issue that will impact many people but if you are running a PVS environment and you have it setup using Citrix best practices with 2 NICs and the streaming NIC is not accessible from your workstation subnet, you will eventually run into this issue. The longer you keep XenCenter open, the worse it will get.
If I get any updates from Citrix on a fix, I will post here. For now the work around will work fine. You just won’t have the Remote Desktop automatic console scanning available for your regular environments anymore. Not really a big loss for me but it might be for you depending on your environment.
Jason Samuel is a visionary product leader and trusted advisor with a proven track record of shaping strategy and driving technology innovation. With extensive expertise in enterprise end-user computing, security, cloud, automation, and virtualization technologies, Jason has become a globally recognized authority in the IT industry. His career spans consulting for hundreds of Fortune 500 enterprises across diverse business sectors worldwide, delivering cutting-edge digital solutions from Citrix, Microsoft, VMware, Amazon, Google, and NVIDIA that seamlessly balance security with exceptional user experiences.
Jason’s leadership is amplified by his dedication to knowledge-sharing as an author, speaker, podcaster, and mentor within the global IT and technology community. Recognized with numerous prestigious awards, Jason’s contributions underscore his commitment to advancing technology and empowering organizations to achieve transformative results. Follow him on LinkedIn.
Brian
October 25, 2014 at 3:10 AM
I just experienced this issue with a local network and found xencenter to be the cause. Thanks for documenting this and for posting the fix.