I recently was hired to create a web application firewall (WAF) using Citrix NetScaler to protect a SAP Hybris based e-shop. This shop has content for several languages, so we had to select the right home page.

The base URL of the website was like that: https://shop.domain.com/shop/language/. SSL was optional. I wanted to set the default language based on browser settings. I based it on HTTP- Header  Accept-Language.

There are several pre-defined codes. I found a list here.

Following rfc3066 I found out, Accept-Language tags will always start with the main language (for example en, de, fr, …) and can optionally be followed by sub attributes. De-AT for example means: Austrian German. There may be some more sub attributes, but they had not been of any interest. A typical header would look like that: en-us, en;q=1.0,fr-ca, fr;q=0.5,pt-br, pt;q=0.5,es;q=0.5

The only portion I was interested in are the left 2 letters. en means English, de means German, fr means French and so on. Easy like that!

Even though users could change their language code they usually don’t. There are tools out there, like “change accept-language switcher” for FireFox, but they get hardly used, and if they get used they get used for reason. So this is something I can count on.

So I decided to base my policy on this header. I made up my mind to use responder, however I could also have used rewriting. Whenever a user just opens the base URL (/), I will forward him to /shop/xy where xy is the country code.

The Citrix NetScaler function I have to use is HTTP.REQ.HEADER("Accept-Language"). I am interested in a substing, starting on position 0, the first 2 digits. So I have to add the SUBSTR (x,y) function, where x is the starting position (0) and y is the length of the substring (2).

The RFC mandates lower case letters for the country code, but the term RFC means request for comment. So different from ANSI or DIN a RFC is just a proposal, so you might not count on this. So I appended a .TO_LOWER, just to make sure. So my redirect would look like that: "/shop/" + HTTP.REQ.HEADER("Accept-Language").SUBSTR(0,2).TO_LOWER + "/".

It worked great. Our UK costumers got sent to English content, the French to French, the Swiz to French, Italian or German (depending on their taste) and the Germans to German content, even costumers in Lichtenstein would be faced with German content (de-LI). It could hardly have been any better. Just Chinese hackers saw a 404, not found, as they got directed to ../zh/, which does not exist at all! Great!

My first thought was: give a damn shit on Chinese, we don’t sell outside of the EU (OK, something our British costumers will have to keep in mind). But there is a bigger problem than just Chinese: Some search engines might use crawlers not sending this header at all. So what can I do?

We currently have some languages: English (en), French (fr), German (de), Italian (it), Dutch (nl) and some more, a total of 13 languages.

So I created a list. A pattern set called suported_languages. It would be good style to give index numbers based on importance. So I gave de (German) 10, as it’s my costumer’s biggest market, French 20, and so on. Reason here is to save computing time.

add policy patset suported_languages
bind policy patset suported_languages de -index 10
bind policy patset suported_languages fr -index 20
...

My costumer defined a standard language, English, for example. I then created 2 policies, one for unsupported languages (not in list), and one for supported ones.

So my first policy expression would look like that:

HTTP.REQ.URL.EQ("/") && HTTP.REQ.HEADER("Accept-Language").SUBSTR(0,2).TO_LOWER.CONTAINS_ANY("suported_languages")

This means: the requested URL is / (this is the empty URL), and (&&) the browser preferred language is inside my list.

The other policy expression could be quite similar, but with a .NOT in the end, so not in my list, however I would have to examine all languages again. I also could bind this policy with a GOTO END (so no other policies get evaluated), so the other policy would just ask for HTTP.REQ.URL.EQ("/"). This will safe CPU time (we already know: It’s not in the list!)

A great, but simple, example for policy expressions!

This article is a very first answer to a question I recently received. The question was: You’re talking about web application firewalls. I’d like to know how to use the WAF for reverse proxying. … I also might use a firewall as it also contains functions like SQL injection prevention.

So what’s in a web application firewall?

A web application firewall, also known as WAF, is a shell around a web application. There are hundreds of threats, and there might be a vulnerability in my application none is aware of. There may be debug code, remained inside my application, there may be comments containing too much information like <!—now connecting to database, using dbuser/ksjd72h?kasjd-->, and you’ll never know. All this might help to break the application.

A WAF will try to address several threats in a very streamlined way. There are general approaches like removing comments. No comment is needed, so they all have to go (however: It’s rather common to place scripts in comments to keep them from being displayed, so we will have to leave comments containing scripts). There are pattern based approaches addressing well known attacks, and there are application descriptions allowing only traffic known to be not malicious.

So what’s the difference to a traditional fire wall?

First of all, a fire wall is a L3/L4 device. As such it’s a packet based animal. It can, of course filter malicious packets. But it can’t filter malicious traffic, if it’s thoughtfully distributed over several packets. A firewall has very limited abilities about data streams. And data streams is that we face in web attacks.

A NetScaler is – in most deployments – a proxy, so it’s a L7 device. Being a proxy it handles data streams instead of packets.

The negative approach

A WAF always supports a negative approach: Well-known malicious traffic gets blocked. Think of a HTTP request for an URL ending with /etc/password on a LINUX host. We would expose sensitive information if a request like that is successful. This negative approach is pattern based. A Citrix NetScaler contains lists with hundreds of well-known malicious requests.

And we would block “incomplete” requests, so requests not containing the usual header fields (Accept-Language, User-Agent, …). This may block DDOS tools like LOIC!

The positive approach

Unfortunately we don’t know all the risk out there. Even worse, we don’t know all weaknesses and vulnerabilities of our applications. We would – of course – fix them immediately if we just knew!

However we know how our application works: We know all URLs allowed we know about cookies and so on. We know the length of URLs, query strings and cookies. So we can describe all allowed traffic and deny all the rest. The positive approach is quite similar to what we do with a fire-wall: we deny everything and allow non malicious traffic only. At the same time we will also replace (for example) all ‘ with ' that’s a very efficient was to prevent SQL injection attacks from occurring.

The hard peace of work is finding out what’s going on inside our web application. We have to white list all allowed URLs (in NetScaler we call these start URLs). This may be done using literals or Pearl compatible Regular Expressions (RegEx). In addition we may even deny certain URLs (in NetScaler we call it Deny URLs). We mainly use it if we allow everything inside a certain subdirectory (let’s say: ^https?://www\.Example\.com/images) and need to deny access to ^https?://www\.Example\.com/images/secret.jpg.

An important one is called “cookie consistency”. I once found a cookie containing 2 fields: Costumer_Number and Logon. Of course I changed it to Costumer_Number=1 and Logon=true. This made me the first costumer (and owner) of the shop. I had been able to change all I wanted (also prices, see all costumer data and so on). No doubt, that’s a bad mistake inside the web app. What can we do? We could, for example, encrypt these cookies, or add a hash to them and protect them from being tampered.

Another bad thing is loss of confident information. A partner company of mine uses a blog to communicate to costumers. They also use the same blog internally, but internal readers may see content with certain keywords that’s not available from outside. This company had a top secret project, called (let’s say) Middle-Earth. This project was planned to be announced at a great conference some month ahead. Unfortunately one of the guys working on project Middle-Earth forgot to specify the proper keyword (like I use to do all the time), so a document containing project specifications about project Middle-Earth was leaked. A WAF could contain blocking rules (“Content-Type”), containing words (numbers, whatever you could specify using RegEx) preventing documents containing this words from traversing the WAF. Same can be done about credit cards (NetScaler supports American Express, Diners Club, Discover, JCB, MasterCard and Visa). So these credit card information can’t leave your webserver, even if your application fails, or an attacker was able to access your log-files. This is a must have to get certified PCI-DSS

There are several more things to do. I just gave a short description about possible threads and what a WAF could do for you.

How to protect a website using Citrix NetScaler?

Well it seems to be easy. A nonsense question. We may use AppQoE (Application level Quality of Experience), a feature introduced with NetScaler version 10, so it’s quite an old feature. Let’s start.

AppQoE is enterprise edition

My first starting point was E-Docs. Let’s be honest: the guy in charge did not really know what he had been talking about. No break through from there. Citrix Blogs! Unfortunately AppQoE is not their pet child. I found just one single article about this subject (telling me AppQoE is great). Raghu Varma Tirumalaraju did not even mention this feature in his book. Marius Sandbu did, however he did not even go as far as E-Docs.

So there seems to be no information on a feature everyone needs?

That’s a bit of surprise as DDOS attacks had been very prominent in media. Think of Twitter, Spotify and some more being down last summer?

So let’s start the other way round: I will (D)DOS a test website. I googled for tools and found LOIC. LOIC is widely used by groups like Anonymous to DDOS servers. LOIC is a quite simple tool sending simple HTTP requests to a server of choice. I loaded LOIC (don’t even think of doing so in UK, they’ll send you to prison immediately, most countries would allow you to use it against your own servers) and fired some thousand requests to my test server, but the server was still responsive. In fact, it was hardly affected. I found some thousand messages in /var/ns.log:

Mar  2 13:28:06 <local0.info> 10.255.0.250 03/02/2017:13:28:06 GMT 82e6de130138 0-PPE-0 : default APPFW AF_400_RESP 49586 0 :  172.19.54.5 559567-PPE0 - test.lab http:/// Bad request headers.No host header <blocked>

Yes! NetScaler blocked all LOIC’s requests, they didn’t pass through. It had been my WAF (Citrix NetScaler Web Application Firewall) protecting my web server.

Damn good news, thanks, WAF!

Protecting from LOIC is an easy one, you could also protect your web server using Citrix NetScaler responder policies on standard edition. Simple do a drop if HTTP.REQ.HEADER("Host").EXISTS.NOT. There may be some other headers missing like User-Agent, Accept, Accept-Language, Accept-Encoding. So you might end with an expression like HTTP.REQ.HEADER("Host").EXISTS.NOT || HTTP.REQ.HEADER("User-Agent").EXISTS.NOT || HTTP.REQ.HEADER("Accept-Language").EXISTS.NOT. In the end, you would check if requests coming is are in compliance with RFC 2616.

Just a, maybe stupid, question: Why do they use a simple tool like that? It’s easy to use, free, can get triggered via IRC. It’s not that high sophisticated, but they don’t care, they just hack some more machines (or IOT devices). It does not need the web server to crash, it’s enough to overload the ISP. Mastercard, Visa, PayPal  and some more might have survived the operation Avenge Assange attack in 2010, but who cares if their internet connections suffers under endless congestion queues causing all packets to be dropped? So you see: Even a great thing like a NetScaler would not have been of any help.

Anyway LOIC was not my tool. I changed to HULK. HULK is just some smart lines of Python 2.7 code. The requests look pretty much like legitimate requests, so they pass my WAF easily and kill my server. They can do even more, you have to specify a target URL. HULC ist launched like that: c:\hulk.py http://test.training.lab/search/showallitems.php. It’s quite easy to change the code, if you need to do so. HULC also adds random parameters, and uses random headers, so successive requests don’t look the same. It would be hard for a NetScaler to defend a website from a HULK attack by simply using responder policies, especially if someone would randomize the URL in use. This would be just some more lines of code in HULC, not a big deal, even for a non experienced Python programmer like me. (My approach would be a responder policy using the Citrix NetScaler Rate Limiting feature to find the most frequently used Source IP or URLs and block them).

So let’s go into Citrix NetScaler AppQoE.

Citrix NetScaler AppQoE in a nut shell

AppQoE means Application level Quality Of Experience. It’s well hidden as you’d suppose it to be in Security, however it’s in App Expert.

In AppQoE Citrix blended some features like priority queuing. Queues are an important part of AppQoE. It simply uses two queues: one for not known legitimate requests, one for well known legitimate requests. Policies are used to assign users to certain queues. There are four levels:

• High
• Medium
• Low
• Lowest

During regular work queues are not used, all requests get treated equally. If the NetScaler considerers itself to be under attack it starts using priority queues. So we have to tell NetScaler how much traffic we can (or are willing to) handle. We turn on AppQoE if traffic exceeds these limits. Maybe its the most difficult thing! Be careful to not cut down your Christmas business!

If we are under attack we have to find out, if the request is legitimate, or not. It’s a bit tricky, as some of these bad guys out there are quite smart. We have two methods: one is transparent to the user (a Java script is run on client side), however there is a chance for the attacker, the other one presents a captcha to the user, so the user has to prove he got a brain. This is a huge obstacle to overcome. The method is defined inside of policy actions. If the user is a legitimate one we start a session for this user and assign it to a privileged queue.

If you click on AppQoE feature on the left plane you see Configure AppQoE Parameters.

AppQoE general Parameters

What do these parameters mean?

• Session Life (secs) We will remember this user as a legitimate user for XXX seconds. It defaults to 300s, 5 minutes. Don’t set the session life parameter too low. Setting it lower than a minute may result in endless loops of testing weather this client is legitimate or not, as the session timed out during testing, so the results are no more valid. I tend to not change it. Never go below 60. On the other hand, an attacker would just neet to solve the captcha once per bot id the value is too high.
• Average Waiting Client queue would not grow beyond a very low number in normal use: Your web servers should be fast enough to handle all legitimate traffic. That’s why I tend to lower this value. Sizing depends very much on web site useage. How many client requests may wait until we consider this website to be under attack? It defaults to 1000000.
• Alternate Response Bandwidth Limit(Mbps) how much bandwidth may all responses waste (default 100 MBt.). We send alternate responses as soon as we exceed this value.
• DOS Attack Threshold The everage number of clients that can queue up without triggering DOS protection. It defaults to 2000. If your website won’t need to support more than 50 you would add 50 (don’t forget to add some spare users for Christmas business!). Don’t set it too tight, exceeding this value may influence user experience.

AppQoE policies

Like any policy it got an expression and an action. The expression may either be a simple true value, or, if you need to be a bit more granular, be based on type of content or URL. Be sure to not exclude content on your server. This would allow a successful attack!

The Action

Like any action this one needs to get a name. I usually call them AppQoE_act_…

There are three types of actions, and you can’t change the type after creating the policy, so spend a moment to think of it:

• None means no action is taken when a threshold is reached.
• ACS means there is an alternate content source (an other LB vServer), so NetScaler won’t handle that traffic. This is what my screen-shot shows.
• NS means, traffic is handled by NetScaler. I will go with this one.

We may use an other TCP profile than usual in case of an attack. It can be used to use advanced TCP features to mitigate damages.

The (high, medium, low, lowest) may get used to split up traffic and handle more important traffic prior to unimportant one. So, for example, we could create a policy handling movies (HTTP.REQ.URL.ENDSWITH("AVI")) and put them into low priority, as wa are under attack and suffer under a lack of bandwidth, and more important data into medium or high. Keep aray from lowest as lowest is also used for malicious requests.

When the policy queue size, that means the number of requests queued for this policy, reaches Policy Queue Depth threshold value, subsequent requests are dropped to the lowest priority level. That means, they don’t respect your priority settings any more. Different to Queue Depth this one is about a policy, not about a vServer as a hole.

The Queue Depth threshold value is a per priorirty level. If the queue size (number of requests in the queue of that particular priorirty) on the virtual server increases to the specified queue depth value, subsequent requests are dropped to the lowest priority level. That means, they don’t respect your priority settings any more. Different to Policy Queue Depth this one is about a vServer, not about this specific policy.

If the Maximum Connections number is exceeded we place even legitimate requests into lowest priority queue. Either this one or Delay has to be specified.

If the Delay (microseconds) is exceeded we place even legitimate requests into lowest priority queue. Either this one or maximum connections has to be specified.

The Alternate Content Server Name* is the name of a vServer if you use ACS. Requests will get sent to this server instead of using the original one. It’s a kind of sorry server. The Alternate Content Path* is the path on this server, for example /div/WeAreSoSorryWeGotHacked.html

The Custom File is is a file displayed to users considered to be illegitimate. It’s an optional parameter. You don’t specify it. Use the menu item under AppQoE on the left side to import a file.

The DOS Action can be a SimpleResponse, or a HICResponse:

• SimpleResponse is a response containing a Java script. The client needs to have Java scripts enabled. This script dies the maths and returns the result to the NetScaler. The NetScaler will treat this user as a legitimate one if the session is not already timed out. Advantage here: The user won’t see what’s going on. Disadvantage: A bot may also be able to run the script.
• HICResponse is a response containing a rather simple captcha. The user has to solve the problem. It’s quite difficult to trick this one.
  There is a prerequisite for this. You need to open up a putty session, then:

  > shell
  Copyright (c) 1992-2013 The FreeBSD Project.
  Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
          The Regents of the University of California. All rights reserved.
  
  root@82e6def331a4# nsapimgr -ys appqos_captcha=1
  Changing cfg_appqoe_captcha_enable from 0 to 1 ...
   Done.
  root@82e6def331a4#  echo "nsapimgr -ys appqos_captcha=1" >> /flash/nsconfig/rc.netscaler
  root@82e6def331a4#
  > _

  This will make Citrix NetScaler able to send the captcha to the client box.
  

  ---

  Logging

I did not read anything about logging. That’s a pity. In fact I could not find out weather my NetScaler supposes itself to be under attack or not. Wait, there is one thing:

stat appqoe policy <policyname>
Output may look like that:

> stat appqoe policy AppQoE_pol_2delete
AppQoE Cumulative Policy Statistics Summary

Cumulative Policy Stats:
Counter                                    Rate(/s)                Total
Server TCP connections                             0                    0
Client TCP connections                             0                    0
Requests received                                  0                    0
Requests bytes                                     0                    0
Responses received                                 0                    0
Response bytes                                     0                    0
ThroughPut(Kbps)                                   0                    0
Alternate responses sent                           0                    0
Alternate responses bytes sent                     0                    0


Timing Stats:
Counter                                    Rate(/s)                Total
Average Client TTLB                                0                    0
Average Server TTLB                                0                    0
Average Server TTFB                                0                    0

 Done
>

> stat appqoe policy AppQoE_pol_2delete
AppQoE Cumulative Policy Statistics Summary

Cumulative Policy Stats:
Counter                                    Rate(/s)                Total
Server TCP connections                             4                 2051
Client TCP connections                             4                 2154
Requests received                                  4                 2051
Requests bytes                                  1251               733826
Responses received                                 4                 1916
Response bytes                                  9489              5048057
ThroughPut(Kbps)                                  84                  587
Alternate responses sent                         113               576640
Alternate responses bytes sent                414706           1610897508


Timing Stats:
Counter                                    Rate(/s)                Total
Average Client TTLB                                0                    0
Average Server TTLB                                0                    0
Average Server TTFB                                0                    0

 Done
>

Means: under Attack. Same policy, same amount of traffic, but different parameters.

Unfortunately we are not yet finished, continue reading with part 2

I hope this helps by a little bit. And I hope Citrix will do a better job in future about explaining this – rather important – feature!

Yesterday I published a blog about DDOS- protection. I used the Citrix NetScaler AppQoE feature to do so. That’s nice, but not enough. I still could beat my server to a pulp easily. Just 10 clients launching a DDOS attack using HULK had been enough. I can’t throttle down the number of users to just 10!

WTF?

Basically AppQoE will just limit the number of users (actually it’s the number of source IPs) being able to connect, not the number of requests. No matter how deep you set the limit, even a single user may screw up a web server, given his connection is fast enough. No protection from NetScaler AppQoE here!

What to do next?

What’s wrong? We limit the number of users. That’s a damn good story as it is possible to do serious attacks with distributed users. But we did not limit the number of requests per user! How can we limit the number of requests per user? Or, the number of requests to a certain URL?

Think you are the boss of a police station. One nice and sunny morning you would tell one of your officers to check cars (that’s the Selector, he has to select cars). Unfortunately a single officer would not be able to check all cars, so you tell him to stop every 10th car. This would be the Idenitfyer. Both together give a good job description: “check every 10th car!”. But it takes an officer to do the job. That’s what our (responder) policy would do.

So we have to set up a stream selector, a stream identifier and a responder policy.

Rate Limiting is the feature or choice!

The very best thing: Rate limiting is standard edition, so everyone may use it. How could Citrix NetScaler Rate Limiting look like to protect from a DOS attack?

The NetScaler Rate Limiting Stream Selector

The first ting to do: select the kind of traffic we want to limit. There are two kinds of approach: the client IP and the URL; or a combination of both. There are built in stream selectors we could use: Top_URL (HTTP.REQ.URL) and Top_CLIENTS (CLIENT.IP.SRC). I did a trace of an attack done by HULK. HULK always appends some random strings like ?KCDPT=SRKGT. So Top_URL is not useful, instead we would need to create our own stream selector based on HTTP.REQ.URL.PATH to strip the query string.

I created a stream selector using both, URL and IP. This is a list of URLs per IP.

add stream selector rate_sel_URL_IP CLIENT.IP.SRC HTTP.REQ.URL.PATH

The stream selector “does not do anything” but selects streams. So our next step would be:

Create a NetScaler Rate Limiting Stream Idenitfyer

The stream identifier checks weather or not to pick a stream. There are several parameters:

• The Selector is the stream selector we created before.
• We have two Modes: Request Rate and Connection. Request Rate tracks requests (that’s what we need), Conection counts connections. That’s not useful in here, we want to block requests, not limit connections!
• The Limit Type may be bursty or smooth. Smooth wants requests to come in evenly, while bursty would allow much higher rates as long as the total number does not exceed the limit for the given time slice (so if you define a limit of 10 and a time slice of 10s there may be 10 requests within a second, but no more during the next nine seconds if you go with bursty and 1 per second for smooth). I go with bursty as this is how requests usually come in.
• The Threshold is what you have been looking for: The number of requests (or connections). Select them wisely! If you permit too many you may kill your server, if you don’t allow enough you will block legitimate requests of your users.
  
• The Time Slice (msec) is the amount of time we use to count our requests, in milliseconds. So 5,000 would be 5 seconds, 60,000 a minute.
• Maximum Bandwidth (Kbps) is an additional limit you may specify for connections. We may leave it blank as we are not interested in connections.
• Traps is the number of traps to be sent in a time slice. 0 means no traps, I guess 1 would be fine, even if your time slice is several minutes.

Usually a single user won’t request a certain URL more often than 10 to 100 times per 5 seconds (be aware of spacer GIFs, some web site programmers use them excessively! I have seen hundreds of requests within a second. And do you know what I think of the web site programmer? What a bloody idiot! But he’s just a web site programmer, no engineer, maybe that’s why).

add ns limitIdentifier rate_id_URL_IP -threshold 10 -timeSlice 5000 -selectorName rate_sel_URL_IP -maxBandwidth 0 -trapsInTimeSlice

The responder policy

So we could find out to malicious requests. But in the end we will need to react. Our stream selector, our stream identifier don’t respond to client requests on their own. That’s why we need to create a responder policy.

The responder action

What to do? There is no best practice. I tend to drop all malicious requests, but you also could answer with a 404, not found, a 500, server error, redirect users to a religious site of your choice (to force him to think about his sins), or with a custom error page. It’s up to you. I will simply drop.

The responder policy

add responder policy res_pol_drop_DOS "SYS.CHECK_LIMIT(\"rate_id_URL_IP\")" DROP

So you can see, my action is a DROP. I tried to do some logging (but it didn’t work). The point of interest for you will be the expression: SYS.CHECK_LIMIT(“rate_id_URL_IP”)

I hope you like it. Leave a comment if you find something you don’t understand (or don’t agree). I also accept comments telling me I did a good job

Why do I love HDX on UDP in Citrix XenDesktop and XenApp? (HDX Enlightened Data Transport EDT)

Well, I’m mainly a network guy. So I’ll take a look at this brand new feature from networking perspective.I’ll start from scratch, so I don’t assume you understand network protocols.

But let me tell you a joke about UDP first:

I got a short joke about UDP, and I don’t care if you got it!

You may, or may not get my joke. Of course I care, so I’ll explain: TCP has guaranteed delivery. So every packet you send will be delivered. UDP doesn’t care if you got the packet, or not.

It also doesn’t care if packets arrive in the right order.

The situation with TCP

TCP uses so called sessions.

It takes 3 packets to establish a session:

client → server: Syn + sequencenumber (Syn means: let’s talk, the sequencenumber is a random number)
192.168.0.1 → 192.168.0.13: SYN, seq. 1000

server → client: Sync/ack + sequence number + acknowledge number (Ok, let’s talk. client’s sequence number is server’s acknowledge number, server’s sequence number is a random number too)
192.168.0.13 → 192.168.0.1 SYN/ACK, seq. 2000, ack. 1000

client → server: Ack  + sequence number + acknowledge number (Ok, I’ve heared, you agree about talking. The client increments his sequence number by 1 and uses server’s sequence number as an acknowledgement number)
192.168.0.1 → 192.168.0.13: ACK, seq. 1001, ack. 2000

This is a bit of overhead, but it’s overhead during session establishment only, so we don’t really care about it.

Sequence/acknowledge numbers are very important. If a client sends a packet (let’s say number 1001) it has to get an acknowledgement for this package (1001).

sending packets

My explanation is based on RFC 793.

This is a very basic principle all over TCP. It’s really great. Let’s say, a server sent packets 20001, 20002, 20003 to the client, however the cient only received packet 20003 and 20001 (so you see, wrong order here) the client will wait for a pre defined amount of time to get 20002. If it does not appear, the client will send an acknowledge for 20001 only. So the server will resend 20002 and 20003. This procedure is called retransmission. Simple like that. (Notice: we retransmitted 20003, even though it already arrived on client side.

To make file transfers fast, client and server both use a so called sliding window, a buffer to send and receive.

If a sender wants to send 10 packets, it’s buffer size is 5 packets, it will put the first 5 packets into the sending buffer and starts sending packets as fast as possible, starting from first packet.

The receiver (the client in my example) has a receiving window with a size of 5 packets as well. It will put all received packets into it’s buffer. As soon as it received the 3rd packet it will send an acknowledge for the 3rd package to the sender (confirming the 3rd package implies also package 1 and 2). The sender now deletes the confirmed 3 packets (the ones I coloured in pink) from it’s sending buffer, shifts all packets left by 3 positions, and add 3 more packets into the – now free – last 3 positions of the sending buffer (the pail pink ones), so they can get transmitted.

The client will receive the next 3 packets, and confirm as soon as the 3rd packet is received.

This mechanism guarantees smooth data transmission, the bigger the buffers are, the more latency we can tolerate without the need of an interruption of the sending process. TCP, therefore, is not very sensitive of latency. In internet we usually use big sending and receiving windows (buffers), so downloads stream smoothly.

The only delay here: If there are less than 3 packets left: The client will wait for more packets for a specified amount of time before confirming the already received packets. This delay does not matter if you think of a download containing of hundreds of packets.

Where is the problem about HDX/ICA?

I’m currently typing this blog article into a published Mozilla Firefox on XenApp. If I type an I, for example, my client sends an I to the server. That’s just one packet to send (less than 3!!!). The server now waits for 2 more packets to come, however I’m typing quite slowly (aprox. 60 strokes per second), so there won’t appear an other packet for a very long time (if you think in processor cycles). Server’s TCP has a timer. If that timer is down to 0 my server’s TCP will send an ACK to my client and, at the same time, forward my request to the Citrix services. So my Browser application just now receives my message! This is additional latency! And latency is not what we want to see!

So there are several delays: Network latency from client to server, the server waiting for more packets, the server processing my request, sending it and network latency (again) until the client finally receives the response. Oh, I forgot: A Citrix server’s response will be (in most cases) less than 3 packets, so my client’s TCP protocol stack will also have to wait for a 3rd packet to come. Unfortunately there won’t be a 3rd packet, so the client got some delay until it sends an ACK to the server and forwards the packet to my Citrx receiver.

The more interactive your application is the worse things get.

An other TCP problem

There is an other aspect I have forgotten to mention. I have to thank Marcus Kohlschmidt, he mentioned it yesterday.

TCP guarantees delivery. 100% correct delivery. There are checksums to make sure every single packet got delivered, delivered correctly.

Now there is no point in forwarding garbled packets. That’s obvious. Every single router will therefore check checksums to avoid forwarding corrupted packets. This router will discard corrupted packets, and cause packet loss. Mail problem here is not just silent packet loss, but much more important the overhead of calculating checksums. Every single router will have to do this, causing congestion on routers. The more hops we have to pass the more overhead it causes. UDP packets don’t contain checksums, so they are forwarded immediately, and may eventually overtake a bunch of TCP packets. So there may be less latency for UDP than for TCP. Unfortunately this is not true if we tunnel UDP inside SSL.

Why can UDP handle Citrix XenApp / XenDesktop HDX (ICA) better?

UDP, you remember?, does not have sessions at all. A client will send a packet to the server. The server will immediately process it, if it receives the packet. That’s good news.

If it did not it won’t. That’s bad news. Even worse: If packets appear in wrong order they get forwarded in wrong order. So let’s say I type “if”, but for some reasons (why? Ask the network guys) the f arrives in front of the i, my Citrix server will think of fi. That’s unusable.

Citrix had to build TCP functionality into HDX (former ICA) to overcome issues like that. So UDP based HDX packets need to get numbered. HDX won’t send an acknowledge, but instead it will send the server reply containing the acknowledge. In my example my server will receive packet number 2, the f, and packet number 1, the i, so it will know they are in wrong order. Or, if it won’t receive packet number 1 at all it will tell the client: I have received a packet number 2, but number 1 is missing.

If a client sends a packet, but does not get a response from server in a timely manner, it will have to resend. So you see, it’s TCP functionality, built into HDX/ICA, so it may be sent on UDP instead.

What about NetScaler Gateway in HDX/ICA proxy mode?

Well, we use SSL here. HTTPS. SSL is a tunneling protocol, so HTTPS would be HTTP tunneled in SSL. SSL will, always, be TCP based.

It’s a great advantage to go with standard protocols. It’s easy to use proprietary protocols using fancy ports like UDP/2598 or even UDP/443 in a closed environment. The only thing you have to do is: tell your firewall staff to permit this ports. But it’s a mess in internet. How can you talk to the firewall staff of your (possibly global) costumers? They will simply refuse. How can you talk to firewall staff of “all hotels in the world“? It’s impossible!

So we still have to go with a TCP based tunnel using 443. All advantages and disadvantages of TCP are true about this tunnel. But inside this tunnel we have light weight UDP packets. The effect of latency will be cut to half. We don’t have to wait for all these timers to count down to 0. HDX/ICA will be much smoother.

Requirements to use UDP-based HDX/ICA

There is a description by Citrix. Maybe you prefer that one over mine.

In Citrix XenDesktop 7.12 I found following paragraph:

New enlightened data transport layer (for evaluation only). IT administrators can evaluate a new HDX data transport layer designed for challenging long-haul WAN and Internet connections. This alternative to TCP delivers a superior user experience while maintaining high server scalability and efficient use of bandwidth. This new transport layer above UDP improves data throughput for all ICA virtual channels including Thinwire display remoting, file transfer (Client Drive Mapping), printing, multimedia redirection and others. In 7.12, this feature is for evaluation only and is disabled by default. It can be enabled in a non-production environment with a new policy setting, HDX Enlightened Data Transport. Set the new policy setting to Preferred to use enlightened data transport when possible, with fallback to TCP. In this release, this feature requires Citrix Receiver for Windows 4.6 or Citrix Receiver for Mac 12.4. For more information, see ICA policy settings.

So it started with 7.12, however it was for experimental use only. With 7.13 it is fully supported and implemented. Everything is there: The VDA installation opens firewall ports, Director is aware if UDP based HDX/ICA and much more. So it’s stable and ready to use with 7.13.

Enabling UDP based HDX/ICA in Citrix XenApp / XenDesktop

You won’t be surprised to hear: It’s a policy! So I open GPEdit.exe and select my GPO

We are looking for HDX adaptive transport. The official name would be HDX Enlightened Data Transport Protocol, but Citrix naming conventions are inconsistent here. The policy itself will look like that:

Default setting is Off. This means: we use TCP based transport only. I suggest changing to Preferred (so we prefer using UDP based transport, but we may use TCP based transport if a client is unable to use UDP, this would be true for an old client, or a fire wall blocking UDP/2598 or UDP/1494). Don’t use Diagnostic Mode. This will force UDP based transport. Elder clients won’t be able to connect.

My current session is UDP based. It is visible in Citrix Desktop Director

Unfortunately session details in connection centre does not show UDP transport (way to go, Citrix!)

Which Receiver?

It’s a brand new feature, so we need to use the latest version of Citrix receiver: 4.6. It is not supported.

What about the NetScaler side?

Well on NetScaler side we need at least NetScaler version 11.1 Build 52.13 (March 1st 2017). The release notes read like that:

The NetScaler Gateway appliance now supports the HDX Enlightened Data Transport (EDT) as a data transmission path. EDT provides a high definition in-session user experience of virtual desktops for users running a Citrix Receiver.

[From Build 51.21] [# 659795, 666135]

So that’s it. Upgrade to (at east) 11.1 Built 52.13 to enable HDX Enlightened Data Transport (EDT)! No policies needed. However you have to enable DTLS on your NetScaler Gateway and rebind certificates.

So let’s enable DTLS: (in my case DTLS already was enabled after upgrading, if your DTLS is enabled you should skip this section).

You’ll find this dialogue after opening your VPN vServer (NetScaler Gateway → Virtual Servers), and then click edit (the pen) with Basic Settings. The setting is hidden, you have to click › more.

After doing this you will have to unbind SSL certificates and rebind (I followed Citrix product documentation about that). I don’t understand why this step is necessary.

Your firewall …

… has to permit UDP 2598 and UDP 1494 from NetScaler SNIP to all your HDX/ICA servers (Citrix XenApp / XenDesktop servers).

Citrix NetScaler Logging and policy trouble shooting

Some times it’s quite hard to understand what’s going on. There is so much mystics about policies. And it’s even harder to understand what went on (past tense). “Johannes, there had been several problems connecting to <any blabla application here>” “I’m sorry, I can’t know. What did they do? The request could have been blocked by a responder policy or by WAF”. Did you ever get an answer to a “What did you do?” question?

So we have to do reverse engineering. The only thing we know is: There had been something going on, and it failed. Maybe we know some of the symptoms.

That’s a mess. And we don’t need to act blind like that. There are logs in a Citrix NetScaler!

Where do logs too?

All logging goes to /var/log/ns.log. This log gets periodically archived and recreated. That’s the source of information if our Citrix NetScaler web application firewall (WAF) blocked!

How do WAF-Logs look like?

Let’s have a look at some entries:

Mar  9 09:26:28 <local0.info> 192.168.0.25 03/09/2017:09:26:28 GMT 82e6de130138 
0-PPE-1 : default APPFW APPFW_STARTURL 5852 0 :  81.153.82.18 40102-PPE1 /PctEsh
SVh4Uv9sK/sBQOR5GSdg0001 app_fw_blog_data Disallow Illegal URL: https://blog.nor
z.at/category/common-citrix-themes/page/3/?as_url_id=AAAAAAW5jxPwYiwdw2xu-S6V6uY
6IgElFoDo1HPM_1hQWBYABzbf-StjgFD_ZnsEPbr_nDarC6Y-P6y_CgyS0u_ubJBmqVmA7df3rvHA5Io
PE2Hq6w%3D%3D <blocked>

So we see it’s a Citrix NetScaler Web Application Firewall (WAF) log (APPFW). It’s the start URL function blocking (APPFW_STARTURL). The policy is called app_fw_blog_data. The vServer is at IP 192.168.0.25. The URL, my attacker (IP address 81.153.82.18, British Telecom, if IP location finder is right) wanted to access, was https://blog.norz.at/category/common-citrix-themes/page/3/ with some parameters attached to it. Taking a closer look at it I can see: It’s a non existing page, it’s a forceful browsing attack!

Let’s see a second one:

Mar  9 09:36:07 <local0.info> 192.168.0.25 03/09/2017:09:36:07 GMT 82e6de130138 
0-PPE-0 : default APPFW AF_400_RESP 193 0 :  36.84.91.100 14386-PPE0 - norz_df h
ttp:/// Bad request headers.No host header <blocked>

This one is also by my WAF (APPFW). It’s the AF_400_RESP function, the HTTP request syntax checker. vServer, again, 192.168.0.25. The request came from 36.84.91.100, an IP from Indonesia (PT Telkom Indonesia). The WAF policy was norz_df. There had been bad request headers, so the WAF did not let this request pass through. I guess it’s a request generated by a scan- bot.

Mar  9 10:08:28 <local0.info> 192.168.0.25 03/09/2017:10:08:28 GMT 82e6de130138 
0-PPE-1 : default APPFW APPFW_DENYURL 6195 0 :  56.181.27.117 43870-PPE1 - norz_
df Disallow Deny URL: https://norz.at/test/ for rule pattern="/test/?" <blocked>

This one: also by my WAF (APPFW). It’s the APPFW_DENYURL feature, so i’s a denied URL. The rule triggered was 6195, blocking access to /test/?. The rest like the ones above. Easy like that.

Mar  9 10:13:09 <local0.info> 192.168.0.25 03/09/2017:10:13:09 GMT 82e6de130138 
0-PPE-1 : default APPFW APPFW_BUFFEROVERFLOW_URL 6241 0 :  56.181.27.117 44091-P
PE1 - norz.at URL length(1420) is greater than maximum allowed(1024): https://no
rz.at/agb?1234567890123456789012345678901234567890123456789012345678901234567890
12345678901234567890123456789012345678901234567890123456789012345678901234567890
12345678901234567890123456789012345678901234567890123456789012345678901234567890
12345678901234567890123456789012345678901234567890123456789012345678901234567890
12345678901234567890123456789012345678901234567890123 <blocked>

Again the WAF (APPFW), the subystem was APPFW_BUFFEROVERFLOW_URL. So there was a buffer overflow about the URL. If you take a look at it you see: there is an enormous query string. (the attacker was me, so it’s easy to find out)

What about other policies like responders?

<put 4 letter word of your choice here> there is no logging about responder policies.

Is this true?

Sorry to say: yes. So we can’t reproduce what’s going on.

But?

You long for a but? OK. I have a but!

But you can make them logging! They don’t log, unless you make them logging. Great news, isn’t it?

How to make (responder) policies log

Let’s have a look at a policy:

This is a Citrix NetScaler responder policy dropping requests originating from well known malicious IPs. (IP reputation is a platinum feature).

Let’s take a closer look: There is an action, very well known to all of us (drop in this case) and there are two more actions: a Log Action and an AppFow Action. The appflow action would send data about this request to an IPFIX (NetFlow) receiver of choice, let’s say, Splunk or Solarwinds, or Citrix tools like Insight Center or MAS. Could be a great idea, but I don’t go that way. Instead I wanted this logs to appear at /var/log/ns.log

So we click to the + near to Log Action. We now ´may create our own log actions. I’ll create a sample log:

This action will write an informational message into /var/log/ns.log. This message will look like that:

Mar  9 12:01:58 <local0.info> 172.31.1.31 03/09/2017:11:01:58 GMT ns01 0-PPE-0 : 
 default RESPONDER Message 50652 0 :  "A request for / from: 36.84.91.107 was dr
opped by res_pol_drop_malicious"

So, my responder policy logs! You see, I included the URL requested and the IP address of the attacker. You may create a string of your choice, using any policy expression you like.

Log destination will be /var/log/ns.log (optional /var/newnslog/ns.log)

Where do I find these log actions? From command-line it’s show audit messageaction. Creating my policy would be add audit messageaction log_drop_malicious INFORMATIONAL "\"A request for\" + HTTP.REQ.URL.URL_RESERVED_CHRRS_SAFE + \" from\ " + CLIENT.IP.SRC + \" was dropped by res_pol_drop_malicious\""

In GUI I can find all these policies in System → Audit → Message Actions.

I hope you find this article is beneficial. Have fun!

Johannes

My NetScaler VPX ran out of disk space. I did several tricks like adding a second disk and mounting ít into the update directors, to be able to upgrade my NetScaler.

Today I found time to investigate this issue. I had no success flooowing CTX133588. I so I started looking for the largest files and directories in /var.

root@82e6de130138# du -a /var | sort -n -r | head -n 10
13033540        /var
11023956        /var/ns_system_backup
466274  /var/log
461740  /var/log/db
459822  /var/wi
434188  /var/netscaler
305070  /var/log/db/default
248492  /var/nslog
183824  /var/ns_system_backup/backup_7b829416-9965-475b-8dfb-d57d18ce9ba0.tgz
183824  /var/ns_system_backup/backup_1676b077-6132-47f5-88a5-aab4832e5c93.tgz
root@82e6de130138# df
Filesystem  1K-blocks    Used    Avail Capacity  Mounted on
/dev/md0       358830  344602     7052    98%    /
devfs               1       1        0   100%    /dev
procfs              4       4        0   100%    /proc
/dev/ad0s1a   1482638  941530   422498    69%    /flash
/dev/ad0s1e  14348494 2009796 11190820    15%    /var
root@82e6de130138#

What a big surprise! 11 GB in /var/ns_system_backup? How comes? I looked into it.

root@82e6de130138# ls -l
total 11019220
drwxr-xr-x  2 root  wheel       3584 Mar  8 17:13 WebServer
-rw-r--r--  1 root  wheel  188077595 Mar  5 17:14 backup_083f6140-a660-41d5-83f2-389403f29334.tgz
-rw-r--r--  1 root  wheel  187946493 Feb 20 17:13 backup_0bd31ba5-fa66-4e40-a912-d6c1a9ffdbdc.tgz
-rw-r--r--  1 root  wheel  187814966 Feb 14 05:13 backup_11653391-2dff-4c22-ba22-3cd9212cd7c2.tgz
-rw-r--r--  1 root  wheel  187751996 Feb  8 05:13 backup_15904e38-2e8f-49f1-ad8c-e153d9daf980.tgz
-rw-r--r--  1 root  wheel  188106376 Mar  8 17:15 backup_1676b077-6132-47f5-88a5-aab4832e5c93.tgz
-rw-r--r--  1 root  wheel  187932332 Feb 19 05:13 backup_189cc344-e721-40ae-81af-55935a0ee7ac.tgz
-rw-r--r--  1 root  wheel  187961509 Feb 22 05:13 backup_195f2b0a-8b21-4be9-b277-50220247b553.tgz
-rw-r--r--  1 root  wheel  188000564 Feb 26 05:14 backup_26ac5a11-0c8c-4034-b79b-64a94fee1ac1.tgz
-rw-r--r--  1 root  wheel  188098718 Mar  7 05:14 backup_3379fb1a-068e-4bc5-acfc-091108cece18.tgz
-rw-r--r--  1 root  wheel  187761902 Feb  9 05:13 backup_34fd7671-fad1-43b4-b662-0e3b7f221475.tgz

a total of 60 files, round about 183 MB each. In addition there was a subdirectory with the name of my admin partition, containing 60 more files, 7 kb each.

All these files had been relatively new. They had been created at 5:13 AM and PM, so my first guess was a cron job.

These tgz files are a full copy of /flash/nsconfig and selected subdirectories of /var. I could not find a cron job doing this backup, and I could not find any documentation about a /var/ns_system_backup folder.

I kept playing round, but did not find. I posted this question on dicussions.citrix.com and Carl Stalhood asked do you have MAS in place? Yes I do!

It’s MAS to be blamed for!

So I opened MAS, System → System Administration and clicked on Instance Backup Settings. MAS does backup on managed devices twice a day by default. I changed to one backup in 10 days. It should keep just 1 backup, but it failed to delete. I also upgraded MAS and hope problem is solved by now …

Versions involved:

MAS: 11.1 52.26
NetScaler: 11.1 Build 52.13 and 51.26

Why would you like to customize a 404 page?

Well It’s all about misleading information. A hacker has very limited chance to get friend with your web server. On the other way, he needs to find out as much as any possible. The more he knows, the more likely his attack will be successful. On the other hand he has to let sleeping dogs lie. With other words: He must not alarm you.

One of the most important things to know is: What kind of web server do I have to deal with?

The first source to look into is a HTTP response header called Server. Information here may be very verbose. I don’t know why this header is part of HTML standard, but actually it is.

The Server response-header field contains information about the software used by the origin server to handle the request. The field can contain multiple product tokens (section 3.8) and comments identifying the server and any significant subproducts. The product tokens are listed in order of their significance for identifying the application. (RFC 2616)

This is an example server header:

Apache/1.3.28 (Unix) mod_ssl/2.8.15 OpenSSL/0.9.7c mod_perl/1.27 PHP/4.3

In this case, it’s a very outdated Apache, using an outdated SSL module, outdated Perl and outdated PHP. It’s easy to change this information using Citrix NetScaler rewrite policies (DELETE_HTTP_HEADER and INSERT_HTTP_HEADER).

But hackes are not that stupid. They will probably verify this information. My personal next try would be: check for a non existing site. We will see a 404, page not found. Being careful I would use an existing URL, however do a minor typo, just like http://norz.at/default.html instead of http://norz.at/default.htm. You would probably not be scared if you would see a request like that watching your logs.

The next thing he would see is a 404, Not Found. It will be specific to your server, if you don’t change it. And a 404 page originating from an IIS6  would, for sure, come from an IIS 6, no matter what the server header tells you.

More reasons to change the 404 page

of course there are even more reasons to change the 404 page: customized 404 pages seem to be funny, they may help people to find the content needed, and so on.

Why not change your web server?

This would be possible. However, you would need to change all your load balanced web servers. There is an other reason: Responder policies. I will never return a “401 Unauthorized” or “403 forbidden“. I would rather return a “404 not found”. Being a hacker I would be very excited to see a 401 or 403!

I would think: here it is, but someone protects it from being accessed. But how could I find out what’s going on, if a Citrix NetScaler uses exactly the same 404 page as the original web server? I would probably think the file is not there.

My solution

My first attempt was creating a simple rewriting policy changing the body with something like “HTTP/1.1 404 OK\n\r\n\r<html><head><title>404 File not found</title></tead><body><h1><font color=\”#802020\”>404 File not found!</font></h1><p><font color=\”#802020\”>The file you requested is not on this server.</font></p></body></html>” in it.

The length of the text is limited, so this is not a good solution. And I would rather like to place the file “somewhere” on my web server, so it’s pretty easy to change.

I spent some time thinking what to do and made up my mind to use the HTTP callout feature. It was my first ever attempt to use HTTP callout, and I’ll describe how it works.

NetScaler’s HTTP callout feature

HTTP callout is intended to be used in policies to check something, i.e. an IP address, against a web based service. So I could send an IP address (CLIENT.IP.SRC) to a web server containing an IP black list. This web server then would respond with something indicating good or bad.

I do something completely different: I will retrieve the content of the 404 page from a web server. To do so I have to navigate to App Expert -> HTTP Callouts.

Like any policy it has to get a name. I do my callout to a vServer, so I have to specify the server here. My request will be attribut based, that means, I will be able to send regular HTTP requests, mine is a HTTP GET. My web server uses several host names for various virtual pages, so I have to specify a proper host expression. This makes sure, we retrieve the file from the right source. The URL Stem Expression is the URL we want to retrieve.

We scroll down to the bottom and select the return type TEXT and the expression should be HTTP.RES.BODY(65538). The number is the number of bytes to retrieve.

So, my policy will connect to a NetScaler vServer called cs_vsrv_norz.at to retrieve a file called /notfound.htm, setting the header Host to norz.at (i.e: http://norz.at/notfound.htm). It will then return all the body of this file, containing links to style definitions, pictures and so on.

command line version:

add policy httpCallout callout_retrieve_404 -vServer cs_vsrv_norz.at -returnType TEXT -hostExpr "\"norz.at\"" -urlStemExpr "\"/notfound.htm\"" -scheme http -resultExpr "HTTP.RES.BODY(65538)"
set policy httpCallout callout_retrieve_404 -vServer cs_vsrv_norz.at -returnType TEXT -hostExpr "\"norz.at\"" -urlStemExpr "\"/notfound.htm\"" -scheme http -resultExpr "HTTP.RES.BODY(65538)"

The rewrite policy

The rewrite policy should be a very simple thing:

The NetScaler rewrite action using a HTTP callout

add rewrite action callout404 replace_http_res "SYS.HTTP_CALLOUT(callout_retrieve_404 )"

It’s a replace policy. Expression to choose target location is all of the HTML body, so HTTP.RES.BODY (65536). To be more precise, it’s the first 65536 byte of the body (a 404 page typically is by far smaller). The Expression is the text we will use to replace the former body with. It is the HTTP callout request, in my case SYS.HTTP_CALLOUT(callout_retrieve_404).

The NetScaler rewrite policy

add rewrite policy rw_pol_404 "HTTP.RES.STATUS.EQ(404)" rw_act_404

This policy will get applied if  the HTTP response status is a 404 (HTTP.RES.STATUS.EQ(404)). I then bound this policy to my web server. That’s it. It was pretty easy.

I was so enthusiastic, when I found out about NetScaler admin partitions! What a great extension to existing NetScalers! However I got disillusioned finding out about limitations. It took me some time to find out how to overcome this issues, but there are still some features missing.

The feature I missed most is doing traces. It’s not listed in the compatibility list, so it’s intended to be there. But it is not! If you click into System and Diagnostics you’ll see just very little content, and definitely no nstrace (this is about NetScaler versions up to 11.1 48.10).

Citrix documentation is always right, and if it’s not, it’s right even though. So I tried to do a nstrace from commandline. It started and stopped without any problem. Unfortunately I could not find the output of my nstrace in /var/nstrace subdirectory.

So I searched for it, and found it in /var/partitions/<partitionname>/nstrace.

So that’s how I do an nstrace inside a NetScaler admin partition:

I use putty to connect to my NetScaler. Masochists might prefer to use the built in terminal from GUI, however I don’t tend to masochism.

switch partition <partitionname>
start nstrace -filter “CONNECTION.SRCIP.EQ(<source IP>)” -size 0 -time 3600 -link
stop nstrace

So I log into my Citrix NetScaler. I change into my partition (currently partition names can’t auto complete, so make sure you know the name; show ns partition will list all partitions.

Next I start the trace. To do so I follow CTX120941.

start ns trace will simply start the trace. -filter will filter a connection. Usually you would use connection objects like CONNECTION.SRCIP.EQ(<source IP>) or CONNECTION.DSTIP.EQ(<destination IP>) to limit the amount of data captured. -size=<size> will limit the amount of data captured per packet. If you want to debug HTTP problems you would very likely set the size to 0, as this would capture all of the packet (0 sometimes means unlimited). -time=<time> will automatically stop the trace after <time> seconds.

After doing your trace you may execute a stop nstrace command to stop your trace. This is not needed if you set the time parameter, but I prefer to stop traces instead of setting a time parameter.

In the end you need to download the trace file. I usually use winscp as a secure FTP client, download it from /var/partitions/<partitionname>/nstrace and view content in your favourite network monitor. I prefer to use WireShark, as it fully supports NetScaler. Citrix support also uses WireShark.

Additional parameters for tracing

-tcpdump ENABLED switches to TCPdump format. TCPdump is a standard UNIX® format for network tracing. Different to NStrace it does not contain L1 information (ports), but it is understood with most network tracing utilities. You may want to use it together with -perNIC ENABLED if you want to debug routing problems. This will create a separate trace file pre NIC. You then have to scroll down both instances of your network monitor in parallel (and synchronize these 2 windows if you scroll down). However you may prefer to download free WireShark and use it instead as it understands NStrace: one window, all L1 information is contained in your trace.

–link Also trace filtered connection’s peer traffic. Only makes sense in combination with -filter. It will trace all traffic filtered plus all traffic resulting from your filtered traffic, so traffic from client to VIP and traffic from SNIP to your back end server. This is a very good one!

-mode SSLplain will decrypt all SSL traffic. Because of this you won’t see any SSL hand shake, instead, all SSL traffic will appear to be plain text. This may be beneficial if you want to debug encrypted traffic. Caution: this may expose sensitive data to you (the admin)

There are several more parameters. You may find them in Citrix NetScaler product documentation.

One of the most annoying issues in Citrix NetScaler are ICA / HDX connection issues. The reason for this is the way connection issues are reported.

There are two potential sources of trouble: Citrix StoreFront and Citrix NetScaer Gateway. So I will divide my blog in three sections: How to find the source of trouble, Trouble shooting Citrix StoreFront and Trouble shooting Citrix NetScaler Gateway.

How to find the source of trouble

It seems to be annoying and hardly possible. I am one of the moderators of a Facebook group about Citrix. Questions about connection issues come up quite often. Most of the answers don’t focus to the right source. They hardly ever ask: Which component is guilty. Instead people give misleading tips. I want to keep away from misleading tips, instead guide you through a well structured trouble shooting guide.

Let’s try to understand what’s going on:

The stages of a Citrix NetScaler Gateway connection

I talk about using Citrix StoreFront website, there is not so much difference to a receiver for web site. If you still use Citrix WebInterface: not much difference there, but my screen shots won’t be of any help.

1. a user connects to the NetScaler Gateway website and is prompted with a logon page
2. the user enters his credentials. This credentials are checked against logon providers like LDAP and RADIUS based sources (Active Directory, RSA, Safe Word, SMS Token and many more).
3. The user will see applications only after logging on successfully. So logon is done and without any issue as soon as we see applications!
   We now know: NetScaler Gateway was able to authenticate the user, it also connected to Citrix StoreFront (or Web Interface) successfully and StoreFront was successful connecting to XML broker service.
   So no need to check here, it’s already checked: Logon works perfectly fine, the connection to StoreFront / Web Interface worked fine, and it’s connection to XML broker service is tested (we would not see any application if any of them fails)
4. The user clicks an application. This click is proxied via NetScaler Gateway and StoreFront (WI) to XML broker service. XML broker service selects a resource, a desktop or an application, connects to this resource’s IP vis HTTP(s) (XenDesktop) or IMA (XenApp up to version 6.5), and stores this user’s credentials inside this machine. The machine returns a so called NFuse ticket (NFuse is the old name of Citrix Web Interface). The IP address together with this NFuse ticket is returned to StoreFront (Web Interface).
5. Getting an STA ticket: This is a first source of problem I want to go into: We have to store the target’s IP address inside our secure environment. The store we use is called STA, and it’s usually one of the XenApp servers or XenDesktop DDCs (desktop delivery controler). The STA returns a so called STA ticket.
6. We now create an ICA file. The ICA file will contain the name of the NetScaler Gateway (FQDN), the NFuse ticket and the STA ticket (don’t mix these up!) together with some information about screen resolution, clip board mapping and so on. I attached a sample ICA file:

   [ApplicationServers]
   Notepad=
   ...
   [Notepad]
   Address=;40;STA324731891;832A84599E0B7449B8578DCB8DBA95  this is STA ID and STA ticket
   AutologonAllowed=ON
   BrowserProtocol=HTTPonTCP
   CGPSecurityTicket=On
   ClearPassword=E16458A6937769  This is the 1st half of the NFuse ticket
   ...
   Domain=\C48CC641E8301B33  This is the 2nd half of the NFuse ticket
   ...
   InitialProgram=#Notepad
   ...
   Launcher=WI
   ...
   LogonTicket=E16458A6937769C48CC641E8301B33  this is the NFuse ticket
   LogonTicketType=CTXS1
   ...
   SSLProxyHost=gateway.norz.at:443  The FQDN of the NetScaler Gateway used by Receiver
   ...
   TransportDriver=TCP/IP

7. This ICA file is returned to the client via NetScaler Gateway. We don’t need to consider this connection to be guilty for our problems as it already tested: it worked fine before!
8. The browser forwards this ICA-File to the Citrix receiver. (Begin of second part!) Citrix receiver will read the ICA file and
9. connect to the NetScaler Gateway. We can see this as we will see a progress bar.
10. The receiver will send the STA ticket to the NetScaler Gateway. NetScaler Gateway will connect to the STA and try to resolve this ticket.
11. As soon an NetScaler Gateway was able to resolve the ticket, NetScaler Gateway will try to connect to the target device (XenApp server, VDI devices)
12. the application / desktop launches.

It’s essential to understand the connection process you want to trouble shoot!

So, where does it break into parts?

I have already mentioned: as soon as the ICA file is created and returned to the client the second part starts. How can we find out? Easy like that: The Citrix receiver (former names: ICA- client, ICA plugin, Citrix client, and approx 1.742.946 names more) is started, we successfully passed the first stage. So this is my first question: did it download the ICA file?

You are not sure if you received an ICA file or not?

• Firefox: The ICA file goes into your download area, typically %username%\AppData\Local\Temp (or %tmp%). However it usually gets deleted immediately.
• Internetexplorer: There is a file created in %tmp%, but it is not accessible, it’s extension is not .ICA. However it usually gets deleted immediately.
• Chrome: It’s the same: the file goes to %TEMP%. Thanks, Hendrik Klinge for this information! It is unchanged, so more or less the same as Firefox.

As the ICA file usually gets deleted immediately you may use Microsoft’s Process Monitor to be 100% sure! You could also edit the ICA file in StoreFront (C:\inetpub\wwwroot\Citrix\Store\App_Data\default.ica). It is a windows INI file. You may change RemoveICAFile=yes to no in [WFClient] section, so it will stay for ever (and spam the %tmp% directory).

More methods to find the stage of the connection process

Usually you will see an error message. It’s stage 1 (StoreFront alone to blame for your issue) if this error message is displayed inside your browser, it’s stage 2 if it’s a windows (Mac, Linux, …) message box.

Trouble shooting Citrix StoreFront

If you got stuck within the first portion of the connection process, your issue is not directly related to NetScaler, you don’t even need to log on to NetScaler!

1. Log on to your StoreFront server and check NetScaler Gateway settings:
   • Your authentication methods have to contain Pass-Through from NetScaler Gateway (right hand side, lower section, Manage Authentication Methods)

     StoreFront: Pass-Through from NetScaler Gateway

   • You need to define a NetScaler Gateway (right hand side, upper section, Manage NetScaler Gateways)

     Storefront: set a NetScaler Gateway

     Storefront: set a NetScaler Gateway, Detail

     Don’t check authentication settings: Authentication worked fine, so there is nothing to do in here!

   • Also check the STAs. The STAs have to be resolvable! (same dialogue as above)

     STA settings in Citrix StoreFront

     Use telnet (or putty) to connect to the desired port. So in my example I would do a telnet XD7-DC.norz.local 80. The screen will turn black if it is able to connect. If I enter “something” I will see some HTML output. I won’t see anything if I connect to an https based server: telnet XD7-DC.norz.local 443 as I won’t be able to do a SSL hand shake. If you mistyped the name of the STA, or the STA is not reachable you will see:
     telnet XD7-DC.norz.lokal 443
     connecting to XD7-DC.norz.lokal….
     The connection attempt will time out. Always do this tests from your StoreFront servers!
     Reasons for a STA not being reachable may be a miss-typed STA name or the (application) firewall blocking connections.

   • Enable remote access! (right hand side, lower section, Configure Remote Access Settings).

     Enable remote access in StoreFront

2. There should not be the need to mention, as this is very basic windows administration strategy, however I see tons of people not being aware of it: Check the event log of your StoreFront servers!

Events and their meanings

If something goes wrong in StoreFront you usually see this message:

There is something wrong in StoreFront

you will know: We never downloaded an ICA file. We are in trouble with StoreFront. Never check Citrix NetScaler Gateway, it was not involved, check events in StoreFront server. It may be hard to locate an event if you load balance your StoreFront servers, so I tend to disable all services but one.

Events pointing to STA problems:

The events can be found, both in administrative events, or in “Application and Service Logs” -> “Citrix Delivery Services”

There will be a set of events: Citrix Store Service, Error 0, Citrix Store Service, Error 1003, Citrix Store Service, Warning 28.

Events related to STA problems

Citrix Store Service Error 0

Store Service Error 0: The server name <your server’s name> cannot be resolved. The specified Secure Ticket Authority could not be contacted and has been temporarily removed from the list of active services.

I think, the meaning of this event is more than clear: Citrix StoreFront could not connect to at least one of the STA servers you specified. There might be a chance to connect if there is more than a single STA server. Anyway: You should fix this problem!

Citrix Store Service, Error 1003 event

Citrix Store Service, Error 1003. All the configured Secure Ticket Authorities failed to respond to this XML transaction: http://<yor server mane>/scripts/ctxsta.dll.

This event will always follow one or more Citrix Store Service, Error 0 events. This is a serious event, it means: It’s absolutely impossible to launch an application or desktop: There is no STA server available. Citrix Store Service, Error 1003 has to be fixed, it’s the reason for your connection problems! No way: You have to fix this problem!

Citrix Store Service, Warning 28

Citrix Store Service, Warning 28: Failed to launch the resource ‘Local.<your application/desktop name>’, unable to obtain a ticket from the configured Secure Ticket Authorities.

This is the final result. We could not launch the application. It’s just a summary, fix Citrix Store Service Error 0 above and you’ll get rid of the 1003 and this one at the same time!

Trouble shooting Citrix NetScaler Gateway

Our problem is related to NetScaler Gateway if we successfully mastered part 1. So let’s trouble shoot problems here.

Unable to launch an application. Contact your helpdesk with following information: Cannot connect to the Citrix XenApp server. …

Before we see an error like this we will see the progress bar indicating: Citrix Receiver received a STA file. This progress bar is of some interest! Unfortunately this message may disappear way too fast, so you will probably just see the message above.

Citrix receiver launching an application

That’s an absolutely thrilling information for all of you! Click on “more information” and you’ll see where we got stuck!

Citrix receiver connecting to NetScaler Gateway

So this picture shows the receiver establishing a connection to Citrix NetScaler Gateway. To be 100% clear: we still are not connected! We are just establishing a connection to NetScaler Gateway, so a TCP Sync packet is sent, but the TCP/IP connection is either still not established, or the SSL connection is not established yet!

Reasons for connections failing during this stage:

There may be several reasons for connections failing during this stage:

1. the name of the gateway can’t get resolved. Check the name in StoreFront.
   StoreFront: set a NetScaler Gateway

   Storefront: set a NetScaler Gateway, Detail

2. The Citrix NetScaler Gateway server certificate is not trusted, or the certificate chain is broken. So as the first step: download NetScaler Gateway’s certificate and open it at your workstation (not in a browser, just from OS). Resolve all problems with this certificate. Don’t even think of continuing without solving this problems, it doesn’t make any sense at all.
3. If you miss the intermediate CA certificate you have to download it and import it into NetScaler and link it.
   NetScaler 11.1: Go to Traffic Management → SSL → CA Certificates. Import the certificate. Next go to Traffic Management → SSL → Server Certificates. Click the NetScaler Gateway server certificate. Than Action and Link. It should display the certificate of the intermediate CA. Click OK.

STA Tickets

So we successfully connected to Citrix NetScaler Gateway. Connection in progress disappeared. The current state is connected: There is a SSL connection from Client to NetScaler Gateway.

During the next stage the Citrix receiver will send the STA ticket to NetScaler Gateway, and it will try to resolve the STA ticket. To do so it has to connect the configured STA.

STAs don’t replicate (actually they don’t even know about each other), so we need to specify exactly the same STA to NetScaler Gateway as we did in StoreFront. We will have to check StoreFront for STAs (see here). We then will check Citrix NetScaler Gateway for STA settings.

We navigate to NetScaler Gateway → Global Settings:

Seting STAs in Citrix NetScaler Gateway

NetScaler Gateway: all STAs are down, so no connection possible

As you see: the bound STA appears to be down. There are 3 reasons for this:

1. the name is wrong, or can’t get resolved. I would put the name into clipboard and then navigate to System → Diagnostics and start the ping utility. Paste the host name into clipboard and see if it is ping-able. You will see, at least, if the host name is resolvable
2. the host name is not resolvable. So the DNS server you configured for your NetScaler gateway is unable to resolve the host name. In both cases the result of this ping will look like that:

   NetScaler Gateway: STA host name not resolvable

3. a firewall is blocking the STA communication.

After resolving all of these issues the STA settings in NetScaler Gateway should look like this:

All STA servers are up now

You will notice the STA IDs, indicating NetScaler Gateway could connect to this STA at least once, and the green light (it may be missing with some elder versions of NetScaler) indicates actual connections.

No more problems about NetScaler Gateway and StoreFront as soon as you are fine until here!

It takes too much time to establish connections from outside, compared to inside?

Don’t blame NetScaler for this:

So NetScaler knows where to connect. NetScaler will use TCP/2598 for this connection: CGP (Citrix Gateway Protocol, former name: Common Gateway Protocol). At least as long as you did not turn off session reliability. I bet my life, you did not. NetScaler Gateway will try to connect via CGP for 30 seconds, than give up and try plain HDX (formerly known as ICA) on TCP/1494. So open up TCP/2598 on your firewall, it will safe you 30 valuable seconds!

Your connections still fail?

Let’s keep thinking: we successfully connected to NetScaler Gateway. We successfully resolved the ticket, so NetScaler Gateway now connects to the target device: a Citrix XenApp server or a Citrix XenDesktop VDI device.

So there are 2 reasons for this issue:

1. a firewall blocks the connection
2. NetScaler Gateway does not know a route to this IP

Just resolve these issues by opening up the firewall ports, or add the route to the desired network.

I hope this helped! Feel free to ask if you see additional problems not covered in here, I’ll answer your question and add the solution here.

Unfortunately I was unable to capture screen shots from Citrix Receiver connection stages due to my (relatively) fast environment. I’d be glad to get your screen shots

I’m just setting up a Web Application Firewall on a Citrix NetScaler 11.1 for a costumer’s shop. My costumer mandated: most of the website has to be available via HTTP. However we don’t want to expose sensitive information to the internet, so we had to create a policy redirecting users to SSL when ever needed.

So how can we do this? First of all I had to find out: which information is sensitive? It takes some time, you have to explore the application (basically a web shop).

Next: how to redirect to SSL, and thereby preserve the URL?

The policy expression

The policy has to be flexible. I want to be able to redirect users from where ever to the same URL via https. So my policy has to be a responder policy. Subtype: redirect.

add responder action res_act_send2ssl redirect 
   "\"https://\"+HTTP.REQ.HOSTNAME + HTTP.REQ.URL.HTTP_URL_SAFE" 
   -responseStatusCode 302

Rather trivial.

the GUI version

What does it do? It redirects to "https://", appends the server name specified (HTTP.REQ.HOSTNAME). If there is just one host name you could rather do something like "https://hostname.exampe.com". Next it appends the URL (HTTP.REQ.URL). The parameter HTTP_URL_SAFE is not needed. It converts all URLs into save URLs, so for examlpe /logon Page is converted into /logon%20Page. It’s good practice to do so.

Last not least: the HTML status code: I’m usually very exact about html status codes: 302 is a temporary redirect, while 301 is a permanent redirect. There is no difference from perspective of a browser: the browser will redirect. However crawlers, like Google™, will make a big difference: it will follow a 301 (and delete the previous content from it’s index), it might not follow a 302 (as it is only a temporary redirect and will be gone soon). In this case we didn’t want crawlers to follow, so we stayed with 302, the Citrix NetScaler default.

The policy

My policy had to be flexible. One of the requirements: We need to add URLs to it, when we come across a new URL. So I came up with the idea to use a Pattern Set.

Pattern Set

I prefer string maps over expression1 || expression2 || ... || expressionn constructions for several reasons: It’s more effective from perspective of NetScaler RAM and CPU (so it causes less load to NetScalers), and even more important, it’s easier to understand, and it’s easier to maintain. Just add an other URL to the Pattern Set.

add policy patset WebApp_secureURLs
bind policy patset WebApp_secureURLs "/login" -index 1
...

the Pattern Set

so this will create a Patern Set, containing URLs. This URLs will be used in the policy expression. You’ll simply have to add URLs if you needSSL for additional URLs

The Policy expression, using a pattern set

add responder policy res_pol_send2ssl 
   "HTTP.REQ.URL.CONTAINS_ANY(\"WebApp_secureURLs\")" res_act_send2SSL

The responder policy

This will create a policy expression using a pattern set. HTTP.REQ.URL us the URL (not containing the host name), CONTAINS_ANY means, the URL has to contain any of the objects inside my pattern set. We could also use EQUALS_ANY or ENDSWITH_ANY. It depends on, weather you are able to specify the URL exact enough or not (we can’t). (\"WebApp_secureURLs\") specifies the pattern set to be used. The double quotes " have to be masked with \" in command line as policy expressions have to be set in double quotes (" ... ").

This is multi langage: a mix of German and English. Look for the flag you prefer ..

Kürzlich haben mich Kollegen gefragt, welche Bücher es im Citrix Umfeld gibt. Ich persönlich gehe lieber in einen Kurs, als dass ich Bücher lese, aber Kurse sind teurer, und nicht jeder lernt auf diese Art gleich gut.

Ich besitze naturgemäß einige XenApp und XenDesktop- Bücher, nicht alle sind gleich gut. Ich habe einige ausgesucht, die ich für Lesenswert halte.

Leider sind Bücher zumeist bereits überholt, wenn sie erscheinen. Citrix veröffentlicht vierteljährlich neue Produktversionen. Wenn ein Buch abgeschlossen ist dauert es in der Regel 6 bis 12 Monate, bis es gekauft werden kann. Daher werden Bücher jeweils für die LTSR (Long Time Service Reliese) Versionen geschrieben.

I had recently been asked by colleges about reading matter. I personally prefer classes over reading, but people are different. I own several XenApp/XenDesktop, NetScaler and XenMobile books. I don’t like all of them, so I picked up some I liked and show them here.

It’s a matter of fact: Books tend to be outdated the day they appear on the market. There is steady development, Citrix brings new versions of it’s products every 3-6 month, while it takes 6-12 month to produce a book (I don’t talk about writing, I talk about production). So books tend to have reference to LTSRs (long time service releases)

Citrix NetScaler

There are some cool NetScaler books out there.

Troubleshooting NetScaler by Raghu Varma Tirumalaraju is a classic one. Raghu Varma Tirumalaraju is a Citrix employee, based in France. This book maybe not focuses on beginners, but experts will find it worth while reading. It’s the most complete one out there. The current version of this book was published on 4/16 and is based on late v.10.5 versions.

Mastering NetScaler VPX by Rick Roetenberg and Marius Sandbu, published in November 2015. Rick and Marius are Dutch Citrix Gurus. The title is a bit miss-leading, as it also perfectly suits for NetScaler MPX and SDX as all NetScaler versions share the same code base (the SPX specific part, so SDX VM management, is missing). It’s a bit lower level, but easier to read.

Implementing NetScaler VPX by Marius Sandbu, published in Oktober 2015, is a classic one. I personally would prefer Mastering NetScaler VPX. Be sure to get the second edition, first edition is elder.

XenApp / XenDesktop

Inside Citrix: The FlexCast Management Architecture Bas van Kaam printed in 5/16 is a bible about XenDesktop 7.6 (the LTSR). It’s currently the latest book on the market. Again, this is not a book for newbies, but it will be what you’re looking for if you are a XenDesktop pro!

Citrix XenDesktop 7.5 und XenApp: Das Praxishandbuch für Administratoren von Nico Lüdemann ist der Klassiker, und das Original ist in Deutsch! Vielleicht das ideale Buch für den Einsteiger, und das einzige, das ich mir auf Deutsch antun würde! Es gibt auch eine Version für XenApp 6

Unfortunately, Nico Luedemann’s book about XenDesktop 7.5is currently not avalable in English.

XenMobile

Citrix XenMobile 10: Installation, Konfiguration, Administration von Thomas Krampe wäre mein erster Tipp! Wieder ein Buch, das man jedenfalls auf Deutsch lesen sollte! Ich hatte meine XenMobile Installation binnen kürzester Zeit am Laufen! Einfach, klar, gut geschrieben!

Citrix had been cloud computing ever since. In fact, Citrix started doing professional Cloud Computing in 1995. They improved their private cloud approach in 2002 by launching Secure Gateway (and all it’s successors). They already did cloud computing when no one even knew about cloud computing. Maybe Citrix is the cloudiest company on the globe.

Citrix dramatically improved their product over time. A big step forward was FMA. Maybe some of you don’t really like FMA, but I think it’s great. But there is one drawback about FMA: it’s expensive.

With IMA you could create a single remote desktop server and add Citrix XenApp to it. That’s it. No data base needed, a single MDB file will do. Install your web interface on the same box, finished! Add a second remote desktop server, including web interface, and you have, what I’d consider to be a farm; with full redundancy. OK, management is not possible as long as the server, containing the data store, is down.

FMA is different: There is a strict separation between back-end and front-end components. Even though I do know it’s possible to install all components on one box, I would strongly advice to keep them separated.

Back-end components:

There are three mandatory components: Data base, controllers (also called desktop delivery controllers, names change frequently) and StoreFront. (Plus license servers, …)

The data base had to be redundant as it was essential to the farm. You could not launch a single application as long as the data base was down. This was mitigated when Citrix introduced connection leasing, but a stable data base still is mandatory.

Controllers are essential too: same here, you can’t launch an application as soon as all controllers are down. Same is about StoreFront.

So a fully redundant XenDesktop / XenApp 7.x deployment consists of three (!) data base servers (one of them, the witness server, may be SQL Express), two controllers and two StoreFront servers. A “somewhat stable” deployment for a small company consists of, at least, a data base server, and two boxes containing controller and StoreFront functionality, so three servers.

Front-end components:

These may be Remote Desktop Servers or virtual PCs. I call these: workers. They may run on bare metal as same as on any hypervisor you like (as long as it’s VMWare, HyperV, Xen, Amazon Web Services, Microsoft Azure and many more). I won’t go into in this. They pay themselves as they do real work.

So let’s go back to the back-end components.

They are some kind of a set it and forget it infrastructure. If you look a bit deeper into these, there is some important work for an admin: data base backup (including regular restore tests!), frequent updates (as Citrix releases a new version every 3 to 6 month) and some more.

Did you ever think: why do I need to keep all these costly components? Bet your life: I thought of getting rid of them several times. But moving all of my (my costumer’s) environment into the cloud is no option in many cases, so I have to keep them on premise.

Let’s think a bit about it: Why? That’s easy: data is mandated to be on premise, so back-end components need to be on prem as well. Right?

Wrong!

Citrix now offers Citrix Cloud services. Maybe you think: just another f***ing cloud provider? No. That’s definitely wrong. Citrix does not run any cloud services at all, instead they host on Microsoft Azure. And Citrix does not want host your workers, they never will. Instead they offer all necessary back-end components. Fully redundant, perfectly sized, updated continuously, including Citrix licenses.

I have been on a sales workshop by Fabian Kinle yesterday. Fabian is a hero of German, of European Citrix sales. Yesterday he wanted to sell Citrix Cloud Services. And he asked: Would you sell Citrix Cloud Services to your costumers. “I would not do” I replied. Fabian didn’t like this on first sight. “But I would guide my costumers all the way through a XenApp / XenDesktop project. In the end, we would decide where to host all these back-end components. Citrix cloud services is, indeed, a very sexy option: I can get rid of all these back-end components and focus to the front-end. From perspective of a consultant this is sexy, as back-end components just cause trouble, but don’t bring much revenue. From perspective of the costumer, it’s pretty much the same.” Fabian started smiling again.

Yes, I would think of that. I would think of that for a rather small environment (since licensing now starts from 25 users), as I get full redundancy for rather little cost. I would consider using it for huge costumers as well, as Citrix will be responsible for all the life cycle management, like scaling it up to a proper size and updates. Your back-end components will always be shiny and fine. Being the architect, I would also like it, as sizing back-end components is my peace of cake no more.

And being the CFO? Hey, I don’t have to buy hardware and licenses and then pay for a year by year subscription, instead, I will rent. It adds flexibility as I can add or remove licenses whenever needed. Don’t forget: after reducing stuff you will still have to pay subscriptions for now no more needed licenses as further growth may bring users back.

Citrix also tries to keep your workers as independent as any possible from cloud providers. They will release tools to help you moving your workers. From premise to cloud, even better, they’ll help you moving from AWS to Azurre (that’s obvious), the other way round (that may be a bit of surprise), or even from cloud to premise (even though no one nowadays may think of this). They don’t care where you host, they’ll help you as long as you use their product.

There is something they care about: Keeping you environment up to date. Maybe even more: They keep their own environment (so your hosted StoreFront, controllers) one step ahead of retail versions. This does not mean you have to update your VDAs on a daily base: there is a broad range of backward compatibility. Just update your part of the site (farm), when ever you feel like updating. Update it, when ever you see a benefit.

The rest is simple maths. Are costs nice, just fair or even insane? That’s your job to do.

I recently was hired to create a web application firewall (WAF) using Citrix NetScaler to protect a SAP Hybris based e-shop. This shop has content for several languages, so we had to select the right home page.

The base URL of the website was like that: https://shop.domain.com/shop/language/. SSL was optional. I wanted to set the default language based on browser settings. I based it on HTTP- Header  Accept-Language.

There are several pre-defined codes. I found a list here.

Following rfc3066 I found out, Accept-Language tags will always start with the main language (for example en, de, fr, …) and can optionally be followed by sub attributes. De-AT for example means: Austrian German. There may be some more sub attributes, but they had not been of any interest. A typical header would look like that: en-us, en;q=1.0,fr-ca, fr;q=0.5,pt-br, pt;q=0.5,es;q=0.5

The only portion I was interested in are the left 2 letters. en means English, de means German, fr means French and so on. Easy like that!

Even though users could change their language code they usually don’t. There are tools out there, like “change accept-language switcher” for FireFox, but they get hardly used, and if they get used they get used for reason. So this is something I can count on.

So I decided to base my policy on this header. I made up my mind to use responder, however I could also have used rewriting. Whenever a user just opens the base URL (/), I will forward him to /shop/xy where xy is the country code.

The Citrix NetScaler function I have to use is HTTP.REQ.HEADER("Accept-Language"). I am interested in a substing, starting on position 0, the first 2 digits. So I have to add the SUBSTR (x,y) function, where x is the starting position (0) and y is the length of the substring (2).

The RFC mandates lower case letters for the country code, but the term RFC means request for comment. So different from ANSI or DIN a RFC is just a proposal, so you might not count on this. So I appended a .TO_LOWER, just to make sure. So my redirect would look like that: "/shop/" + HTTP.REQ.HEADER("Accept-Language").SUBSTR(0,2).TO_LOWER + "/".

It worked great. Our UK costumers got sent to English content, the French to French, the Swiz to French, Italian or German (depending on their taste) and the Germans to German content, even costumers in Lichtenstein would be faced with German content (de-LI). It could hardly have been any better. Just Chinese hackers saw a 404, not found, as they got directed to ../zh/, which does not exist at all! Great!

My first thought was: give a damn shit on Chinese, we don’t sell outside of the EU (OK, something our British costumers will have to keep in mind). But there is a bigger problem than just Chinese: Some search engines might use crawlers not sending this header at all. So what can I do?

We currently have some languages: English (en), French (fr), German (de), Italian (it), Dutch (nl) and some more, a total of 13 languages.

So I created a list. A pattern set called suported_languages. It would be good style to give index numbers based on importance. So I gave de (German) 10, as it’s my costumer’s biggest market, French 20, and so on. Reason here is to save computing time.

add policy patset suported_languages
bind policy patset suported_languages de -index 10
bind policy patset suported_languages fr -index 20
...

My costumer defined a standard language, English, for example. I then created 2 policies, one for unsupported languages (not in list), and one for supported ones.

So my first policy expression would look like that:

HTTP.REQ.URL.EQ("/") && HTTP.REQ.HEADER("Accept-Language").SUBSTR(0,2).TO_LOWER.CONTAINS_ANY("suported_languages")

This means: the requested URL is / (this is the empty URL), and (&&) the browser preferred language is inside my list.

The other policy expression could be quite similar, but with a .NOT in the end, so not in my list, however I would have to examine all languages again. I also could bind this policy with a GOTO END (so no other policies get evaluated), so the other policy would just ask for HTTP.REQ.URL.EQ("/"). This will safe CPU time (we already know: It’s not in the list!)

A great, but simple, example for policy expressions!

This article is a very first answer to a question I recently received. The question was: You’re talking about web application firewalls. I’d like to know how to use the WAF for reverse proxying. … I also might use a firewall as it also contains functions like SQL injection prevention.

So what’s in a web application firewall?

A web application firewall, also known as WAF, is a shell around a web application. There are hundreds of threats, and there might be a vulnerability in my application none is aware of. There may be debug code, remained inside my application, there may be comments containing too much information like <!—now connecting to database, using dbuser/ksjd72h?kasjd-->, and you’ll never know. All this might help to break the application.

A WAF will try to address several threats in a very streamlined way. There are general approaches like removing comments. No comment is needed, so they all have to go (however: It’s rather common to place scripts in comments to keep them from being displayed, so we will have to leave comments containing scripts). There are pattern based approaches addressing well known attacks, and there are application descriptions allowing only traffic known to be not malicious.

So what’s the difference to a traditional fire wall?

First of all, a fire wall is a L3/L4 device. As such it’s a packet based animal. It can, of course filter malicious packets. But it can’t filter malicious traffic, if it’s thoughtfully distributed over several packets. A firewall has very limited abilities about data streams. And data streams is that we face in web attacks.

A NetScaler is – in most deployments – a proxy, so it’s a L7 device. Being a proxy it handles data streams instead of packets.

The negative approach

A WAF always supports a negative approach: Well-known malicious traffic gets blocked. Think of a HTTP request for an URL ending with /etc/password on a LINUX host. We would expose sensitive information if a request like that is successful. This negative approach is pattern based. A Citrix NetScaler contains lists with hundreds of well-known malicious requests.

And we would block “incomplete” requests, so requests not containing the usual header fields (Accept-Language, User-Agent, …). This may block DDOS tools like LOIC!

The positive approach

Unfortunately we don’t know all the risk out there. Even worse, we don’t know all weaknesses and vulnerabilities of our applications. We would – of course – fix them immediately if we just knew!

However we know how our application works: We know all URLs allowed we know about cookies and so on. We know the length of URLs, query strings and cookies. So we can describe all allowed traffic and deny all the rest. The positive approach is quite similar to what we do with a fire-wall: we deny everything and allow non malicious traffic only. At the same time we will also replace (for example) all ‘ with &apos; that’s a very efficient was to prevent SQL injection attacks from occurring.

The hard peace of work is finding out what’s going on inside our web application. We have to white list all allowed URLs (in NetScaler we call these start URLs). This may be done using literals or Pearl compatible Regular Expressions (RegEx). In addition we may even deny certain URLs (in NetScaler we call it Deny URLs). We mainly use it if we allow everything inside a certain subdirectory (let’s say: ^https?://www\.Example\.com/images) and need to deny access to ^https?://www\.Example\.com/images/secret.jpg.

An important one is called “cookie consistency”. I once found a cookie containing 2 fields: Costumer_Number and Logon. Of course I changed it to Costumer_Number=1 and Logon=true. This made me the first costumer (and owner) of the shop. I had been able to change all I wanted (also prices, see all costumer data and so on). No doubt, that’s a bad mistake inside the web app. What can we do? We could, for example, encrypt these cookies, or add a hash to them and protect them from being tampered.

Another bad thing is loss of confident information. A partner company of mine uses a blog to communicate to costumers. They also use the same blog internally, but internal readers may see content with certain keywords that’s not available from outside. This company had a top secret project, called (let’s say) Middle-Earth. This project was planned to be announced at a great conference some month ahead. Unfortunately one of the guys working on project Middle-Earth forgot to specify the proper keyword (like I use to do all the time), so a document containing project specifications about project Middle-Earth was leaked. A WAF could contain blocking rules (“Content-Type”), containing words (numbers, whatever you could specify using RegEx) preventing documents containing this words from traversing the WAF. Same can be done about credit cards (NetScaler supports American Express, Diners Club, Discover, JCB, MasterCard and Visa). So these credit card information can’t leave your webserver, even if your application fails, or an attacker was able to access your log-files. This is a must have to get certified PCI-DSS

There are several more things to do. I just gave a short description about possible threads and what a WAF could do for you.

How to protect a website using Citrix NetScaler?

Well it seems to be easy. A nonsense question. We may use AppQoE (Application level Quality of Experience), a feature introduced with NetScaler version 10, so it’s quite an old feature. Let’s start.

AppQoE is enterprise edition

My first starting point was E-Docs. Let’s be honest: the guy in charge did not really know what he had been talking about. No break through from there. Citrix Blogs! Unfortunately AppQoE is not their pet child. I found just one single article about this subject (telling me AppQoE is great). Raghu Varma Tirumalaraju did not even mention this feature in his book. Marius Sandbu did, however he did not even go as far as E-Docs.

So there seems to be no information on a feature everyone needs?

That’s a bit of surprise as DDOS attacks had been very prominent in media. Think of Twitter, Spotify and some more being down last summer?

So let’s start the other way round: I will (D)DOS a test website. I googled for tools and found LOIC. LOIC is widely used by groups like Anonymous to DDOS servers. LOIC is a quite simple tool sending simple HTTP requests to a server of choice. I loaded LOIC (don’t even think of doing so in UK, they’ll send you to prison immediately, most countries would allow you to use it against your own servers) and fired some thousand requests to my test server, but the server was still responsive. In fact, it was hardly affected. I found some thousand messages in /var/ns.log:

Mar  2 13:28:06 <local0.info> 10.255.0.250 03/02/2017:13:28:06 GMT 82e6de130138 0-PPE-0 : default APPFW AF_400_RESP 49586 0 :  172.19.54.5 559567-PPE0 - test.lab http:/// Bad request headers.No host header <blocked>

Yes! NetScaler blocked all LOIC’s requests, they didn’t pass through. It had been my WAF (Citrix NetScaler Web Application Firewall) protecting my web server.

Damn good news, thanks, WAF!

Protecting from LOIC is an easy one, you could also protect your web server using Citrix NetScaler responder policies on standard edition. Simple do a drop if HTTP.REQ.HEADER("Host").EXISTS.NOT. There may be some other headers missing like User-Agent, Accept, Accept-Language, Accept-Encoding. So you might end with an expression like HTTP.REQ.HEADER("Host").EXISTS.NOT || HTTP.REQ.HEADER("User-Agent").EXISTS.NOT || HTTP.REQ.HEADER("Accept-Language").EXISTS.NOT. In the end, you would check if requests coming is are in compliance with RFC 2616.

Just a, maybe stupid, question: Why do they use a simple tool like that? It’s easy to use, free, can get triggered via IRC. It’s not that high sophisticated, but they don’t care, they just hack some more machines (or IOT devices). It does not need the web server to crash, it’s enough to overload the ISP. Mastercard, Visa, PayPal  and some more might have survived the operation Avenge Assange attack in 2010, but who cares if their internet connections suffers under endless congestion queues causing all packets to be dropped? So you see: Even a great thing like a NetScaler would not have been of any help.

Anyway LOIC was not my tool. I changed to HULK. HULK is just some smart lines of Python 2.7 code. The requests look pretty much like legitimate requests, so they pass my WAF easily and kill my server. They can do even more, you have to specify a target URL. HULC ist launched like that: c:\hulk.py http://test.training.lab/search/showallitems.php. It’s quite easy to change the code, if you need to do so. HULC also adds random parameters, and uses random headers, so successive requests don’t look the same. It would be hard for a NetScaler to defend a website from a HULK attack by simply using responder policies, especially if someone would randomize the URL in use. This would be just some more lines of code in HULC, not a big deal, even for a non experienced Python programmer like me. (My approach would be a responder policy using the Citrix NetScaler Rate Limiting feature to find the most frequently used Source IP or URLs and block them).

So let’s go into Citrix NetScaler AppQoE.

Citrix NetScaler AppQoE in a nut shell

AppQoE means Application level Quality Of Experience. It’s well hidden as you’d suppose it to be in Security, however it’s in App Expert.

In AppQoE Citrix blended some features like priority queuing. Queues are an important part of AppQoE. It simply uses two queues: one for not known legitimate requests, one for well known legitimate requests. Policies are used to assign users to certain queues. There are four levels:

• High
• Medium
• Low
• Lowest

During regular work queues are not used, all requests get treated equally. If the NetScaler considerers itself to be under attack it starts using priority queues. So we have to tell NetScaler how much traffic we can (or are willing to) handle. We turn on AppQoE if traffic exceeds these limits. Maybe its the most difficult thing! Be careful to not cut down your Christmas business!

If we are under attack we have to find out, if the request is legitimate, or not. It’s a bit tricky, as some of these bad guys out there are quite smart. We have two methods: one is transparent to the user (a Java script is run on client side), however there is a chance for the attacker, the other one presents a captcha to the user, so the user has to prove he got a brain. This is a huge obstacle to overcome. The method is defined inside of policy actions. If the user is a legitimate one we start a session for this user and assign it to a privileged queue.

If you click on AppQoE feature on the left plane you see Configure AppQoE Parameters.

AppQoE general Parameters

What do these parameters mean?

• Session Life (secs) We will remember this user as a legitimate user for XXX seconds. It defaults to 300s, 5 minutes. Don’t set the session life parameter too low. Setting it lower than a minute may result in endless loops of testing weather this client is legitimate or not, as the session timed out during testing, so the results are no more valid. I tend to not change it. Never go below 60. On the other hand, an attacker would just neet to solve the captcha once per bot id the value is too high.
• Average Waiting Client queue would not grow beyond a very low number in normal use: Your web servers should be fast enough to handle all legitimate traffic. That’s why I tend to lower this value. Sizing depends very much on web site useage. How many client requests may wait until we consider this website to be under attack? It defaults to 1000000.
• Alternate Response Bandwidth Limit(Mbps) how much bandwidth may all responses waste (default 100 MBt.). We send alternate responses as soon as we exceed this value.
• DOS Attack Threshold The everage number of clients that can queue up without triggering DOS protection. It defaults to 2000. If your website won’t need to support more than 50 you would add 50 (don’t forget to add some spare users for Christmas business!). Don’t set it too tight, exceeding this value may influence user experience.

AppQoE policies

Like any policy it got an expression and an action. The expression may either be a simple true value, or, if you need to be a bit more granular, be based on type of content or URL. Be sure to not exclude content on your server. This would allow a successful attack!

The Action

Like any action this one needs to get a name. I usually call them AppQoE_act_…

There are three types of actions, and you can’t change the type after creating the policy, so spend a moment to think of it:

• None means no action is taken when a threshold is reached.
• ACS means there is an alternate content source (an other LB vServer), so NetScaler won’t handle that traffic. This is what my screen-shot shows.
• NS means, traffic is handled by NetScaler. I will go with this one.

We may use an other TCP profile than usual in case of an attack. It can be used to use advanced TCP features to mitigate damages.

The (high, medium, low, lowest) may get used to split up traffic and handle more important traffic prior to unimportant one. So, for example, we could create a policy handling movies (HTTP.REQ.URL.ENDSWITH("AVI")) and put them into low priority, as wa are under attack and suffer under a lack of bandwidth, and more important data into medium or high. Keep aray from lowest as lowest is also used for malicious requests.

When the policy queue size, that means the number of requests queued for this policy, reaches Policy Queue Depth threshold value, subsequent requests are dropped to the lowest priority level. That means, they don’t respect your priority settings any more. Different to Queue Depth this one is about a policy, not about a vServer as a hole.

The Queue Depth threshold value is a per priorirty level. If the queue size (number of requests in the queue of that particular priorirty) on the virtual server increases to the specified queue depth value, subsequent requests are dropped to the lowest priority level. That means, they don’t respect your priority settings any more. Different to Policy Queue Depth this one is about a vServer, not about this specific policy.

If the Maximum Connections number is exceeded we place even legitimate requests into lowest priority queue. Either this one or Delay has to be specified.

If the Delay (microseconds) is exceeded we place even legitimate requests into lowest priority queue. Either this one or maximum connections has to be specified.

The Alternate Content Server Name* is the name of a vServer if you use ACS. Requests will get sent to this server instead of using the original one. It’s a kind of sorry server. The Alternate Content Path* is the path on this server, for example /div/WeAreSoSorryWeGotHacked.html

The Custom File is is a file displayed to users considered to be illegitimate. It’s an optional parameter. You don’t specify it. Use the menu item under AppQoE on the left side to import a file.

The DOS Action can be a SimpleResponse, or a HICResponse:

• SimpleResponse is a response containing a Java script. The client needs to have Java scripts enabled. This script dies the maths and returns the result to the NetScaler. The NetScaler will treat this user as a legitimate one if the session is not already timed out. Advantage here: The user won’t see what’s going on. Disadvantage: A bot may also be able to run the script.
• HICResponse is a response containing a rather simple captcha. The user has to solve the problem. It’s quite difficult to trick this one.
  There is a prerequisite for this. You need to open up a putty session, then:

  > shell
  Copyright (c) 1992-2013 The FreeBSD Project.
  Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
          The Regents of the University of California. All rights reserved.
  
  root@82e6def331a4# nsapimgr -ys appqos_captcha=1
  Changing cfg_appqoe_captcha_enable from 0 to 1 ...
   Done.
  root@82e6def331a4#  echo "nsapimgr -ys appqos_captcha=1" >> /flash/nsconfig/rc.netscaler
  root@82e6def331a4#
  > _

  This will make Citrix NetScaler able to send the captcha to the client box.
  

  ---

  Logging

I did not read anything about logging. That’s a pity. In fact I could not find out weather my NetScaler supposes itself to be under attack or not. Wait, there is one thing:

stat appqoe policy <policyname>
Output may look like that:

> stat appqoe policy AppQoE_pol_2delete
AppQoE Cumulative Policy Statistics Summary

Cumulative Policy Stats:
Counter                                    Rate(/s)                Total
Server TCP connections                             0                    0
Client TCP connections                             0                    0
Requests received                                  0                    0
Requests bytes                                     0                    0
Responses received                                 0                    0
Response bytes                                     0                    0
ThroughPut(Kbps)                                   0                    0
Alternate responses sent                           0                    0
Alternate responses bytes sent                     0                    0


Timing Stats:
Counter                                    Rate(/s)                Total
Average Client TTLB                                0                    0
Average Server TTLB                                0                    0
Average Server TTFB                                0                    0

 Done
>

> stat appqoe policy AppQoE_pol_2delete
AppQoE Cumulative Policy Statistics Summary

Cumulative Policy Stats:
Counter                                    Rate(/s)                Total
Server TCP connections                             4                 2051
Client TCP connections                             4                 2154
Requests received                                  4                 2051
Requests bytes                                  1251               733826
Responses received                                 4                 1916
Response bytes                                  9489              5048057
ThroughPut(Kbps)                                  84                  587
Alternate responses sent                         113               576640
Alternate responses bytes sent                414706           1610897508


Timing Stats:
Counter                                    Rate(/s)                Total
Average Client TTLB                                0                    0
Average Server TTLB                                0                    0
Average Server TTFB                                0                    0

 Done
>

Means: under Attack. Same policy, same amount of traffic, but different parameters.

Unfortunately we are not yet finished, continue reading with part 2

I hope this helps by a little bit. And I hope Citrix will do a better job in future about explaining this – rather important – feature!

Yesterday I published a blog about DDOS- protection. I used the Citrix NetScaler AppQoE feature to do so. That’s nice, but not enough. I still could beat my server to a pulp easily. Just 10 clients launching a DDOS attack using HULK had been enough. I can’t throttle down the number of users to just 10!

WTF?

Basically AppQoE will just limit the number of users (actually it’s the number of source IPs) being able to connect, not the number of requests. No matter how deep you set the limit, even a single user may screw up a web server, given his connection is fast enough. No protection from NetScaler AppQoE here!

What to do next?

What’s wrong? We limit the number of users. That’s a damn good story as it is possible to do serious attacks with distributed users. But we did not limit the number of requests per user! How can we limit the number of requests per user? Or, the number of requests to a certain URL?

Think you are the boss of a police station. One nice and sunny morning you would tell one of your officers to check cars (that’s the Selector, he has to select cars). Unfortunately a single officer would not be able to check all cars, so you tell him to stop every 10th car. This would be the Idenitfyer. Both together give a good job description: “check every 10th car!”. But it takes an officer to do the job. That’s what our (responder) policy would do.

So we have to set up a stream selector, a stream identifier and a responder policy.

Rate Limiting is the feature or choice!

The very best thing: Rate limiting is standard edition, so everyone may use it. How could Citrix NetScaler Rate Limiting look like to protect from a DOS attack?

The NetScaler Rate Limiting Stream Selector

The first ting to do: select the kind of traffic we want to limit. There are two kinds of approach: the client IP and the URL; or a combination of both. There are built in stream selectors we could use: Top_URL (HTTP.REQ.URL) and Top_CLIENTS (CLIENT.IP.SRC). I did a trace of an attack done by HULK. HULK always appends some random strings like ?KCDPT=SRKGT. So Top_URL is not useful, instead we would need to create our own stream selector based on HTTP.REQ.URL.PATH to strip the query string.

I created a stream selector using both, URL and IP. This is a list of URLs per IP.

add stream selector rate_sel_URL_IP CLIENT.IP.SRC HTTP.REQ.URL.PATH

The stream selector “does not do anything” but selects streams. So our next step would be:

Create a NetScaler Rate Limiting Stream Idenitfyer

The stream identifier checks weather or not to pick a stream. There are several parameters:

• The Selector is the stream selector we created before.
• We have two Modes: Request Rate and Connection. Request Rate tracks requests (that’s what we need), Conection counts connections. That’s not useful in here, we want to block requests, not limit connections!
• The Limit Type may be bursty or smooth. Smooth wants requests to come in evenly, while bursty would allow much higher rates as long as the total number does not exceed the limit for the given time slice (so if you define a limit of 10 and a time slice of 10s there may be 10 requests within a second, but no more during the next nine seconds if you go with bursty and 1 per second for smooth). I go with bursty as this is how requests usually come in.
• The Threshold is what you have been looking for: The number of requests (or connections). Select them wisely! If you permit too many you may kill your server, if you don’t allow enough you will block legitimate requests of your users.
  
• The Time Slice (msec) is the amount of time we use to count our requests, in milliseconds. So 5,000 would be 5 seconds, 60,000 a minute.
• Maximum Bandwidth (Kbps) is an additional limit you may specify for connections. We may leave it blank as we are not interested in connections.
• Traps is the number of traps to be sent in a time slice. 0 means no traps, I guess 1 would be fine, even if your time slice is several minutes.

Usually a single user won’t request a certain URL more often than 10 to 100 times per 5 seconds (be aware of spacer GIFs, some web site programmers use them excessively! I have seen hundreds of requests within a second. And do you know what I think of the web site programmer? What a bloody idiot! But he’s just a web site programmer, no engineer, maybe that’s why).

add ns limitIdentifier rate_id_URL_IP -threshold 10 -timeSlice 5000 -selectorName rate_sel_URL_IP -maxBandwidth 0 -trapsInTimeSlice

The responder policy

So we could find out to malicious requests. But in the end we will need to react. Our stream selector, our stream identifier don’t respond to client requests on their own. That’s why we need to create a responder policy.

The responder action

What to do? There is no best practice. I tend to drop all malicious requests, but you also could answer with a 404, not found, a 500, server error, redirect users to a religious site of your choice (to force him to think about his sins), or with a custom error page. It’s up to you. I will simply drop.

The responder policy

add responder policy res_pol_drop_DOS "SYS.CHECK_LIMIT(\"rate_id_URL_IP\")" DROP

So you can see, my action is a DROP. I tried to do some logging (but it didn’t work). The point of interest for you will be the expression: SYS.CHECK_LIMIT(“rate_id_URL_IP”)

I hope you like it. Leave a comment if you find something you don’t understand (or don’t agree). I also accept comments telling me I did a good job

Why do I love HDX on UDP in Citrix XenDesktop and XenApp? (HDX Enlightened Data Transport EDT)

Well, I’m mainly a network guy. So I’ll take a look at this brand new feature from networking perspective.I’ll start from scratch, so I don’t assume you understand network protocols.

But let me tell you a joke about UDP first:

I got a short joke about UDP, and I don’t care if you got it!

You may, or may not get my joke. Of course I care, so I’ll explain: TCP has guaranteed delivery. So every packet you send will be delivered. UDP doesn’t care if you got the packet, or not.

It also doesn’t care if packets arrive in the right order.

The situation with TCP

TCP uses so called sessions.

It takes 3 packets to establish a session:

client → server: Syn + sequencenumber (Syn means: let’s talk, the sequencenumber is a random number)
192.168.0.1 → 192.168.0.13: SYN, seq. 1000

server → client: Sync/ack + sequence number + acknowledge number (Ok, let’s talk. client’s sequence number is server’s acknowledge number, server’s sequence number is a random number too)
192.168.0.13 → 192.168.0.1 SYN/ACK, seq. 2000, ack. 1000

client → server: Ack  + sequence number + acknowledge number (Ok, I’ve heared, you agree about talking. The client increments his sequence number by 1 and uses server’s sequence number as an acknowledgement number)
192.168.0.1 → 192.168.0.13: ACK, seq. 1001, ack. 2000

This is a bit of overhead, but it’s overhead during session establishment only, so we don’t really care about it.

Sequence/acknowledge numbers are very important. If a client sends a packet (let’s say number 1001) it has to get an acknowledgement for this package (1001).

sending packets

My explanation is based on RFC 793.

This is a very basic principle all over TCP. It’s really great. Let’s say, a server sent packets 20001, 20002, 20003 to the client, however the cient only received packet 20003 and 20001 (so you see, wrong order here) the client will wait for a pre defined amount of time to get 20002. If it does not appear, the client will send an acknowledge for 20001 only. So the server will resend 20002 and 20003. This procedure is called retransmission. Simple like that. (Notice: we retransmitted 20003, even though it already arrived on client side.

To make file transfers fast, client and server both use a so called sliding window, a buffer to send and receive.

If a sender wants to send 10 packets, it’s buffer size is 5 packets, it will put the first 5 packets into the sending buffer and starts sending packets as fast as possible, starting from first packet.

The receiver (the client in my example) has a receiving window with a size of 5 packets as well. It will put all received packets into it’s buffer. As soon as it received the 3rd packet it will send an acknowledge for the 3rd package to the sender (confirming the 3rd package implies also package 1 and 2). The sender now deletes the confirmed 3 packets (the ones I coloured in pink) from it’s sending buffer, shifts all packets left by 3 positions, and add 3 more packets into the – now free – last 3 positions of the sending buffer (the pail pink ones), so they can get transmitted.

The client will receive the next 3 packets, and confirm as soon as the 3rd packet is received.

This mechanism guarantees smooth data transmission, the bigger the buffers are, the more latency we can tolerate without the need of an interruption of the sending process. TCP, therefore, is not very sensitive of latency. In internet we usually use big sending and receiving windows (buffers), so downloads stream smoothly.

The only delay here: If there are less than 3 packets left: The client will wait for more packets for a specified amount of time before confirming the already received packets. This delay does not matter if you think of a download containing of hundreds of packets.

Where is the problem about HDX/ICA?

I’m currently typing this blog article into a published Mozilla Firefox on XenApp. If I type an I, for example, my client sends an I to the server. That’s just one packet to send (less than 3!!!). The server now waits for 2 more packets to come, however I’m typing quite slowly (aprox. 60 strokes per second), so there won’t appear an other packet for a very long time (if you think in processor cycles). Server’s TCP has a timer. If that timer is down to 0 my server’s TCP will send an ACK to my client and, at the same time, forward my request to the Citrix services. So my Browser application just now receives my message! This is additional latency! And latency is not what we want to see!

So there are several delays: Network latency from client to server, the server waiting for more packets, the server processing my request, sending it and network latency (again) until the client finally receives the response. Oh, I forgot: A Citrix server’s response will be (in most cases) less than 3 packets, so my client’s TCP protocol stack will also have to wait for a 3rd packet to come. Unfortunately there won’t be a 3rd packet, so the client got some delay until it sends an ACK to the server and forwards the packet to my Citrx receiver.

The more interactive your application is the worse things get.

An other TCP problem

There is an other aspect I have forgotten to mention. I have to thank Marcus Kohlschmidt, he mentioned it yesterday.

TCP guarantees delivery. 100% correct delivery. There are checksums to make sure every single packet got delivered, delivered correctly.

Now there is no point in forwarding garbled packets. That’s obvious. Every single router will therefore check checksums to avoid forwarding corrupted packets. This router will discard corrupted packets, and cause packet loss. Mail problem here is not just silent packet loss, but much more important the overhead of calculating checksums. Every single router will have to do this, causing congestion on routers. The more hops we have to pass the more overhead it causes. UDP packets don’t contain checksums, so they are forwarded immediately, and may eventually overtake a bunch of TCP packets. So there may be less latency for UDP than for TCP. Unfortunately this is not true if we tunnel UDP inside SSL.

Why can UDP handle Citrix XenApp / XenDesktop HDX (ICA) better?

UDP, you remember?, does not have sessions at all. A client will send a packet to the server. The server will immediately process it, if it receives the packet. That’s good news.

If it did not it won’t. That’s bad news. Even worse: If packets appear in wrong order they get forwarded in wrong order. So let’s say I type “if”, but for some reasons (why? Ask the network guys) the f arrives in front of the i, my Citrix server will think of fi. That’s unusable.

Citrix had to build TCP functionality into HDX (former ICA) to overcome issues like that. So UDP based HDX packets need to get numbered. HDX won’t send an acknowledge, but instead it will send the server reply containing the acknowledge. In my example my server will receive packet number 2, the f, and packet number 1, the i, so it will know they are in wrong order. Or, if it won’t receive packet number 1 at all it will tell the client: I have received a packet number 2, but number 1 is missing.

If a client sends a packet, but does not get a response from server in a timely manner, it will have to resend. So you see, it’s TCP functionality, built into HDX/ICA, so it may be sent on UDP instead.

What about NetScaler Gateway in HDX/ICA proxy mode?

Well, we use SSL here. HTTPS. SSL is a tunneling protocol, so HTTPS would be HTTP tunneled in SSL. SSL will, always, be TCP based.

It’s a great advantage to go with standard protocols. It’s easy to use proprietary protocols using fancy ports like UDP/2598 or even UDP/443 in a closed environment. The only thing you have to do is: tell your firewall staff to permit this ports. But it’s a mess in internet. How can you talk to the firewall staff of your (possibly global) costumers? They will simply refuse. How can you talk to firewall staff of “all hotels in the world“? It’s impossible!

So we still have to go with a TCP based tunnel using 443. All advantages and disadvantages of TCP are true about this tunnel. But inside this tunnel we have light weight UDP packets. The effect of latency will be cut to half. We don’t have to wait for all these timers to count down to 0. HDX/ICA will be much smoother.

Requirements to use UDP-based HDX/ICA

There is a description by Citrix. Maybe you prefer that one over mine.

In Citrix XenDesktop 7.12 I found following paragraph:

New enlightened data transport layer (for evaluation only). IT administrators can evaluate a new HDX data transport layer designed for challenging long-haul WAN and Internet connections. This alternative to TCP delivers a superior user experience while maintaining high server scalability and efficient use of bandwidth. This new transport layer above UDP improves data throughput for all ICA virtual channels including Thinwire display remoting, file transfer (Client Drive Mapping), printing, multimedia redirection and others. In 7.12, this feature is for evaluation only and is disabled by default. It can be enabled in a non-production environment with a new policy setting, HDX Enlightened Data Transport. Set the new policy setting to Preferred to use enlightened data transport when possible, with fallback to TCP. In this release, this feature requires Citrix Receiver for Windows 4.6 or Citrix Receiver for Mac 12.4. For more information, see ICA policy settings.

So it started with 7.12, however it was for experimental use only. With 7.13 it is fully supported and implemented. Everything is there: The VDA installation opens firewall ports, Director is aware if UDP based HDX/ICA and much more. So it’s stable and ready to use with 7.13.

Enabling UDP based HDX/ICA in Citrix XenApp / XenDesktop

You won’t be surprised to hear: It’s a policy! So I open GPEdit.exe and select my GPO

We are looking for HDX adaptive transport. The official name would be HDX Enlightened Data Transport Protocol, but Citrix naming conventions are inconsistent here. The policy itself will look like that:

Default setting is Off. This means: we use TCP based transport only. I suggest changing to Preferred (so we prefer using UDP based transport, but we may use TCP based transport if a client is unable to use UDP, this would be true for an old client, or a fire wall blocking UDP/2598 or UDP/1494). Don’t use Diagnostic Mode. This will force UDP based transport. Elder clients won’t be able to connect.

My current session is UDP based. It is visible in Citrix Desktop Director

Unfortunately session details in connection centre does not show UDP transport (way to go, Citrix!)

Which Receiver?

It’s a brand new feature, so we need to use the latest version of Citrix receiver: 4.6. It is not supported.

What about the NetScaler side?

Well on NetScaler side we need at least NetScaler version 11.1 Build 52.13 (March 1st 2017). The release notes read like that:

The NetScaler Gateway appliance now supports the HDX Enlightened Data Transport (EDT) as a data transmission path. EDT provides a high definition in-session user experience of virtual desktops for users running a Citrix Receiver.

[From Build 51.21] [# 659795, 666135]

So that’s it. Upgrade to (at east) 11.1 Built 52.13 to enable HDX Enlightened Data Transport (EDT)! No policies needed. However you have to enable DTLS on your NetScaler Gateway and rebind certificates.

So let’s enable DTLS: (in my case DTLS already was enabled after upgrading, if your DTLS is enabled you should skip this section).

You’ll find this dialogue after opening your VPN vServer (NetScaler Gateway → Virtual Servers), and then click edit (the pen) with Basic Settings. The setting is hidden, you have to click › more.

After doing this you will have to unbind SSL certificates and rebind (I followed Citrix product documentation about that). I don’t understand why this step is necessary.

Your firewall …

… has to permit UDP 2598 and UDP 1494 from NetScaler SNIP to all your HDX/ICA servers (Citrix XenApp / XenDesktop servers).

Citrix NetScaler Logging and policy trouble shooting

Some times it’s quite hard to understand what’s going on. There is so much mystics about policies. And it’s even harder to understand what went on (past tense). “Johannes, there had been several problems connecting to <any blabla application here>” “I’m sorry, I can’t know. What did they do? The request could have been blocked by a responder policy or by WAF”. Did you ever get an answer to a “What did you do?” question?

So we have to do reverse engineering. The only thing we know is: There had been something going on, and it failed. Maybe we know some of the symptoms.

That’s a mess. And we don’t need to act blind like that. There are logs in a Citrix NetScaler!

Where do logs too?

All logging goes to /var/log/ns.log. This log gets periodically archived and recreated. That’s the source of information if our Citrix NetScaler web application firewall (WAF) blocked!

How do WAF-Logs look like?

Let’s have a look at some entries:

Mar  9 09:26:28 <local0.info> 192.168.0.25 03/09/2017:09:26:28 GMT 82e6de130138 
0-PPE-1 : default APPFW APPFW_STARTURL 5852 0 :  81.153.82.18 40102-PPE1 /PctEsh
SVh4Uv9sK/sBQOR5GSdg0001 app_fw_blog_data Disallow Illegal URL: https://blog.nor
z.at/category/common-citrix-themes/page/3/?as_url_id=AAAAAAW5jxPwYiwdw2xu-S6V6uY
6IgElFoDo1HPM_1hQWBYABzbf-StjgFD_ZnsEPbr_nDarC6Y-P6y_CgyS0u_ubJBmqVmA7df3rvHA5Io
PE2Hq6w%3D%3D <blocked>

So we see it’s a Citrix NetScaler Web Application Firewall (WAF) log (APPFW). It’s the start URL function blocking (APPFW_STARTURL). The policy is called app_fw_blog_data. The vServer is at IP 192.168.0.25. The URL, my attacker (IP address 81.153.82.18, British Telecom, if IP location finder is right) wanted to access, was https://blog.norz.at/category/common-citrix-themes/page/3/ with some parameters attached to it. Taking a closer look at it I can see: It’s a non existing page, it’s a forceful browsing attack!

Let’s see a second one:

Mar  9 09:36:07 <local0.info> 192.168.0.25 03/09/2017:09:36:07 GMT 82e6de130138 
0-PPE-0 : default APPFW AF_400_RESP 193 0 :  36.84.91.100 14386-PPE0 - norz_df h
ttp:/// Bad request headers.No host header <blocked>

This one is also by my WAF (APPFW). It’s the AF_400_RESP function, the HTTP request syntax checker. vServer, again, 192.168.0.25. The request came from 36.84.91.100, an IP from Indonesia (PT Telkom Indonesia). The WAF policy was norz_df. There had been bad request headers, so the WAF did not let this request pass through. I guess it’s a request generated by a scan- bot.

Mar  9 10:08:28 <local0.info> 192.168.0.25 03/09/2017:10:08:28 GMT 82e6de130138 
0-PPE-1 : default APPFW APPFW_DENYURL 6195 0 :  56.181.27.117 43870-PPE1 - norz_
df Disallow Deny URL: https://norz.at/test/ for rule pattern="/test/?" <blocked>

This one: also by my WAF (APPFW). It’s the APPFW_DENYURL feature, so i’s a denied URL. The rule triggered was 6195, blocking access to /test/?. The rest like the ones above. Easy like that.

Mar  9 10:13:09 <local0.info> 192.168.0.25 03/09/2017:10:13:09 GMT 82e6de130138 
0-PPE-1 : default APPFW APPFW_BUFFEROVERFLOW_URL 6241 0 :  56.181.27.117 44091-P
PE1 - norz.at URL length(1420) is greater than maximum allowed(1024): https://no
rz.at/agb?1234567890123456789012345678901234567890123456789012345678901234567890
12345678901234567890123456789012345678901234567890123456789012345678901234567890
12345678901234567890123456789012345678901234567890123456789012345678901234567890
12345678901234567890123456789012345678901234567890123456789012345678901234567890
12345678901234567890123456789012345678901234567890123 <blocked>

Again the WAF (APPFW), the subystem was APPFW_BUFFEROVERFLOW_URL. So there was a buffer overflow about the URL. If you take a look at it you see: there is an enormous query string. (the attacker was me, so it’s easy to find out)

What about other policies like responders?

<put 4 letter word of your choice here> there is no logging about responder policies.

Is this true?

Sorry to say: yes. So we can’t reproduce what’s going on.

But?

You long for a but? OK. I have a but!

But you can make them logging! They don’t log, unless you make them logging. Great news, isn’t it?

How to make (responder) policies log

Let’s have a look at a policy:

This is a Citrix NetScaler responder policy dropping requests originating from well known malicious IPs. (IP reputation is a platinum feature).

Let’s take a closer look: There is an action, very well known to all of us (drop in this case) and there are two more actions: a Log Action and an AppFow Action. The appflow action would send data about this request to an IPFIX (NetFlow) receiver of choice, let’s say, Splunk or Solarwinds, or Citrix tools like Insight Center or MAS. Could be a great idea, but I don’t go that way. Instead I wanted this logs to appear at /var/log/ns.log

So we click to the + near to Log Action. We now ´may create our own log actions. I’ll create a sample log:

This action will write an informational message into /var/log/ns.log. This message will look like that:

Mar  9 12:01:58 <local0.info> 172.31.1.31 03/09/2017:11:01:58 GMT ns01 0-PPE-0 : 
 default RESPONDER Message 50652 0 :  "A request for / from: 36.84.91.107 was dr
opped by res_pol_drop_malicious"

So, my responder policy logs! You see, I included the URL requested and the IP address of the attacker. You may create a string of your choice, using any policy expression you like.

Log destination will be /var/log/ns.log (optional /var/newnslog/ns.log)

Where do I find these log actions? From command-line it’s show audit messageaction. Creating my policy would be add audit messageaction log_drop_malicious INFORMATIONAL "\"A request for\" + HTTP.REQ.URL.URL_RESERVED_CHRRS_SAFE + \" from\ " + CLIENT.IP.SRC + \" was dropped by res_pol_drop_malicious\""

In GUI I can find all these policies in System → Audit → Message Actions.

I hope you find this article is beneficial. Have fun!

Johannes

My NetScaler VPX ran out of disk space. I did several tricks like adding a second disk and mounting ít into the update directors, to be able to upgrade my NetScaler.

Today I found time to investigate this issue. I had no success flooowing CTX133588. I so I started looking for the largest files and directories in /var.

root@82e6de130138# du -a /var | sort -n -r | head -n 10
13033540        /var
11023956        /var/ns_system_backup
466274  /var/log
461740  /var/log/db
459822  /var/wi
434188  /var/netscaler
305070  /var/log/db/default
248492  /var/nslog
183824  /var/ns_system_backup/backup_7b829416-9965-475b-8dfb-d57d18ce9ba0.tgz
183824  /var/ns_system_backup/backup_1676b077-6132-47f5-88a5-aab4832e5c93.tgz
root@82e6de130138# df
Filesystem  1K-blocks    Used    Avail Capacity  Mounted on
/dev/md0       358830  344602     7052    98%    /
devfs               1       1        0   100%    /dev
procfs              4       4        0   100%    /proc
/dev/ad0s1a   1482638  941530   422498    69%    /flash
/dev/ad0s1e  14348494 2009796 11190820    15%    /var
root@82e6de130138#

What a big surprise! 11 GB in /var/ns_system_backup? How comes? I looked into it.

root@82e6de130138# ls -l
total 11019220
drwxr-xr-x  2 root  wheel       3584 Mar  8 17:13 WebServer
-rw-r--r--  1 root  wheel  188077595 Mar  5 17:14 backup_083f6140-a660-41d5-83f2-389403f29334.tgz
-rw-r--r--  1 root  wheel  187946493 Feb 20 17:13 backup_0bd31ba5-fa66-4e40-a912-d6c1a9ffdbdc.tgz
-rw-r--r--  1 root  wheel  187814966 Feb 14 05:13 backup_11653391-2dff-4c22-ba22-3cd9212cd7c2.tgz
-rw-r--r--  1 root  wheel  187751996 Feb  8 05:13 backup_15904e38-2e8f-49f1-ad8c-e153d9daf980.tgz
-rw-r--r--  1 root  wheel  188106376 Mar  8 17:15 backup_1676b077-6132-47f5-88a5-aab4832e5c93.tgz
-rw-r--r--  1 root  wheel  187932332 Feb 19 05:13 backup_189cc344-e721-40ae-81af-55935a0ee7ac.tgz
-rw-r--r--  1 root  wheel  187961509 Feb 22 05:13 backup_195f2b0a-8b21-4be9-b277-50220247b553.tgz
-rw-r--r--  1 root  wheel  188000564 Feb 26 05:14 backup_26ac5a11-0c8c-4034-b79b-64a94fee1ac1.tgz
-rw-r--r--  1 root  wheel  188098718 Mar  7 05:14 backup_3379fb1a-068e-4bc5-acfc-091108cece18.tgz
-rw-r--r--  1 root  wheel  187761902 Feb  9 05:13 backup_34fd7671-fad1-43b4-b662-0e3b7f221475.tgz

a total of 60 files, round about 183 MB each. In addition there was a subdirectory with the name of my admin partition, containing 60 more files, 7 kb each.

All these files had been relatively new. They had been created at 5:13 AM and PM, so my first guess was a cron job.

These tgz files are a full copy of /flash/nsconfig and selected subdirectories of /var. I could not find a cron job doing this backup, and I could not find any documentation about a /var/ns_system_backup folder.

I kept playing round, but did not find. I posted this question on dicussions.citrix.com and Carl Stalhood asked do you have MAS in place? Yes I do!

It’s MAS to be blamed for!

So I opened MAS, System → System Administration and clicked on Instance Backup Settings. MAS does backup on managed devices twice a day by default. I changed to one backup in 10 days. It should keep just 1 backup, but it failed to delete. I also upgraded MAS and hope problem is solved by now …

Versions involved:

MAS: 11.1 52.26
NetScaler: 11.1 Build 52.13 and 51.26