Creating a Citrix NetScaler Test environment

Being a Citrix Certified Instructor I am very much aware of the Red/Green/Blue website used during official Citrix NetScaler training (CNS-220, CNS-222). I created my own test website. I usually use it during product demonstrations to present anything from basic load balancing to web application firewall.

I am also aware about problems with the original Citrix labs: They sometimes seem to not load balance. Actually they do, but, because this page is compromised of several files, it may appear to show the same colour all the time. I wanted to avoid this, so my pages don’t use external style-sheets, scripts and images, instead I added everything into the HTML file (you may include images using base 64 encoding).

You may download my test website from here. I will update my page every now and then. You can download it as often as you like. The download will ask you for your E-Mail address. I promise not to send any SPAM to you, instead I’ll just count the numbers of downloads.

Requirements and prerequisites

My environment is made of a single Windows server (I tested using 2012R2 Server) and a NetScaler VPX. You may very well use some entry level virtualization solution like VMWare workstation or Hyper-V on your laptop computer, but professional environment like Xen-Server, KVM and simmilar may also be used of course.

My download does not include the machines, but the website only. There is no license included, however you may request a demo license using your Citrix account)

Installation procedure

Import a Citrix NetScaler VPX into your virtualization solution. (www.citrix.com -> downloads -> NetScaler ADC -> Reliese xxx -> Virtual Appliances).

Install a Windows Server (I tested using 2012R2, but I guess it will work with any version from 2008). This server should have 4 GB RAM as a minimum

IP addressing

I used 192.168.0.100 as a NSIP, 192.168.0.110 as a SNIP, 192.168.200 ff for virtual servers

Windows machine used 192.168.0.20 to 24

Windows set up

Roles and features

After setting up this windows machine you have to set up IIS. Start Server Manager (if it’s not already started) and click “add roles and features”. Click Next 3 times.

Select Active Directory Certificate authority,  Web Server IIS and DNS. If asked select following roll- services:

• .NET Extensibility 4.5
• ISAPI Extensions
• ISAPI Filters
• .NET Extensibility 3.5
• Certificate Authority
• Certificate enrolment web service

Setingt up the Certificate Authority:

• stand alone CA
• root CA
• create a new key
• SHA 256 (or highter)
• confirm all the rest of the questions

IP configuration

select your network adapter. Change IP address. Set 192.168.0.20 255.255.255.0 as an IP address (you may use any other address range you like, but I use 192.168.0.x in my example). DNS should be 127.0.0.1, gateway depending on your settings.

Click advanced. add 4 more IP addresses (192.168.0.21 to 192.168.0.24).

IIS settings

Copy my files into c:\inetpub directory.

Open Internet Information Server Management.

Open your server and select sites. Right click your server and select add website. Create 4 virtual websites:

Sitename: Sitie1 (2,3,4)
Site path: C:\inetpub\wwwroot1 (2,3,4)
type: http
IP address: 192.168.0.21 (22,23,24)
hostname: (empty)

ASPx is just needed for the Citrix NetScaler Web Application Firewall test page. Check, if ASPX works correctly surfing to http://192.168.0.24/Allow.aspx. If it does not: follow this Microsoft instructions.

additional software

If you want to use this machina as a workstation as well install Google’s chrome Browser and Mozilla FireFox. Alternatively you may create a dedicated work station or use your desktop work station.

You’ll very likely need the SSH terminal putty, the secure copy tool WinSCP and the network monitor WireShark. They can be considered to be the tools used by a NetScaler admin during his daily work.

Labs:

Prerequisites

in DNS manager create a new Forward lookup zone called test.lab.

Create hosts:

• colours.training.lab 192.168.0.200
• cs-test.training.lab 192.168.0.201
• aaa.training.lab 192.168.0.202

1st lab: create a load balancing vServer

Server:

• srv_red -> 192.168.0.21
• srv_green -> 192.168.0.22
• srv_blue -> 192.168.0.23

Services:

• svc_red (HTTP/80)
• svc_green (HTTP/89)
• svc_blue (HTTP/80)

Loadbalancing vServer

• lb_vsrv_colors (192.168.0.200/HTTP/80)

additional labs:

• add persistence (source IP, cookie based, …)
• disable services and see what hapens (re-enable these)
• unbind red service, create an additional loadbalancing vServer (non addressable), called lb_vsrv_red. Set this one in protection as a backup virtual server. Disable service blue and green. Which status does lb_vsrv_colors have now? Does it work? Why? rebind red service.

2nd lab: certificates

• use the wizard to create a key and a CSR (hostname *.training.lab). Surf to 192.168.0.20/certsrv. Request a certificate. download this certiticate as BASE 64. Install it into NetScaler
• create a lb vServer lb_vsrv_colors_secure (192.168.0.200/SSL/443). Bind the 3 services and your newly created certificate. Surf to https://colours.training.lab

3rd lab: content switching

• create a new content switching vServer cs_vsrv_browser 192.168.0.201/HTTP/80
• create two new cs-policies
  • HTTP.REQ.HEADER(“User-Agent”).CONTAINS(“Trident”)
  • HTTP.REQ.HEADER(“User-Agent”).CONTAINS(“Chrome”)
• bind these policies to cs_vsrv_browser. The Trident policy should invoke the red, the Chrome policy the blue server. Surf to cs-test.training.lab using an MS- Internetexplorer, a Google Chrome and a FireFox.

4th lab: responding

• create a responder policy to forward users from http://colors.training.lab/ to https://colors.training.lab/ and bind it to lb_vsrv_colours
• create a responder policy forwarding users from https://colors.training.lab/ to https://colors.training.lab/home.htm
• unbind the responder policy from lb_vsrv_colours

5th lab: rewriting

• create a rewriting policy rewriting requests for http://colors.training.lab into http://colors.training.lab/home.htm and bind it to lb_vsrv_colours
• remove server header from HTTP-response and bind it to lb_vsrv_colours
• add a server header into http response stating your server to be an Apache and bind it to lb_vsrv_colours

Sometimes we have to schedule commands in a Citrix NetScaler. A good example would be:
force HA failover
It’s obvious, we don’t want to fail over during day time to not disconnect TCP connections, to not interrupt users. The best time would be something like 3:30 AM. It’s obvious, we don’t want to set an alarm for 3:00 to get up, take a shower, brush teeth, just to force an HA fail over. At least I don’t want!

Scheduling an HA fail over for off peak hours is important for both, Citrix NetScalers proxying big files for download and for NetScaler Gateways: During HA fail-over we will loose TCP-sessions, so downloads will break and HDX (ICA) sessions will get disconnected.

Starting to dig into Citrix NetScaler

Inside a NetScaler there are two operating systems working at the same time and therefore two different shells:

1. the Citrix NetScaler shell, the first one you connect too using putty (or even better: smarTTY)
2. the BSD shell. It can be reached typing
   shell
   into NetScaler’s command line

There is no chance to schedule commands in NetScaler OS. But BSD is just an ordinary UNIX (please don’t call BSD a Linux, it is not). My first guess would be to use at, however at is not there. So we need to use crontab.

Crontab in UNIX is used to schedule commands on a regular base. So crontab would be great to schedule a backup of Citrix NetScaler configuration, it’s not perfect for one time commands.

We could install at into BSD, but I never install software into a NetScaler and I would strongly advise you to keep away from doing this. So we need to use crontab.

How to execute a NetScaler shell command from BSD?

That’s a big question. BSD shell just allows to execute BSD commands. So what now?

nscli

nscli is a UNIX command on a NetScaler, allowing users to execute NetScaler commands from BSD

root@82e3d3135738# man mscli
No manual entry for mscli

shit.

root@82e3d3135738# nscli --help
Usage: nscli [-norc]
[-U []:]
[-D ] [-s]
[[-k] ]>

where:
-norc causes the personal initialization file, ~/.nsclirc,
to be skipped
is the IP of the target NetScaler
is used to log in to the target NetScaler
is an integer between 0 and 9
-s stifles "exec:" and "Done" messages
is any nscli command
and
-k causes the program to keep-a-going after command
root@82e3d3135738#

much better! so we have to execute a command like that:

nscli -U 127.0.0.1:nsroot show ns runningconfig

so we specify a NetScaler IP (no SNIP, sorry guys, we’re dealing with BSD!), an user name and NetScaler commands after this.

It works fine, unfortunately we get prompted for a password. So we can’t easily use this command in a batch file? Yes we can. There is some information missing: we may specify a password as well. No too beautiful, as this batch file will also contain the password in plain text, but possible. The command would look like that:

nscli -U 127.0.0.1:nsroot:your_Password_goes_here show ns runningconfig

easy? Yes, it is! You may even skip the IP using this command locally:

nscli -U :nsroot:your_Password_goes_here show ns runningconfig

This leading : assumes an IP of 127.0.0.1.

Using crontab on a NetScaler

Using crontab on a NetSaler would be more than just easy. Just add a standard crontab entry into /etc/crontab.

30 3 * * * root nscli -U 127.0.0.1:nsroot:your_Password_goes_here force ha failover -force

That’s simple.

Next we’ll have to kill cron and start cron (cron start) again, so it will reread crontab.

root@82e3d3135738# cron start
cron: cron already running, pid: 965
root@82e3d3135738# kill 965
root@82e3d3135738# cron start

Unfortunately this entry won’t disappear after executing, so it will get executed tomorrow and the day after tomorrow as well. So you have to remove this entry tomorrow morning. Still by far better than getting up in the middle of the night, isn’t it?

What else could we do?

We could also use this for daily tasks, such as backing up ns.conf, purging log files and many more!

BUT

never reboot your NetScaler! Why? All content in /etc gets discarded. /etc is just RAM, no disk based file system.

What to do?

Well we need to rewrite /etc/crontab with every reboot! I’m pretty sure you won’t like to do this. There has to be an other way, a more automatic way, to write data into crontab!

We could use /etc/rc.conf to fill crontab after reboot. Unfortunately we face the same problems here: It will get discarded during boot. However there is a file called /flash/nsconfig/rc.netscaler (see CTX122271). This is the template for the /etc/rc.conf.

There is a good description in Citrix forums by Rob Harp about how to use it. Rob’s example is about doing daily backups. I’d suggest reading his article.

An important note in the end

Keep in mind: Changes to BSD shell is executed on this very Citrix NetScaler only. It will never get executed on the other node of a HA or cluster! You’ll probably have to do these changes with all nodes!

Why would you like to customize a 404 page?

Well It’s all about misleading information. A hacker has very limited chance to get friend with your web server. On the other way, he needs to find out as much as any possible. The more he knows, the more likely his attack will be successful. On the other hand he has to let sleeping dogs lie. With other words: He must not alarm you.

One of the most important things to know is: What kind of web server do I have to deal with?

The first source to look into is a HTTP response header called Server. Information here may be very verbose. I don’t know why this header is part of HTML standard, but actually it is.

The Server response-header field contains information about the software used by the origin server to handle the request. The field can contain multiple product tokens (section 3.8) and comments identifying the server and any significant subproducts. The product tokens are listed in order of their significance for identifying the application. (RFC 2616)

This is an example server header:

Apache/1.3.28 (Unix) mod_ssl/2.8.15 OpenSSL/0.9.7c mod_perl/1.27 PHP/4.3

In this case, it’s a very outdated Apache, using an outdated SSL module, outdated Perl and outdated PHP. It’s easy to change this information using Citrix NetScaler rewrite policies (DELETE_HTTP_HEADER and INSERT_HTTP_HEADER).

But hackes are not that stupid. They will probably verify this information. My personal next try would be: check for a non existing site. We will see a 404, page not found. Being careful I would use an existing URL, however do a minor typo, just like http://norz.at/default.html instead of http://norz.at/default.htm. You would probably not be scared if you would see a request like that watching your logs.

The next thing he would see is a 404, Not Found. It will be specific to your server, if you don’t change it. And a 404 page originating from an IIS6  would, for sure, come from an IIS 6, no matter what the server header tells you.

More reasons to change the 404 page

of course there are even more reasons to change the 404 page: customized 404 pages seem to be funny, they may help people to find the content needed, and so on.

Why not change your web server?

This would be possible. However, you would need to change all your load balanced web servers. There is an other reason: Responder policies. I will never return a “401 Unauthorized” or “403 forbidden“. I would rather return a “404 not found”. Being a hacker I would be very excited to see a 401 or 403!

I would think: here it is, but someone protects it from being accessed. But how could I find out what’s going on, if a Citrix NetScaler uses exactly the same 404 page as the original web server? I would probably think the file is not there.

My solution

My first attempt was creating a simple rewriting policy changing the body with something like “HTTP/1.1 404 OK\n\r\n\r<html><head><title>404 File not found</title></tead><body><h1><font color=\”#802020\”>404 File not found!</font></h1><p><font color=\”#802020\”>The file you requested is not on this server.</font></p></body></html>” in it.

The length of the text is limited, so this is not a good solution. And I would rather like to place the file “somewhere” on my web server, so it’s pretty easy to change.

I spent some time thinking what to do and made up my mind to use the HTTP callout feature. It was my first ever attempt to use HTTP callout, and I’ll describe how it works.

NetScaler’s HTTP callout feature

HTTP callout is intended to be used in policies to check something, i.e. an IP address, against a web based service. So I could send an IP address (CLIENT.IP.SRC) to a web server containing an IP black list. This web server then would respond with something indicating good or bad.

I do something completely different: I will retrieve the content of the 404 page from a web server. To do so I have to navigate to App Expert -> HTTP Callouts.

Like any policy it has to get a name. I do my callout to a vServer, so I have to specify the server here. My request will be attribut based, that means, I will be able to send regular HTTP requests, mine is a HTTP GET. My web server uses several host names for various virtual pages, so I have to specify a proper host expression. This makes sure, we retrieve the file from the right source. The URL Stem Expression is the URL we want to retrieve.

We scroll down to the bottom and select the return type TEXT and the expression should be HTTP.RES.BODY(65538). The number is the number of bytes to retrieve.

So, my policy will connect to a NetScaler vServer called cs_vsrv_norz.at to retrieve a file called /notfound.htm, setting the header Host to norz.at (i.e: http://norz.at/notfound.htm). It will then return all the body of this file, containing links to style definitions, pictures and so on.

command line version:

add policy httpCallout callout_retrieve_404 -vServer cs_vsrv_norz.at -returnType TEXT -hostExpr "\"norz.at\"" -urlStemExpr "\"/notfound.htm\"" -scheme http -resultExpr "HTTP.RES.BODY(65538)"
set policy httpCallout callout_retrieve_404 -vServer cs_vsrv_norz.at -returnType TEXT -hostExpr "\"norz.at\"" -urlStemExpr "\"/notfound.htm\"" -scheme http -resultExpr "HTTP.RES.BODY(65538)"

The rewrite policy

The rewrite policy should be a very simple thing:

The NetScaler rewrite action using a HTTP callout

add rewrite action callout404 replace_http_res "SYS.HTTP_CALLOUT(callout_retrieve_404 )"

It’s a replace policy. Expression to choose target location is all of the HTML body, so HTTP.RES.BODY (65536). To be more precise, it’s the first 65536 byte of the body (a 404 page typically is by far smaller). The Expression is the text we will use to replace the former body with. It is the HTTP callout request, in my case SYS.HTTP_CALLOUT(callout_retrieve_404).

The NetScaler rewrite policy

add rewrite policy rw_pol_404 "HTTP.RES.STATUS.EQ(404)" rw_act_404

This policy will get applied if  the HTTP response status is a 404 (HTTP.RES.STATUS.EQ(404)). I then bound this policy to my web server. That’s it. It was pretty easy.

I was so enthusiastic, when I found out about NetScaler admin partitions! What a great extension to existing NetScalers! However I got disillusioned finding out about limitations. It took me some time to find out how to overcome this issues, but there are still some features missing.

The feature I missed most is doing traces. It’s not listed in the compatibility list, so it’s intended to be there. But it is not! If you click into System and Diagnostics you’ll see just very little content, and definitely no nstrace (this is about NetScaler versions up to 11.1 48.10).

Citrix documentation is always right, and if it’s not, it’s right even though. So I tried to do a nstrace from commandline. It started and stopped without any problem. Unfortunately I could not find the output of my nstrace in /var/nstrace subdirectory.

So I searched for it, and found it in /var/partitions/<partitionname>/nstrace.

So that’s how I do an nstrace inside a NetScaler admin partition:

I use putty to connect to my NetScaler. Masochists might prefer to use the built in terminal from GUI, however I don’t tend to masochism.

switch partition <partitionname>
start nstrace -filter “CONNECTION.SRCIP.EQ(<source IP>)” -size 0 -time 3600 -link
stop nstrace

So I log into my Citrix NetScaler. I change into my partition (currently partition names can’t auto complete, so make sure you know the name; show ns partition will list all partitions.

Next I start the trace. To do so I follow CTX120941.

start ns trace will simply start the trace. -filter will filter a connection. Usually you would use connection objects like CONNECTION.SRCIP.EQ(<source IP>) or CONNECTION.DSTIP.EQ(<destination IP>) to limit the amount of data captured. -size=<size> will limit the amount of data captured per packet. If you want to debug HTTP problems you would very likely set the size to 0, as this would capture all of the packet (0 sometimes means unlimited). -time=<time> will automatically stop the trace after <time> seconds.

After doing your trace you may execute a stop nstrace command to stop your trace. This is not needed if you set the time parameter, but I prefer to stop traces instead of setting a time parameter.

In the end you need to download the trace file. I usually use winscp as a secure FTP client, download it from /var/partitions/<partitionname>/nstrace and view content in your favourite network monitor. I prefer to use WireShark, as it fully supports NetScaler. Citrix support also uses WireShark.

Additional parameters for tracing

-tcpdump ENABLED switches to TCPdump format. TCPdump is a standard UNIX® format for network tracing. Different to NStrace it does not contain L1 information (ports), but it is understood with most network tracing utilities. You may want to use it together with -perNIC ENABLED if you want to debug routing problems. This will create a separate trace file pre NIC. You then have to scroll down both instances of your network monitor in parallel (and synchronize these 2 windows if you scroll down). However you may prefer to download free WireShark and use it instead as it understands NStrace: one window, all L1 information is contained in your trace.

–link Also trace filtered connection’s peer traffic. Only makes sense in combination with -filter. It will trace all traffic filtered plus all traffic resulting from your filtered traffic, so traffic from client to VIP and traffic from SNIP to your back end server. This is a very good one!

-mode SSLplain will decrypt all SSL traffic. Because of this you won’t see any SSL hand shake, instead, all SSL traffic will appear to be plain text. This may be beneficial if you want to debug encrypted traffic. Caution: this may expose sensitive data to you (the admin)

There are several more parameters. You may find them in Citrix NetScaler product documentation.

I needed to use a Citrix NetScaler both, as a SAML identity provider (IDP) and service provider (SP). So I set up my test environment accordingly.

What my test environment looked like:

You see, I created two admin partitions on my Citrix NetScaler, one for the service provider (SP partition), cotaining both, the SAML SP and a web server, and one for my identity provider (IDP partition), containing the IDP.

I used this partitions to emulate “2 different NetScalers” as it does not make sense to have both, SAML-SP and SAML-IDP in the same data center (you could do conventional LDAP/RADIUS/TACACS authentication instead).

How SAML works:

SAML authentication uses an external server for authentication, the so called SAML Identity Provider (SAML-IDP).

The SAML Service Provider (SAML-SP) is local, close to the resource, and calls for Authentication to the SAML-IDP.

So a user connects to a resource. If the user had not been authenticated before, he gets gets forwarded to the logon server, the so called SAML-SP.

The SAML-SP forwards the user to the SAML-IDP for actual authentication. The SAML-IDP does the authentication.

After successful authentication, the SAML-IDP forwards the user to the SAML-SP. It also passes the so called assertion, the prove this user was authenticated successfully. You could think of an assertion like a man’s ID card. As soon as the SAML-SP has validated the assertion, it forwards the user to the resource.

SAML-SP and resource are always located on the same Citrix NetScaler, the SAML-IDP is usually located “somewhere else on the internet”.

Certificates

SAML uses certificates to establish trust between SAML-SP and SAML-IDP.

The SAML-SP uses a server certificate to authenticate to the SAML-IDP. This certificate (not the private key, of course) has to be on the SAML-IPD as well, so it can get checked.

The SAML-IDP uses a certificate to digitally sign (and encrypt) the assertion. This certificate (again: not the private key) has to be present on the SAML-SP, so the SAML-SP is able to decrypt and validate the assertion.

It’s possible to use the same certificates for both, SSL between client and SAML-IDP / SAML-SP, and to prove identity, however I would rather use private (and therefore more trustworthy) certificates to prove identity.

NetScaler as a SAML Service Provider (SAML-SP)

A Citrix NetScaler may be a SAML identity provider for any SAML service provider. An other NetScaler may be the service provider, but also services like Microsoft Azure, Microsoft Office 365, Citrix Sharefile and many more may use a NetScaler as an authentication source.

In my example I just created a simple load-balancing vServer and added authentication to it. There is nothing special about it, in fact I used my test server (a description might be found there).

add server www 10.127.255.250
add service lb_svc_www www HTTP 80
add lb lb_vserver lb_vsrv_www HTTP 192.168.0.4 80
bind lb lb_vserver lb_vsrv_www lb_svc_www

The NetScaler SAML Authentication policy

The NetScaler SAML Service provider action

GUI: Navigate to:

Security → AAA-Application Traffic  → Policies → Authentication → Basic Policies →  SAML

With SAML Actions click Add.

add authentication samlAction saml_sp_server -samlIdPCertName lets_encrypt -samlSigningCertName lets_encrypt -samlRedirectUrl "https://idp.norz.at/saml/login" -samlUserField "Name ID" -samlIssuerName "https://sp.norz.at"


The NetScaler SAML Service provider policy

GUI: Navigate to:

Security → AAA-Application Traffic  → Policies → Authentication → Advanced Policies →  SAML

add authentication Policy SAML_SP_pol -rule true -action saml_sp_server

The SAML Service Provider (SAML-SP) Authentication vServer.

Click add

Provide name and IP (port s usually 443, protocol can’t be changed)

bind a server certificate (this one gets exposed to users, so it has to be trusted!)

bind the authentication policy you previously created

add authentication vserver SAML_SP SSL 192.168.0.4 443
set ssl vserver SAML_SP -ssl3 DISABLED
bind authentication vserver SAML_SP -policy SAML_SP_pol -priority 100 -gotoPriorityExpression NEXT

NetScaler as a SAML Identity Provider (SAML IDP)

A Citrix NetScaler may also get used as a SAML Identity Provider (SAML-IDP). This allows to authenticate to any authentication source like LDAP, RADIUS, Certificates, TACACS, local (to IDP), Negotiate, O-Auth, SAML, WebAuth, EPA or Citrix StoreFront. In my example I authenticate to TACACS (TACACS policy is not included)

Creating the a SAML Identity Provider Policy

Creating the a SAML Identity Provider Action (Saml IDP Action) on a Citrix NetScaler

Navigate to:

Security → AAA-Application Traffic  → Policies → Authentication → Advanced Policies →  SAML IDP. Go to Profiles.

Click Add.

add authentication samlIdPProfile SAML_IDP_profile2 -samlSPCertName SP-assertions-signing-cert -samlIdPCertName IDP-Signing-Cert -assertionConsumerServiceURL "https://sp.josel.net/cgi/samlauth" -samlIssuerName "https://sp.josel.net" -signatureAlg RSA-SHA256 -digestMethod SHA256

The SAML-IDP policy

add authentication samlIdPPolicy SAML_IDP_Policy -rule true -action saml_idp

The authentication policy

I don’t go into authentication policies here. Just follow Citrix bast practices, there are many guides out there. I created a policy similar to CTX113820.

The SAML Identity Provider (SAML-IDP) Authentication vServer.

click add

provide name, IP address and port, usually 443 (the protocol can’t get changed)

bind a server certificate. This one gets exposed to the user, the user has to trust this certificate!

bind an authentication method and a SAML IDP policy

select both, the IDP and the authentication policy

Trouble shooting

I used following tools:

Citrix NetScaler’s log (Yes, there is a log on a NetSaler and SAML issues get logged there! You look at /var/log/ns.log)

FireFox add-on SAML-Message Decoder (also available for Chrome)

Citrix NetScaler Network traces

Issues:

I have seen several issues recently:

SAML-SP fails to forward to SAML-IDP

detected: error in browser

check settings on in SAML-SP’s SAML Authentication action: Redirect URL

SAML-IDP fails to forward to SAML-SP

detected: error in browser

check settings on in SAML-IDP’s SAML-IDP Authentication action: Assertion Consumer Service URL

Certificate not trusted on SAML-IDP

detected: confusing message in browser, log in IDP’s /var/log/ns.log

add SAML-SP’s signing certificate to SAML-IDP’s SAML-IDP profile: SP-Certificate Name

I hope. that helps. Just drop me a message if you need more information. You’re very much welcome to link to my blog / my website. Thanks!

Recently I found out: DNS is a big bandwidth waster on my internet connection. Strange, isn’t it? DNS? So I started a network trace on my firewall: someone is abusing one of my DNS servers. I guess it’s a kind of malware using my DNS server, but I am not really sure. The domain was X99MOYU.NET belonging to a Chinese company called ZhuHai NaiSiNiKe.. Content of the website is not available (403, Access Denied).

My first idea was to reply with 127.0.0.1 as a DNS response. So I created a zone for this domain and added records. However they kept sending billions of queries.

Next step was: hiding my DNS server behind a Citrix NetScaler. Now all queries have to traverse my NetScaler. Easy like that: I created a DNS load balancing virtual server and my external IPs point there. Next step: Bind a responder policy to it.

I created a responder policy dropping all queries to domains not hosted in my environment. I did this using a NetScaler pattern-set (as pattern-sets are more efficient and easier to read than endless complex expressions combined with or like:
!DNS.REQ.QUESTION.DOMAIN.CONTAINS(“norz.at”) || !DNS.REQ.QUESTION.DOMAIN.CONTAINS(“norz.cc”) || …)

command line:

add policy patset MyDomains
bind policy patset MyDomains norz.co -index 1
bind policy patset MyDomains norz.at -index 2
add responder policy res_pol_dns_invalid_request "!DNS.REQ.QUESTION.DOMAIN.CONTAINS_ANY(\"MyDomains\")" DROP

My pattern set is called MyDomains. So this policy will get applied (and drop the request) if the DNS query is for a server not contained in MyDomains.

MyDomains contains all domains I host like norz.at, norz.cc, …

Next problem: I opened up my load balancing vServer and wanted to bind a responder policy. Strange, no chance to bind a responder policy!

command line:

I had to go back to my policy, start the policy manager and bind it. Of course I could have bound it using command line as well

bind lb vserver lb_vSvc_DNS -policyName res_pol_dns_invalid_request -priority 100 -gotoPriorityExpression END -type REQUEST

I hope you like it and would be glad to hear some commands about the subject …

(a nice but partly failed try)

Complex web applications may lead to complex NetScaler configuration. And sometimes an administrator may get lost troubleshooting complex websites, especially sites using content switching.

This is an example of a real world website: The portal page is assembled of several independent web applications. Each application is hosted on a specific group of load balanced servers. There are rewriting policies replacing some content on a website, there are also rewriting policies on a global base (and responders, URL transformation, FEO optimization, app firewall, caching, …). Some of the global and some of the server specific content, was not replaced like desired, but some content gets replaced. The current configuration is confusing the admins, and it also confused me.

Main problem here: I can’t look into traffic between a content switching and a load balancing vServer, so I can’t see what’s actually going on in here. Second problem: there is a total of 800 rewriting policies. That’s confusing me, there are too many for me, I can’t keep track of all these policies, I simply don’t remember what they are good for and where they got bound too!

The current solution also used NetScaler MAC based forwarding, but MAC based forwarding had partly undesired influence on some of the load balancing vServers, and on the NetScaler as a hole as it blows up the TCP connection tables (by adding MAC addresses to it).

That’s where admin partition came in my focus!

We got admin partitions in NetScaler 11 (10.5e), a possibility to split up a NetScaler into several “virtual” ones. That’s great. I made up my mind to put each load balancing server into a specific admin partitions while I let the content switching vServer in the default (root) partition.

This is a sketch of solution I desired:

The first big problem I faced: two partitions can’t connect into the same subnet. This had been a must have as I would not have been able to change the current networking and routing configuration in a 10,000+ server data centre without an excessive change process lasting for several month. So we stopped here, almost a year ago.

The new version 11.1 offers a feature called partition shared vLan; this seemed to be the solution! So I tried to set up vLan 1 as Partition shared vLan. This was impossible. I guess, vLan 1 is not a real vLan at all. It’s not comparable to the rest of vLans, but I actually don’t really know.

But I could create a vLan, make it a Partition shared vLan, and bind it to the interface.

Creating vLans

add vLan 1000 -sharing ENABLED -aliasName PartitionShared_vLan

(so we add vLan 1000 with partition sharing enabled. You may skip the alias name, but I always like to add some documentation)

bind vlan 1000 -ifnum 1/2

(we bind this vLan to the designated interface)

Next step: Let’s create the partitions

add partition WebServerApp1

(This partition will be used for a webserver of app1, so I’ll call it WebServerApp1)

Open this partition, scroll down to network isolation, click add binding and select vLan 1000

click on VLANS

and bind vLan 1000

bind partition WebServer -vlan 1000
Currently you can’t unbind vLan 1.

I repeat this step for all admin partitions desired. Now I can put all of my load balancing servers into dedicated admin partitions.

Currently there are several restrictions about NetScaler basic and advanced features in admin partitions:

Restrictions about admin partitions in NetScaler 11.1 build 48.10

A comparison of features may be found here. (Thanks, Balaji, to provide this link)

So there are currently serious ones missing in admin partitions! I highlighted some I was interested in. To me the ones I miss most are App Firewall and Front End Optimization. I would have put this into admin partitions, as this is done on a per application base. I don’t miss Surge Protection, Http Dos Protection and Priority Queuing as this is done during connect on the content switching vServer.

This project does not use NetScaler Gateway. So NetScaler Gateway missing is no problem for me, however I missed the chance to isolate NetScaler Gateway in many other projects. NetScaler Gateway is usually governed by other departments, so it should be in a separate admin partition. Our beloved NetScaler will degenerate into a battle ground between the application delivery and the network group, if we can’t completely isolate it.

I suddenly faced a strange problem (why did it not work?):

Simple: I could not communicate from default partition to WebServerApp1 admin partition. It was a completely impossible thing to do. I tried to send ICMP packets from default to WebServerApp1 admin partition, but without success. Even ARP didn’t work at all.

I started monitoring, both from NetScaler using NSTrace and from a switch board (an other restriction here: NSTrace is only available from command line inside an admin partition, it does not exist in GUI).

I set up a switch board for monitoring. Pinging from default partition to 10.0.1.10 (the vServer inside the admin partition), I saw ARP requests going out of NetScaler, but no ARP replies coming back from the admin partition. Same the other way round. However I could ping all IPs from both partitions from an external server (i.e. 10.0.1.100) and vice versa. My networking problems seem to be internal to NetScaler only.

I added a static ARP entry into default partition for 10.0.1.10 and 10.0.1.1 into the WebServerApp1 partition and tried again. No success.

Sending packets between admin partitions is currently not possible!

I also added virtual MAC addresses to the partition. No success either. There is something spooky going on inside a NetScaler’s internal networking logic making admin partition to admin partition traffic an impossible thing to do.

My current work around is a router VM based on VyOS. I could fix all of my problems by now, I love my deployment, but I hate this tiny little VM: it should simply not be there!

Comments (and a possible solution) are highly welcome …

last update: November 14 /2017

Or: The power of the ANY service type

This is a work around for a well-known problem in NetScaler: Binding NetScaler Gateways to content switching vServers.

This solution does not follow Citrix best practices. Avoid using it, if you can!

My solution will work with NetScaler 10 upward. I didn’t test with 9.x as they are not considered to be secure any more.

The Problem

Up to 11.0 it was impossible to bind a NetSaler Gateway to a Content Switching vServer. By now (firmware versions 12) this is limited to a single NetScaler Gateway. This limitation may be an obstacle to overcome in certain environments. Most companies nowadays suffer under a lack of public IPs. But mos of all: Users don’t like complex environments with tons of different URLs to handle, one for mobile devices, one for PCs, one for trusted, one for untrusted devices and so on. Instead they want to use a single URL for all use cases.

Content switching may mitigate this issue by hiding very different configurations behind a single URL. But this is not true for NetScaler Gateways. In days of old we could not bind any gateway to a content switching vServer at all, now (starting from version 11) we can bind a maximum of one gateway to it.

Why may one gateway not be enough? First of all, it is complexity. It may confuse you if you have to bind tons of different scenarios to one gateway. In my real world experience I see often buggy environments being buggy, as complexity may over work the admins. But there may also be technical reasons. One of my costumer would have to bind round about 50 LDAP sources of costumers and partners. All of them are geographical dispersed and some of them may even be misconfigured and therefore slow. Logon to the last ADs in the list would be painful. Splitting the gateway up into some gateways would speed up things very much.

The solution

This question came up in one of my NetScaler classes. We set up all needed NetScaler Gateways. They are addressable and use private addresses of a separate address space (this address space does not exist outside of NetScaler).

We set up a content switching vServer. I would prefer a SSL-bridge to avoid SSL offloading, however we needed something to base content switching on, so we used a SSL vServer. This is far from being a perfect solution, but it works.

How to bind them together?

My first thought was: pointing the services of the load balancing vServer to the NetScaler gateways. But this does not work, we faced an error stating this IP address is already in use.

That’s my trick: I create load balancing vServers of type ANY and point its services to the corresponding gateways. That’s why these gateway servers use private addresses that don’t exist in your environment. This traffic will never leave this NetScaler.

(graphic by courtesy of Andre Buck)

What’s wrong about this setup?

It does not follow Citrix best practices. So you should avoid using it. On the other hand: everything we do is fully supported: The content switching vServer, the load balancing vServers bound to it, load balancing vServers of type any, and last, not least, the gateways.

We won’t be able to log on to the NetScaler Gateways using smart cards (certificate based logon), if we use SSL-Offloading lb vServers, as these certificates won’t be visible to the NetScaler Gateway.

Why would you use it even though?

It’s currently the only chance to bind more than one NetScaler Gateway to a content switching vServer on a NetScaler.

I recently was asked to teach Citrix SD-WAN. My first thought was: wtf? I asked Google, and Google, knowing everything, spoke to me in infinite wisdom: Citrix SD-WAN’s previous name is Branch Repeater. And Branch Repeater, I did already know this, once was the new name for WanScaler (a product I have been certified on, but never used in real life). Meanwhile the product got rebranded again and is now called Citrix NetScaler SD-WAN. So it is just a rebranded product?

WanScaler once was a great product, caching WAN traffic, and thereby preventing content from traversing a WAN multiple times. “Compression rates” of 1:100,000 had been possible, would probably still be possible, if … Yes, if we nowadays would not encrypt everything. Caching and encryption don’t go together well, never did and will never ever do. That’s why I didn’t recently hear much about WanScaler or CloudBridge. It has its reason to exist, mainly in ICA environment, but never got a big success.

But I do what people want me to do, so I started reading into it. I built my own test environment consistng of 2 SD-WANs, 2 WanEms, a server and a client. And found absolutely thrilling information! It is not just a 3^rd re-branding of a product of very limited area of application, instead it is a brand new approach to WAN: Software Defined WAN, SD-WAN.

There are two different types of appliances: These classical WanScalers, Repeaters, which ever name you want to use (SD-WAN WO [WAN-optimization]), and this brand new type of software defined WAN (SD-WAN SE [Standard edition]). And, in addition, and a blended version, called NetScaler SD-WAN enterprise of course, worth thinking about it!

What’s so totally new about it?

I will just focus on software defined WAN (SD-WAN SE). SD-Wan nowadays is a hype. Gartner says:

By end of 2019, 30% of enterprises will use SD-WAN products in all their branches, up from less than 1% today.

That’s an ambitious prognosis! And Citrix is right here, one of just 5 solutions currently on the market. Gartner:

Organisations looking for WAN optimization or dynamic selection capabilities should consider this vendor, especially when Citrix applications are also present

What means: dynamic selection capabilities?

It’s all about finding out, where to send packets too. Still not clear?

Current deployments:

Usually we have a MPLS connection to branches. MPLS is fast, has low jitter and is reliable, guaranteed SLAs of 99,9% are usual (this means: less than 1 hour down per month), in real life European SLAs will be even much higher. It’s very common to bundle MPLS with a GSM LTE or GSM G4 connection in active passive, as a last mile outage due to construction works is a very likely thing to happen. This will result in a SLA of 99,999%, meaning: 5 minutes down time per year. In addition we usually also have Internet connections (with much lover SLAs of about 98-99%, 7 to 15 hours down per month) in place.

Citrix NetScaler SD-WAN

What’s wrong about it?

Simple like that: we have 3 connections, one costly, one moderate and one cheap. And we only use the costly one for WAN transfer. If we need to upgrade (Gartner speaks about 15% increase of WAN traffic per year, so there are always upgrades coming up), we need to upgrade the most costly one. It’s a damn expensive solution.

Can we make things better?

Yes we can! SD-WAN would be a solution allowing all this 3 connections to be used at the same time. You think about link load balancing? You’re totally wrong! Keep on reading.

A Citrix NetScaler SD-WAN is a virtual WAN solution. Tracerouting your WAN from inside, you would just see a single hop, not two of them, so it’s technicaly to be considered as a tunnel. It’s a tunnel, aggregating of all this three connections. But the really important thing is: it’s a UDP based tunnel using UDP port 4980.

What’s great about an UDP based tunnel? It’s just a less reliable tunnel, isn’t it?

Yes and no. UDP is not reliable at all. That’s true. UDP does not have connections like TCP has. No sequence numbers, no acknowledgement numbers. If we need reliability, we have to add it on a higher layer. However, using a stateless protocol, we can send one packet of a single TCP connection on MPLS, the next one on GSM. Even more: the packet and its acknowledge don’t even need to use the same connection. And it’s easy to avoid a congestion, because we can dynamically swerve a line if quality of this connection is decreasing.

And that’s what it is: We have a tunnel between data centre and branch office. And the tunnel intelligently and dynamically selects the best matching connection for a certain kind of traffic. We can hardly predict where a packet would flow. Its policy based, so we can assign certain requirements about quality for each kind of traffic.

Asymmetric connections? What about our firewall?

It’s all based on UDP. There is no connection on layer 4. Our Sessions are layer 7 only, and a firewall is a L3/4 device. The only thing you have to do: allow UDP 4980 originating from all branches to traverse your firewall!

Which connection is the best one?

It depends (this is the universal answer for every architect to all kind of questions).

Think about ICA? It’s the least latency one. User experience is very sensitive about latency. Jitter? We don’t care much about it, as long as we can keep latency below a certain value.

SIP (VOIP telephony)? Latency is not a big issue. Latency above a certain amount will make our phone calls less interactive, but everything below 200ms would be fine. Instead its jitter we have to care about. Jitter would distort spoken words, so they are harder to understand. And we are concerned about packet loss of course, as packet loss has an even stronger impact on understandability than jitter.

TCP based Videos? Well, neither of them is a problem, as we usually buffer videos. Even packet loss is not a big problem, as long as it doesn’t exceed a certain value. Instead its mere bandwidth. Videos nowadays fill up our costly WAN links causing congestion and packet loss.

File transfer? During browsing of directories, latency clearly is an issue, but this is not true about up- or download of files. WanScalers always had their method to deal with this latency issues, and it didn’t disappear in the enterprise or WO version. Again, it’s just bandwidth. However we would prioritize it a bit higher than video as less speed immediately impacts user experience.

So every type of connection has its own, very special, requirements. Pure prioritization won’t be sufficient.

All our Connections are pooled into the so called SD-WAN network connection. Our NetScaler SD-WAN Box will continuously monitor all possible connections and select the one best matching for every TCP stream, even for every single TCP packet. And even more: We have no need for symmetry. We could send an ICA packet on MPLS while we get the acknowledge packet over the internet! So asymmetric connections (like ADSL or cable TV) are an issue no more.

I did a short survey with my costumers: SIP traffic is not increasing very much. ICA traffic is increasing slowly while bulk traffic, mainly HTTP and videos, is exploding. Gartner calculates with annual growth of 15% continuing until 2019. So why upgrade our expensive MPLS connections and not use cheap internet instead?

How often do we measure latency?

At least every 50 ms on an idle connection. SD-WAN protocol sends measurement information with each and every packet it transmits. So the more busy your connection is, the more measurement data we transmit.

Security

Sending data over the internet is a risky thing. Citrix NetScaler SD-WAN uses 256 BIT AES IPSec. No data will traverse the internet in an unencrypted way. The crypto library supports NSA Suite B, I think, this is sufficient.

There is another aspect. If I want to hack into a connection, I need to pick up every single packet of this very data stream. I face dramatically more overhead if a single packet is missing. How can I collect all packets, if packets are distributed randomly over several very different connections, provided by different providers? This would even be challenging for these intelligence services all over the world! How could they ever reassemble a TCP stream? In my opinion this is a strong plus on security, even a plus over MPLS.

Resilience

We continuously measure the quality of a connection. If we see any parameter changing (latency, jitter, congestion, bandwidth) we will immediately change our assessment. Internet can have both, less latency and less jitter than MPLS. But it might change within milliseconds. It’ important to react. SD-WAN does immediately!

SIP (VOIP) is very sensitive to both, packet loss and jitter. At the same time it’s not a top bandwidth waster. We could therefore easily duplicate data over two or more lines and send it simultaneously. The data arriving first will be forwarded to the user, the other packets will be discarded. This would reduce the risk of packet loss and, at the same time, reduce both jitter and latency. This is an other great feature of Citrix NetScaler SD-WAN, and turned on by default for SIP. Isn’t it a great chance for our VOIP calls?

There is a great Video on YouTube about resilence. It’s a marketing video, taken at Synergy 2016, but I like it, as it truly shows how it works.

More chances to safe costs

We currently use MPLS because it is secure, available and offers guaranteed SLAs. Using Citrix NetScaler SD-WAN we don’t need to care about security as it’s built into SD-WAN. However SLAs of internet connections are not on top. What about using three internet connections at the same time. Three connections using very different technology like a GSM based, a cable based and a DSL based connection, instead of MPLS? This would offer SLAs above MPLS and, at the same time, be by far less costly? I think, evaluating this would be worthwhile!

Links

I have spoken to guys from Danish government evaluating Citrix NetScaler SD-WAN, some weeks ago. They are very interested in SD-WAN, mainly as they already have a department using Citrix NetScaler SD-WAN: Danish AgriFish. It’s all about costs and random disruption of ICA connections, and it works perfectly well. AgriFish is enthusiastic about it, other governmental authorities will follow. I link this (Citrix) success story here. I’m sceptic about success stories (never trust statistics you didn’t fake yourselves, but these guys I met face to face could prove this AgriFish one, so I can trust in it!

Why would you like to customize a 404 page?

Well It’s all about misleading information. A hacker has very limited chance to get friend with your web server. On the other way, he needs to find out as much as any possible. The more he knows, the more likely his attack will be successful. On the other hand he has to let sleeping dogs lie. With other words: He must not alarm you.

One of the most important things to know is: What kind of web server do I have to deal with?

The first source to look into is a HTTP response header called Server. Information here may be very verbose. I don’t know why this header is part of HTML standard, but actually it is.

The Server response-header field contains information about the software used by the origin server to handle the request. The field can contain multiple product tokens (section 3.8) and comments identifying the server and any significant subproducts. The product tokens are listed in order of their significance for identifying the application. (RFC 2616)

This is an example server header:

Apache/1.3.28 (Unix) mod_ssl/2.8.15 OpenSSL/0.9.7c mod_perl/1.27 PHP/4.3

In this case, it’s a very outdated Apache, using an outdated SSL module, outdated Perl and outdated PHP. It’s easy to change this information using Citrix NetScaler rewrite policies (DELETE_HTTP_HEADER and INSERT_HTTP_HEADER).

But hackes are not that stupid. They will probably verify this information. My personal next try would be: check for a non existing site. We will see a 404, page not found. Being careful I would use an existing URL, however do a minor typo, just like http://norz.at/default.html instead of http://norz.at/default.htm. You would probably not be scared if you would see a request like that watching your logs.

The next thing he would see is a 404, Not Found. It will be specific to your server, if you don’t change it. And a 404 page originating from an IIS6  would, for sure, come from an IIS 6, no matter what the server header tells you.

More reasons to change the 404 page

of course there are even more reasons to change the 404 page: customized 404 pages seem to be funny, they may help people to find the content needed, and so on.

Why not change your web server?

This would be possible. However, you would need to change all your load balanced web servers. There is an other reason: Responder policies. I will never return a “401 Unauthorized” or “403 forbidden“. I would rather return a “404 not found”. Being a hacker I would be very excited to see a 401 or 403!

I would think: here it is, but someone protects it from being accessed. But how could I find out what’s going on, if a Citrix NetScaler uses exactly the same 404 page as the original web server? I would probably think the file is not there.

My solution

My first attempt was creating a simple rewriting policy changing the body with something like “HTTP/1.1 404 OK\n\r\n\r<html><head><title>404 File not found</title></tead><body><h1><font color=\”#802020\”>404 File not found!</font></h1><p><font color=\”#802020\”>The file you requested is not on this server.</font></p></body></html>” in it.

The length of the text is limited, so this is not a good solution. And I would rather like to place the file “somewhere” on my web server, so it’s pretty easy to change.

I spent some time thinking what to do and made up my mind to use the HTTP callout feature. It was my first ever attempt to use HTTP callout, and I’ll describe how it works.

NetScaler’s HTTP callout feature

HTTP callout is intended to be used in policies to check something, i.e. an IP address, against a web based service. So I could send an IP address (CLIENT.IP.SRC) to a web server containing an IP black list. This web server then would respond with something indicating good or bad.

I do something completely different: I will retrieve the content of the 404 page from a web server. To do so I have to navigate to App Expert -> HTTP Callouts.

Like any policy it has to get a name. I do my callout to a vServer, so I have to specify the server here. My request will be attribut based, that means, I will be able to send regular HTTP requests, mine is a HTTP GET. My web server uses several host names for various virtual pages, so I have to specify a proper host expression. This makes sure, we retrieve the file from the right source. The URL Stem Expression is the URL we want to retrieve.

We scroll down to the bottom and select the return type TEXT and the expression should be HTTP.RES.BODY(65538). The number is the number of bytes to retrieve.

So, my policy will connect to a NetScaler vServer called cs_vsrv_norz.at to retrieve a file called /notfound.htm, setting the header Host to norz.at (i.e: http://norz.at/notfound.htm). It will then return all the body of this file, containing links to style definitions, pictures and so on.

command line version:

add policy httpCallout callout_retrieve_404 -vServer cs_vsrv_norz.at -returnType TEXT -hostExpr "\"norz.at\"" -urlStemExpr "\"/notfound.htm\"" -scheme http -resultExpr "HTTP.RES.BODY(65538)"
set policy httpCallout callout_retrieve_404 -vServer cs_vsrv_norz.at -returnType TEXT -hostExpr "\"norz.at\"" -urlStemExpr "\"/notfound.htm\"" -scheme http -resultExpr "HTTP.RES.BODY(65538)"

The rewrite policy

The rewrite policy should be a very simple thing:

The NetScaler rewrite action using a HTTP callout

add rewrite action callout404 replace_http_res "SYS.HTTP_CALLOUT(callout_retrieve_404 )"

It’s a replace policy. Expression to choose target location is all of the HTML body, so HTTP.RES.BODY (65536). To be more precise, it’s the first 65536 byte of the body (a 404 page typically is by far smaller). The Expression is the text we will use to replace the former body with. It is the HTTP callout request, in my case SYS.HTTP_CALLOUT(callout_retrieve_404).

The NetScaler rewrite policy

add rewrite policy rw_pol_404 "HTTP.RES.STATUS.EQ(404)" rw_act_404

This policy will get applied if  the HTTP response status is a 404 (HTTP.RES.STATUS.EQ(404)). I then bound this policy to my web server. That’s it. It was pretty easy.

I was so enthusiastic, when I found out about NetScaler admin partitions! What a great extension to existing NetScalers! However I got disillusioned finding out about limitations. It took me some time to find out how to overcome this issues, but there are still some features missing.

The feature I missed most is doing traces. It’s not listed in the compatibility list, so it’s intended to be there. But it is not! If you click into System and Diagnostics you’ll see just very little content, and definitely no nstrace (this is about NetScaler versions up to 11.1 48.10).

Citrix documentation is always right, and if it’s not, it’s right even though. So I tried to do a nstrace from commandline. It started and stopped without any problem. Unfortunately I could not find the output of my nstrace in /var/nstrace subdirectory.

So I searched for it, and found it in /var/partitions/<partitionname>/nstrace.

So that’s how I do an nstrace inside a NetScaler admin partition:

I use putty to connect to my NetScaler. Masochists might prefer to use the built in terminal from GUI, however I don’t tend to masochism.

switch partition <partitionname>
start nstrace -filter “CONNECTION.SRCIP.EQ(<source IP>)” -size 0 -time 3600 -link
stop nstrace

So I log into my Citrix NetScaler. I change into my partition (currently partition names can’t auto complete, so make sure you know the name; show ns partition will list all partitions.

Next I start the trace. To do so I follow CTX120941.

start ns trace will simply start the trace. -filter will filter a connection. Usually you would use connection objects like CONNECTION.SRCIP.EQ(<source IP>) or CONNECTION.DSTIP.EQ(<destination IP>) to limit the amount of data captured. -size=<size> will limit the amount of data captured per packet. If you want to debug HTTP problems you would very likely set the size to 0, as this would capture all of the packet (0 sometimes means unlimited). -time=<time> will automatically stop the trace after <time> seconds.

After doing your trace you may execute a stop nstrace command to stop your trace. This is not needed if you set the time parameter, but I prefer to stop traces instead of setting a time parameter.

In the end you need to download the trace file. I usually use winscp as a secure FTP client, download it from /var/partitions/<partitionname>/nstrace and view content in your favourite network monitor. I prefer to use WireShark, as it fully supports NetScaler. Citrix support also uses WireShark.

Additional parameters for tracing

-tcpdump ENABLED switches to TCPdump format. TCPdump is a standard UNIX® format for network tracing. Different to NStrace it does not contain L1 information (ports), but it is understood with most network tracing utilities. You may want to use it together with -perNIC ENABLED if you want to debug routing problems. This will create a separate trace file pre NIC. You then have to scroll down both instances of your network monitor in parallel (and synchronize these 2 windows if you scroll down). However you may prefer to download free WireShark and use it instead as it understands NStrace: one window, all L1 information is contained in your trace.

–link Also trace filtered connection’s peer traffic. Only makes sense in combination with -filter. It will trace all traffic filtered plus all traffic resulting from your filtered traffic, so traffic from client to VIP and traffic from SNIP to your back end server. This is a very good one!

-mode SSLplain will decrypt all SSL traffic. Because of this you won’t see any SSL hand shake, instead, all SSL traffic will appear to be plain text. This may be beneficial if you want to debug encrypted traffic. Caution: this may expose sensitive data to you (the admin)

There are several more parameters. You may find them in Citrix NetScaler product documentation.

Let’s talk about a WAF, a Web Application Firewall on a Citrix NetScaler. What’s to be concerned off? Is it worth while considering a NetScaler to be your WAF?

I do work for several companies, including Citrix Consulting Services. Recently I worked on some Web Application Firewall projects, so I have some experience on it.

Usual concerns

1. will a Citrix NetScaler be really safe WAF?
2. How well does it scale?
3. Is it easy to implement?

1: How safe is a Citrix NetScaler Web Application Firewall (WAF)

As far as I know one of the biggest websites world wide is using NetScaler WAF. They are storing hundreds of millions of customer records (including billing and credit card information). As far as we know, they never got hacked so far. Their website seems to be safe. Same about a huge NGO with is political exposed very much. They are attractive to hackers from all over the world. They also still are not known to be hacked during the last some years.

I also know of banks trusting in Citrix NetScaler Web Application Firewall (WAF), they are successful.

Nicht zuletzt gibt es mehrere Zertifizierungen, die unsere NetScaler WAF derzeit durchführt: NSS-Labs empfiehlt NetScaler WAF genauso wie ICSA-Labs.

So I consider NetScaler WAF to be secure, if it’s set up correctly.

2: How well will it scale?

Well, that’s a problem indeed. And it depends (every architect’s standard answer to each and every question). To be honest, a WAF is overhead. Huge overhead. Every single packet, flowing in (and flowing out in many cases), has to get inspected. So WAF has to be considdered a burden for the CPU of a NetScaler.

Like every feature on NetScaler, WAF is not multithreaded, meaning: Every Packet Processing Engine (PPE) is processing a TCP packet flow, independently from all other PPEs. And does everything on it’s own, not calling a singe operating system funchtion. There is just one thread, picking up the packet, doing all policies (responder, rewriting, WAF, …) and forwarding it to it’s destination. This feature is great, as it makes a NetScaler a very stable box, but it may cause some CPU-cores to be overloaded for a relatively long time. Overload on a CPU-core means latency for a user. To avoid overload, average CPU has to be under 75-80 %.

So, if you go NetScaler WAF, you’ll have to be able to scale out. Scaling out may mean, to upgrade your box with bigger licenses. Bigger licenses may mean: unlocking CPU cores and RAM. But it may also mean: Add more NetScaler boxes. Adding more NetScaler boxes seems to mean: Cluster. But as I personally would avoid a Cluster, I’d rather load-balance NetScalers. So a typical WAF-deployment would look like this:

A pair of NetScaler VPX (or SDX, MPX) boxes (in HA, tier 1) load balancing NetScaler MPX boxes (tier 2). These MPX boxes do both SSL-on- / off-loading and WAF. HA is not needed.

The tier 1 HA pair is just a load balancing vServer of type SSL bridge, using SSL session ID for persistence. The vServer is in source IP mode (SIP-mode), to preserve IP addresses for tier 2

This setup scales up easily and – at the same time – avoids cluster typical problems like features not being available or being hardly tested. We don’t use HA as load balancing in tier 1 takes care of high availability. We may scale out easily, if performance is insufficuient. We may even upgrade these boxes independent from each other. Of course we need one additional box, in case of one of these boxes is going down (n+1 principle)

3: Is it easy to implement?

Citrix sales (and some consultants) tend to answer this questions with a clear and simple yes, as NetScaler comes with an integrated learning feature doing all stuff for you. That’s really great!

Me, being rather a consultant than a sales guy – however – would rather say no. Being a customer I’d absolutely like to have a consultant with long-term experience working on this project.

One of the biggest problems in security is a false feeling of safety. A WAF will always give you a sound feeling of security. Feeling secure, makes people careless. But bever forget: What if there is something wrong about your WAF?

You got it: Your feeling of security may be as same as wrong, as your WAF setup.

All of us are always a bit shy looking at Citrix Synergy: What will it bring? Well, this time, Citrix comes up with brand new names for all products. It’s the first time Citrix is renaming the product. Until now the mane resisted all renaming by marketing departement.

Citrix aquired NetScaler back in 2005. The original company “NetScaler” was founded by Michel K Susai in 1997.

It’s nothing less but the biggest name change of all time in Citrix history.

I currently don’t know the upcoming name for Citrix Receiver (was Receiver, Plugin for Hosted Applications, Plugin, Citrix Client and many, many more)

So which names will we have to deal with in future?

There is a white paper about name changes. All products will get renamed. There are three fields of products:

• Citrix Workspace

  • Citrix Content Collaboration (was ShareFile)
  • Citrix Entpoint Management (was XenMobile)
  • Citrix Secure Browsing (was XenApp secure Browser)
  • Citrix Hypervisor (was XenServer)
  • Citrix App Layering
  • Citrix Virtual Apps (was XenApp)
  • Citrix Virtual Desktops (was XenDesktop)
  • Citrix Endpoint Management
    • Citrix Secure Mail
    • Citrix Secure Web

• Citrix Networking

  • Citrix ADC (was NetScaler ADC – “NetScaler“)
  • Citrix SD-WAN (was WanScaeler, Cloudbridge, SD-WAN, NetScaler SD-WAN)
  • Citrix Web App Firewall (was NetScaler App Firewall, NetScaler App Security)
  • Citrix Gateway (was NetScaler Unified Gateway)
  • Citrix Application Delivery Management (was NetScaler MAS, MAS, NMAS)
  • Citrix Secure Web Gateway (was NetScaler Secure Web Gateway)
  • Citrix Intelligent Traffic Management (Cedexis Plattform)

• Citrix Analytics

  • Citrix Analytics for Networking
  • Citrix Analytics for Workspaces

So we see: Everything is less difficult than it was before. We will clearly understand each other during our conversations through out the next some years. We have to be happy!

If you read about slowloris, you always read about NetScaler doing a great job. Tests in our lab environment show: NetScaler will successfully block these attacks. Ad there is hardly anything we have to do about it: it’s built into the system. Great news indeed!

The only thing we have to do is reduce client idle timeout to a lower value (default 180 seconds). I’d propose something below 20 seconds.

Unfortunately NetScaler will not log these attacks. WTF? Yes, that’s true. NetScaler won’t log a blocked slowloris attack. I recently set up a Citrix NetScaler WAF in the lab environment of a big bank, and they wanted me to log these Slowloris. I understand very well, why they want to log these. However we can’t.

This blog article could be over right now, but I did some research. Of course we have counters for this kind of attack. And we expose many of them to NITRO. So it could be possible, to do logging based on NITRO calls. Lets dig a little bit into NITRO. To do so I open my browser and surf to https://SNIP/nitro/v1/stat/protocolhttp (SNIP is the subnet address of my NetScaler). After logging on it returns a JSON list of counters. Most of them are of no importance for us, but I’m interested in httperrincompleteheaders.

That’s a good point to start from!

How to log httperrincompleteheaders on Citrix NetScaler ADC

So my approach would be like this:

• do a query to NITRO API and store the results in a NetScaler variable.
• log, if this counter increases.

There are some obstacles to overcome.

First of all, we need to find a way to query NetScaler NITRO from within a NetScaler. And here it is:

define a HTTP Callout.

We may use an HTTP callout to query Nitro. That’s quite simple to do, it’s just an http get. Problem here is authentication. I found a solution, unfortunately it’s not a very elegant one.

add policy httpCallout query_incomplete_header -IPAddress 192.168.30.110 -port 80 -returnType NUM -urlStemExpr "\"/nitro/v1/stat/protocolhttp\"" -headers X-NITRO-USER("nsroot") X-NITRO-PASS("nsroot") Accept("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8") host("192.168.30.110") -scheme http -resultExpr q/HTTP.RES.BODY(20480).AFTER_STR("httperrincompleteheaders\": \"").BEFORE_STR("\"").TYPECAST_NUM_AT/

In GUI, callouts are located in AppExpert

What’s in this policy?

• -IPAddress: This is the IP address we’re actually calling. It has to be a SNIP, HTTP access enabled.
• -Port: The port, usually 80 (SSL doesn’t make any sense for NetScaler internal communication; it’s waste of ressources)
• -returnType: The type of data this callout has to return. Possible values: TEXT, NUM or BOOL. We’re interested in numbers.
• -urlStemExpr: The URL we call (/nitro/v1/stat/protocolhttp)
• -headers: headers we have to set. They are nescessary, see below
  • X-NITRO-USER(“nsroot”) username
  • X-NITRO-PASS(“nsroot”) password for this user
  • Accept(“text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8”) the encoding, our policy can understand
  • host(“192.168.30.110”) server’s hostname. Don’t skip this one!!!
• -scheme: htt/https
• -resultExprthe: data we’re interested in

This query should return a number: The number of requests containing incomplete headers. So we have to take a closer look at the http response in RAW format:

{ "errorcode": 0, "message": "Done", "severity": "NONE", "protocolhttp": { "spdytotstreams": "0", "spdystreamsrate": 0, "httptotrequests": "13211", "httprequestsrate": 3, "httptotresponses": "13209", "httpresponsesrate": 3, "httptotrxrequestbytes": "524189", "httprxrequestbytesrate": 84, "httptotrxresponsebytes": "2498358", "httprxresponsebytesrate": 1004, "httptotgets": "1244", "httpgetsrate": 0, "httptotposts": "12", "httppostsrate": 0, "httptotothers": "11955", "httpothersrate": 2, "httptot10requests": "0", "http10requestsrate": 0, "httptot11requests": "2102", "http11requestsrate": 0, "httptotclenrequests": "20", "httpclenrequestsrate": 0, "httptotchunkedrequests": "0", "httpchunkedrequestsrate": 0, "httptottxrequestbytes": "0", "httptxrequestbytesrate": 0, "httptot10responses": "4166", "http10responsesrate": 1, "httptot11responses": "9043", "http11responsesrate": 2, "httptotclenresponses": "8185", "httpclenresponsesrate": 2, "httptotchunkedresponses": "1", "httpchunkedresponsesrate": 0, "httperrnoreusemultipart": "0", "httperrnoreusemultipartrate": 0, "httptotnoclenchunkresponses": "4166", "httpnoclenchunkresponsesrate": 1, "httptottxresponsebytes": "0", "httptxresponsebytesrate": 0, "httperrincompleteheaders": "0", "httperrincompleterequests": "0", "httperrincompleterequestsrate": 0, "httperrincompleteresponses": "0", "httperrincompleteresponsesrate": 0, "httperrserverbusy": "4", "httperrserverbusyrate": 0, "httperrlargecontent": "0", "httperrlargechunk": "0", "httperrlargectlen": "0", "spdyv2totstreams": "0", "spdyv2streamsrate": 0, "spdyv3totstreams": "0", "spdyv3streamsrate": 0 } }

We have to extract data behind “httperrincompleteheaders”: and before the next “, and convert this to a number. A possible string would be:

HTTP.RES.BODY(20480).AFTER_STR(“httperrincompleteheaders\”: \””).BEFORE_STR(“\””).TYPECAST_NUM_AT

HTTP Callout is finnished!

Next we need a NetScaler variable.

Creating a Variable to store our data in

Citrix NetScalers have built in functionality called variables. They can be created either from GUI or command line.

add ns variable HTTP_INCOMPLETE_HEADERS -type ulong

In GUI, variables are located in AppExpert
Assigning data to this variable

It’s surprisingly difficult to assign data to a variable! A simple $variable=7 won’t do the job. Instead we have to create an assignment. Luckily assignments are – at the same time – policy expressions for responder and rewriting policies. So we’ll have to create an assignment for this variable:

add ns assignment set_incomplete_header -variable "$HTTP_INCOMPLETE_HEADERS" -set "SYS.HTTP_CALLOUT(query_number_req)"

In GUI assignments are located in Appexpert

This assignment will replace the value in my variable defined above with the return value of the callout created first.

The trigger

We could check for slowloris when ever a request comes in, however this may be way too often, as NITRO calls are of some overhead for our system. So I created a trigger. I won’t do a step by step instruction for this here, I will just give you an idea how it works:

I create a Citrix NetScaler service of type ANY, pointing “somewhere”. (it does not matter as we won’t use this service at all). I assign a health monitor of type HTTP to it, specifying the IP the HTTP vServer (CS or LB does not matter). This health monitor periodically sends HTTP requests there, they are easy to identify and we are free to define the frequency we like. This health monitor is our trigger.

The policies we will use

Next to do is creating policies. We may use responder or rewriting policies (I used responder policies, but it does not matter). This policy will do the callout and compare it to the stored value in our Citrix NetScaler variable. We would need to bind a logging policy to it (see here). Unfortunately this is not possible. It’s a restriction in NetScaler: “Log action is not suported with assignment action“. So we have to do two identic policies, one doing nothing but logging, the other incrementing the counter.

Logging policy

add responder policy log_IncompleteRequests "CLIENT.IP.SRC.EQ(192.168.30.110) && $HTTP_INCOMPLETE_HEADERS.EQ(SYS.HTTP_CALLOUT(query_incomplete_header)).NOT " NOOP -logAction Log_new_number rule "CLIENT.IP.SRC.EQ(192.168.30.110) && $HTTP_INCOMPLETE_HEADERS.EQ(SYS.HTTP_CALLOUT(query_incomplete_header)).NOT "
This Citrix NetScaler responder policy checks if client’s IP is the SNIP and the incomplete header counter increased. If so the responder policy actually does nothing (NOOP), but it logs.

incrementing policy

add responder policy count_IncompleteRequests "CLIENT.IP.SRC.EQ(192.168.30.110) && $HTTP_INCOMPLETE_HEADERS.EQ(SYS.HTTP_CALLOUT(query_incomplete_header)).NOT" set_incomplete_header rule "CLIENT.IP.SRC.EQ(192.168.30.110) && $HTTP_INCOMPLETE_HEADERS.EQ(SYS.HTTP_CALLOUT(query_incomplete_header)).NOT"

This policy can’t log (see above), however set the counter. Policy expression is the same as above.

Binding the policies

The last task is binding these policies. Simple like that. Bind the logging policy first with “goto next”, bind the incrementing policy (with “goto next”, if you still have more policies to check for).

I hope you liked my tricks. I’d be happy to hear your thoughts on this, just drop some words in the comment box to let me know about your thoughts. Feel free to link to my page when ever you like.

Johannes

How to protect your cookies using Citrix NetScaler

I recently did a web application firewall (WAF) project for a big company owning and hosting hundreds of websites. They did several penetration tests. One of them focussed on cookies. Citrix NetScaler did a great job protecting cookies, cookie tampering was impossible, but they had been able to steal cookies.

Stealing cookies is not that easy, especially if a website is well protected and XSS (cross site scripting) is blocked it is near to impossible. It would be easy stealing cookies using XSS: post document.cookies to a website of attacker’s choice, that’s it. But it’s also easy in a lab environment: Just pick them up, copy them to an other box and feed them back in. This is, what they did. And NetScaler failed (actually no surprise to me).

How does Cookie protection work?

We have several methods. In most cases, we need sessionization. Citrix NetScaler will store information about cookies (hashes) and will drop tampered cookies. We also could encrypt cookies. This would make cookie tampering hard for an attacker, as he has to guess (brut force) the key. We could cache (session-) cookies on Citrix NetScaler. In addition we could mark cookies HttpOnly or Secure. All these methods target cookie tampering, not cookie stealing.

A Cookie-Monster stealing cookies? What can we do?

Sure, NetScaler can’t do anything. NetScaler will add a session cookie to the existing cookies (change the default name from citrix_ns_id to something else, an attacker does not nescessarily need to know about Citrix NetScaler ADC protecting our website). A NetScaler will make 100% sure none can tamper cookies. But it will definitely allow stealing of cookies.

The solution

We need to put additional cookies into the data stream identifying the client. We need to find something specific.

The IP address

Unfortunately the IP address is not half as specific as people think. Mobile phone networks are typically NATing their users to the internet. It’s rather likely for us to share the same IP if we share the same mobile phone provider.  An other drawback of the IP is: it may change while users move from one network to an other, let’s say move from home (there is Wi-Fi connected to DSL) to the street (using LTE coming from a totally different provider).

Even though there are some draw-backs: The IP address is a good thing to use. It would be easy for an attacker to fake it, if he only knew. So we have to keep it a secret and encrypt it.

User-Agent

There is a wide variation in User-Agent strings being sent from client to server. Clients differ in language, browser type and version, operating system and many more. It would be easy for an attacker to fake this string, if he only knew. So we have to keep it a secret and encrypt it.

More things?

sure, what ever you have, use it. My customer uses several things in parallel.

Implementing the solution

General thoughts?

We will add cookies, so we need names for these. I’m a great fan of cheating. The more you cheat the less likely an attacker would understand your setup. So I’ll call my example cookies Tmp-Data and Default-Printer.

Creating these cookies

I create two rewriting policies in response direction:

Citrix NetScaler Policy Actions:

add rewrite action rw_act_setCookie_IP insert_http_header Set-Cookie "\"Default-Printer=\" + CLIENT.IP.SRC.TYPECAST_TEXT_T.ENCRYPT"

This policy action extracts client’s IP from HTTP request, converts it into text, and encrypts it.

add rewrite action rw_act_setCookie_User-Agent insert_http_header Set-Cookie "\"Tmp-Data=\" + HTTP.REQ.HEADER(\"User-Agent\").ENCRYPT"

This policy action extracts the User-Agent string from the original HTTP request and encrypts it.

Citrix NetScaler Policies

add rewrite policy rw_pol_setCookie_IP true rw_act_setCookie_IP
add rewrite policy rw_pol_setCookie_User-Agent true rw_act_setCookie_User-Agent

We are using true as a policy condition because we want this to be in done every request.

Binding these Policies

Just bind these policies to a vServer of choice. No matter if it’s a cs vServer, or a lb vServer.

Checking incoming traffic

General thoughts?

Citrix NetScaler WAF will protect all cookies, including the ones we created, from being tampered. So we don’t have to worry about these cookies being tampered. But what if a request comes in, not containing these cookies? That’s more than possible: Every user session starts with a request not containing cookies. So we must allow requests without pre-existing cookies, or we could strip all cookies from an initial request. We must not allow requests containing all application cookies, but not our ones. If you are dealing with an existing website: There may already be persistent cookies stored on a client device. Persistent cookies usually don’t contain sensitive information. They store things like settings rather than the user’s identity.

What kind of policy will we use to check for cookies? There are two possible answer. My first one would be: Responder. Just drop requests, or redirect them to a safe location. Drawback of this is, it’s not all done at the same place. And responder policies usually don’t log (we may force them to log). So why not use Application Firewall policies instead? There are three built in profiles: APPFW_BYPASS, APPFW_RESET, APPFW_DROP, APPFW_BLOCK (for details see here). I will use APPFW_DROP to drop silently.

The Citrix NetScaler Application Firewall policy:

the policy dropping requests containing stolen cookies

add appfw policy appfw_pol_drop_wrongcookie "HTTP.REQ.COOKIE.VALUE(\"Default-Printer\").EQ(\"\").NOT && (HTTP.REQ.COOKIE.VALUE(\"Default-Printer\").DECRYPT.EQ(CLIENT.IP.SRC.TYPECAST_TEXT_T).NOT || HTTP.REQ.COOKIE.VALUE(\"Tmp-Data\").DECRYPT.EQ(HTTP.REQ.HEADER(\"User-Agent\")).NOT)" APPFW_DROP

This policy will be triggered if cookie “Default-Printer” is not empty and either “Default-printer” does not match the IP or “Temp-Data” does not match User-Agent.

the policy dropping requests with missing cookies

add appfw policy appfw_pol_drop_missingcookie "(HTTP.REQ.COOKIE.VALUE(\"Default-Printe\").EQ(\"\") || HTTP.REQ.COOKIE.VALUE(\"Default-Printe\").EQ(\"\")) && (HTTP.REQ.COOKIE.VALUE(\"<the session cookie of your application goes here>\").EQ(\"\").NOT" APPFW_DROP

This policy drops requests not containing “Default-Printer” cookie or “Temp-Data” cooke and at the same time contains your application’s session cookie.

I hope you liked my tricks. I’d be happy to hear your thoughts on this, just drop some words in the comment box to let me know about your thoughts. Feel free to link to my page when ever you like.

Johannes

Recently I had several requests related to NITRO. NITRO is Citrix NetScaler’s API. Any device may communicate to a NetScaler using NITRO. Even a browser! Citrix exposes several settings and counters and even allows changes. NITRO is the central source for scripting NetScalers.

I, being rather an administrator than a programmer, am not that much interested in using NITRO with C++/C#, Java, …, instead I have an administrator’s view on it. My first steps with NITRO had been around retrieving and logging counters. I wrote a blog about logging slowloris attacks. This was NITRO calls from within a NetScaler.

This one is about NITRO in general.

NITRO in a nutshell

Citrix NetScaler’s NITRO is an API. Even though it’s partly possible to be useed it with standard HTTP, it’s based on REST and JSON. That means: Requests and responses follow a structured, XML like, format. That’s fine from one side, but turned out to be a bit challenging for me.

What does it expose?

There is a good SDK available. NITRO exposes:

• Configurations. http(s)://<netscaler-ip-address>/nitro/v1/config/<resource-type>
• Statistics. http(s)://<netscaler-ip-address>/nitro/v1/stat/<resource-type>

So it’s possible to get read/write access to Citrix NetScaler’s configuration as well as read access to NetScaler statistics!

The Nike^® way: Let’s do IT (or: a first try)

My Citrix NetScaler ADC Testsystem’s NSIP is 192.168.30.100. So I surf to http://192.168.30.100/nitro/v1/stat/.

I get promptet for username and password. An easy one for all of you as my testsystem uses nsroot/nsroot . If you want to logon using a script you would follow these guide lines in Citrix NetScaler NITRO SDK.

Immediately after sending my credentials our first success:

Or, if I select RAW data in Firefox:

This is a complete list of objects Citrix NetScaler NITRO exposes counters for.

Next step: Retrieve counters for a specific class:

Let’s say: I’m currently mainly interested in WAF, so the counters I’ll try to extract is appfw. My URL would be http://192.168.30.100/nitro/v1/stat/appfw. And here they are! Again, this is a great JSON file and can be processed easily. This can even be done from inside Citrix NetScaler using http callout.

Let’s go into the config side!

So my first guess would be: surf to http://192.168.30.100/nitro/v1/config/. And it works!

View Citrix NetScaler Firmware version:

http://192.168.30.100/nitro/v1/config/nsversion

Get Citrix NetScaler basic configuration

http://192.168.30.100/nitro/v1/config/nsconfig

This is some information like NS-IP, cookie version, HA status, time zone, last config update / save, system time and more.

I see, I would have to save my “valuable” configuration. This would be possible using http://192.168.30.100/nitro/v1/config/nsconfig?action=save, however it does not work. Why? Because I send a http get instead of a put. See here for details. I could use fiddler to change my get into a put (putting the right content into the body), but that’s way to complex for me to do, so I left my configuration unsaved.

(just kidding, of course I did! It’s not that easy, but in the end I made it, that’s why I’m still here, some minutes before midnight)

Nitro: Which lb-vServers are on my Citrix NetScaler?

http://192.168.30.100/nitro/v1/config/lbvserver (http://192.168.30.100/nitro/v1/config/csvserver)

(I have collapsed several servers, so you can see there are several of them)

This will retrieve a list of all vServers out of Citrix NetScaler using a Nitro call (I have collapsed several servers, so you can see there are several of them)

http://192.168.30.100/nitro/v1/config/lbvserver/lb_vsrv_colors

Nitro: give me a list of SSL certificates?

http://192.168.30.100/nitro/v1/config/sslcertkey

A complete list. Simmilar to lb-vServer a specific certificate would be: http://192.168.30.100/nitro/v1/config/sslcertkey/ns-server-certificate.

There is much more to query for. NetScaler GUI constantly does NITRO queries to Citrix NetScaler. Using a proxy like fiddler could help you seeing these calls.

I hope, you liked my blog. It’s fun to play around, dig into NetScaler and I really had fun to see my NetScaler from a programmer’s perspective. I’d be more than just happy to see comments on this article. Your comments keep me writing more blogs …

This will be my shortest blog about the subject ever. Citrix finally did it! They created a “Built-in secure front-end SSL profile” called ns_default_ssl_profile_secure_frontend.

What do you need to do? Just bind this profile to your vServer. That’s it. Isn’t it great? Compare this to my last blog about the subject!

That’s the way it should be done. Hey, Citrix ADC guys, you’re doing great!