Monday, September 27, 2010

A clustered Samba Fileserver, Part I - Presentation

I got, a few weeks ago, a request to create a Samba (CIFS) fileserver that would be used to serve a variety of files to hundreds of clients. There are a few requirements:
  • It should be high performance;
  • It should have some mechanisms to replicate to another storage;
  • It should be scalable both in terms of performance and available space.

I came up quite quickly with a solution: a cluster of fileservers, replicating block devices and presenting to Samba a clustered filesystem. That way, I can have two nodes writing to their respective devices at the same time, and have both changes immediately available to the other nodes.

On Linux systems, DRBD is the de facto block device replication solution, and GFS2 is a well supported clustered filesystem, OCFS being another one.

Finally upgraded my Mac Book Pro to Snow Leopard 10.6.3

Ok, not really a new one, but it took me time.

Installation

Small issue. A long time ago, I played with Linux on the Mac and I created a few new partitions. When I was done, I tried deleting them, with no luck. I then converted them to UFS and simply mounted them.

However, when I tried to upgrade to 10.6.3 from Leopard, the installer wouldn't let me, claiming that my startup disk was not a startup disk. Rebooting and trying the upgrade directly from the installation media didn't help either.

An attempt to kill the partition failed with MediaKit not able to delete it. It even claimed it there was no such partition.

At that point, my only choice would have been to replace the partition scheme and do a fresh install. So ..

Using an external USB drive, I used Time Machine to take a backup of everything: applications, parameters, files and so on. I then rebooted, selected the 1-partition scheme and reinstalled the whole OS.

First login: miracle! The installer asked me whether I wanted to restore a Time Machine backup! Of course I want. It did so and reloaded all my stuff. So cool! Later, I had to do a couple of adjustments: reinstall some applications that were 32-bits and not 64, install the new PGPMail, XCode and the Mac Ports, and recompile all my installed ports. As of now, everything works fine.

First step: upgrade

After reinstalling everything, I ran not one, but two sessions of upgrades. The combined total was around 1GB downloaded, where my install was around 5GB. That's kind of frustrating to see that in about a year, the whole OS will be completely replaced by patches and updates.

Tests

Perceptibly, Snow Leopard is snappier and the dock reacts faster than it did in 10.5. For computation intensive applications, it behaves the same and no CPU overhead is seen.


The only changes I saw was that my stacked applications now have a scroll bar, where the previous version scaled the icons instead. I don't have enough items in my Documents folder to tell if it behaves the same.



Conclusions

Snow Leopard installation is awesome and, while it took the largest part of my Sunday morning to do it, partially due to the Time Machine Backup, it went smoothly and without any hitch or bump. The possibility to restore a Time Machine Backup after the first boot is a nice and real cool functions and helped me in migrating my stuff. Should I have to do it again, I would use a Firewire disk rather than a USB one, as my dataset was around 43GB.

The OS doesn't feel different, it doesn't look different and it doesn't behave differently. Hugely appreciated when dealing with some people whose learning curve is almost horizontal ( :) ), and everything stayed at its place.

I haven't had the time yet to run all the tests I wanted, but I think I won't be disappointed.

Bibliography

XCode, http://developer.apple.com/technologies/xcode.html (Developer account needed)
MacPorts: http://www.macports.org/

Tuesday, September 21, 2010

It's a matter of PCI DSS ...

Recently, during my trip to the Grand Canyon, I booked a trip in helicopter, and I had to rent a tripod for my camera.OK, there were a lot more expenses than that, but these were actually the most surprising.

Why? Because for both, the clerk took my credit card, and wrote down all the details, including the 3-digits verification code! I was so shocked that I couldn't even speak: I just granted two parties to print money. From my account. Without virtually no possibility to dispute.

The PCI DSS mandates that:

3.2.2 Do not store the card-verification code or value (three-digit or four-digit number printed on the front or back of a payment card) used to verify card-not-present transactions.

So basically, a small shop in Page, AZ and a small hotel in Tusayan, AZ just gave themselves the right to ignore the very basic security measures to protect my information. If this was done inside their computer systems, they could be prohibited from issuing any payment request.


At this time, I'm monitoring my bank account, as I don't really know what happened to the piece of paper. But the lesson is learned: next time someone starts to write the CVV, I'll just cancel the transaction and ask for the note. Remember: even when on holidays, bad things can happen ...

Monday, September 20, 2010

Feature request submitted to Fedora

During my tests of nginx on fedora 13 (which is great!), I found it would be way easier to move the whole server{} section in its own file, rather than have a nginx.conf that's a mile long. Another driver is that that section can actually covers yours, and you'll scratch your head - like I did - to find why what should just pass a request from nginx to a back-end Apache is serving the "This is the default nginx page."

Let's hope the maintainers will agree :)

Practical Guide to Fedora and Red Hat Enterprise Linux, A (5th Edition)

Saturday, September 18, 2010

NGinx and Load balancing

Not long ago, I was asked to install a front end for some web applications. For high availability sake, it was decided to install two web servers, each accessing the same back-end database.

The choice was between a few commercial products, Foundry being the most known, and several opensource alternatives. I thus decided to play with one of them, Nginx.

I have already installed Apache and Squid as reverse proxies and front-end load balancers, but Nginx was still new for me. So why hesitate?

From a few articles I read on the Internet ([1], [2] and [3]) show that Nginx outperfoms Apache in several scenarios, one of them being a large number of concurrent connections. In that regard, in then makes sense to think of:

  • Nginx as a Front-end, eventually with a twin and using VRRP or clusterd to provide front-end high availability, and to eventually serve the static parts;
  • Several Apache back-ends with your favorite language: php, perl, ... or any other Tomcat, JBoss, Zope ...;
  • A way to centralize the database (More on this later).
The front-end can have multiple roles, from just acting as a reverse proxy between clients and back-end servers to also encrypt the traffic on-the-fly, compress, take decisions based on geoIP and so on. The sky is the limit!

My first test was an easy setup: a Nginx front-end and two back-end Apache servers. This was easily accomplished, with only a few directives:

In the http{} section, I declared my server farm:

upstream loadbalanced {
      server 192.168.1.71:80;
      server 192.168.1.72:80;
}
And in the server{} section, I declared that everything has to be sent to these upstream servers:

location / {
     proxy_pass http://loadbalanced;
}


This is the most basic setup and performs a round-robin selection between the two servers: each new connection is redirected to the next server. From here, let's try two things:

  1. Shut the Apache process on one of the back-end server
  2. Shut the complete server down

Scenario 1 is detected immediately, and Nginx forwards all requests to the second back-end. Mission accomplished!

Scenario 2, OK as well: Nginx tried for a bit, then switched to second machine. Again, if we except a 30 seconds wait time, no error was returned. This can be tuned at will, see [4]. In the same document, you will see the options to control the load balancing, to make sticky sessions and so on.

My second test was a wee bit more "complex": why waste precious CPU cycles on the application servers when the front-end can compress, encrypt and serve static contents such as pictures. This leaves plenty of CPU resources to execute all the scripts on the back-end servers.

So, objectives:

  1. The front-end compresses and encrypts the connection with the client;
  2. The connection between the back-ends and the front-end is in clear text;
  3. The front-end serves static content.
That's an easy job.

First, let's create a self-signed certificate:

openssl genrsa -out nginx-key.pem
openssl req -new -key nginx-key.pem -out nginx-req.pem
<Bunch of questions suppressed>
openssl x509 -req -signkey nginx-key.pem -in nginx-req.pem -out nginx-cert.pem
Next, let's configure Nginx for SSL. Most distributions have a default "ssl.conf" file in /etc/nginx/conf.d. In there, you can find most of the needed declarations.

#
# HTTPS server configuration
#

server {
    listen       443;
    server_name  _;

    ssl                  on;
    ssl_certificate      /etc/nginx/ssl/nginx-cert.pem;
    ssl_certificate_key  /etc/nginx/ssl/nginx-key.pem;

    ssl_session_timeout  5m;

    ssl_protocols  SSLv3 TLSv1;
    ssl_ciphers ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:-MEDIUM:-LOW:-SSLv2:+EXP;
    ssl_prefer_server_ciphers   on;

    location / {
    proxy_pass http://loadbalanced;
    }
}
No big mysteries there if you are a bit familiar with the Apache configuration. The ssl_protocols and ssl_ciphers declarations are in the openssl-like format. Again, I would strongly advise disabling SSLv2 as it has some weaknesses, and leaving only the "HIGH" encryption.

This alone gives me the encryption by the front-end. To compress, simply add

gzip  on;

within the server{} section.

The next and last part is to serve the static content from nginx itself. To make things easy, I isolated the images in /images. To serve them directly from nginx rather than from the back-end server, I'll declare that all URLs that start with a '/images' shall be served from the local system rather than being passed to the upstream servers:

location /images {
    root /usr/share/nginx;
}

And that's it. From here, my front-end encrypts, compresses and serves the image from its local drive.

Bibliography

[1] http://joeandmotorboat.com/2008/02/28/apache-vs-nginx-web-server-performance-deathmatch/

[2] http://www.wikivs.com/wiki/Apache_vs_nginx

[3] http://blog.webfaction.com/a-little-holiday-present

[4] http://wiki.nginx.org/NginxHttpUpstreamModule 

NVidia GLX extensions

Recently, I started having issues with my video rendering: some screensavers were not working anymore and some games were just ... slow.

The basic troubleshooting gave its verdict: the GLX extensions were not taken into account by the NVIDIA (I have a GeForce 9000) driver. Why? Because the loaded extension was the one from the Xorg foundation rather than the one supplied by NVidia, as shown in the relevant Xorg.X.log where the vendor attribute associated with the glx module was "X.Org Foundation". This shows in the Xorg.X.log as an entry prefixed with "NVIDIA(0)" stating that the glx module isn't from NVidia.


The file in question, "/usr/lib64/xorg/modules/extensions/libglx.so", is provided by the package "xserver-xorg-core", where NVidia glx module is called libglx.so.XXX.YY, in my case libglx.so.256.44.

To solve this, as root:

service gdm stop
cd /usr/lib64/xorg/modules/extensions
mv libglx.so libglx.so.old
ln -s libglx.so.256.44 libglx.so
service gdm start

And voila, upon restart, the vendor now shows as "NVIDIA Corporation", and all opengl-aware applications are happy again!



Friday, September 17, 2010

Back from Arizona

*Snirf* That's the end of my vacation. But the good point is back to testing!

Sunday, September 12, 2010

Last week of vacations!

Alas! Every good thing has to come to an end. This is my last week of vacations, and, after that, I have to resume my work life.

Of course, there are positive aspects: I will also resume my tests of nginx and a few other things I have been working on.

Also, don't forget to check the album as it will be updated soon with all my pictures from my tour in Arizona.

Thursday, September 9, 2010

Grand Canyon

I'm enjoying some time in the Grand Canyon. That's truly amazing, and the views are exceptional. I'll post my pictures as soon as I'm back in New York.

Friday, September 3, 2010

Smile, it's Freeday!

Woohoo! Today is my last day at work before two weeks of freedom. Time to dust my books and prepare my hiking shoes.

Thursday, September 2, 2010

A few new pictures on our album

New pictures in the album! Expect some more after our trip to the Grand Canyon.

nginx

Started playing with nginx, that's heavy stuff! More news when I've done more tests.

Nginx HTTP Server