technical Archive - Fresh out of the labs

Archive for the ‘technical’ Category

NoSQL is a term doing us a disservice

February 1st, 2012

Although we edit NoSQL software, I always disliked calling our product a NoSQL database. I know we say in the third paragraph of our web page that wrpme is a NoSQL database, but see, right after it says that we prefer to call it a postmodern database as proposed by Dr. Richard Hipp.

The obvious reason why we use this fancy word is because we’re French which makes us pretty much the pinnacle of snobbery and pedantry. The other reason is because we weren’t able to come up with a better term.

There’s also the thing that we don’t like NoSQL is because it’s a negative term that suggests our software is here to destroy Relational Database Management Systems (RDBMS) which are the one true evil.

I think I can safely say that no one is trying to destroy anything.

Where I say good things about RDMBS

I guess the NoSQL term originated from someone who was fed up with yet another pervert use of relational databases.

I’ve seen my share, I can relate to that feeling.

The first thing that comes to my mind is any software that uses a RDBMS to serialize data. That's pretty overkill, isn't it? After all, I however think I prefer the “one table to rule them all” scheme. Wait. One table is good, but, I’ve heard primary keys are good too so if we make everything a primary key it must be very good. Oh, and more indexes please, I can’t get enough of them. I know that I will somehow need to research those binary blobs, so index them please. And index the indexes to make the indexes faster.

So yes, it can get very silly.

Let’s not forget one thing though: an incredible amount of intelligence has been poured for more than forty years into RDBMS. People much more intelligent than you, maybe almost as intelligent as us (and that’s saying a lot, see paragraph two), have worked very hard to solve extremely complex problems. And succeeded.

Please, please, please, make sure it dwells very well in your mind: RDBMS work, and they work reliability, and they can adapt to your business case very well, and they when you account everything they do. They have their limits - like everything in the universe - but every time you book a flight, order a book or send money they’re proving to yourself and to the world how dependable they are.

NoSQL engines are for the most part crude, useless and unreliable and as for us, we know we still have a long way to go in terms of flexibility, features and proven reliability.

When you complain that your relational database is too slow, the problem is not the database. The problem is most likely how you use it.

Let’s talk performances

So, am I killing our business? Not really.

RDBMS are fast, but ~~NoSQL~~ postmodern databases can be damn fast. Although you may not need the speed, you may like the fact you need less computing power to handle the same load.

Additionally, to be fast, relational databases have to be used properly. Let’s be realistic for a second, it’s hard to be good at SQL. Non-relational databases are “more obvious” and closer to how the typical programmer thinks and for simple use cases you are most likely to do the right thing with a NoSQL engine than with a relational one.

Ever tried hammering a reasonably sized RDBMS with one thousand distinct clients? A real one? With atomic, consistent, isolated and durable (ACID) transactions? With each client querying the database like there’s no freaking tomorrow? Did it also end up with a database administrator in the air vent with a crossbow aimed at you? I think I made my point.

RDBMS, in certain contexts, can be slow because one of their best features, ACID transactions, come with a hefty price. This is not because RDBMS are poorly done, it’s because to truly ensure that your transaction is atomic, consistent, isolated and durable the database needs to do a lot of work.

And while we’re on the topic of ACID transactions, this important feature is also the reason why they don’t scale very well. Distributing ACID transactions is difficult. You can’t just add commodity servers and expect a linear increase in performance.

Did I say difficult? I wanted to say near-impossible.

To scale a modern relational database, you partition data into buckets and spread the load over buckets (this is an over-simplification, but bear with me). This is called partitioning or sharding, depending on how you split the data. The limit with this approach is that it requires carefully planning the partitions as it’s much more difficult to adjust later. This is not unlike partitioning a hard drive when you install your operating system. This is generally not something you tweak later on production systems, even if you can.

I know, I know. Some new engines are coming out, claiming they can offer the speed of NoSQL and the reliability of ACID transactions.

Are they lying?

Well, we haven’t benchmarked them (yet), but one thing is certain: if they offer truly ACID transactions, they have a performance tax to pay. They can be clever about it and there’s clearly room for achieving great things, but they will always be disadvantaged.

In other words, state of the art relational databases with ACID transactions will always be an order magnitude slower than state of the art non-relational databases without ACID transactions.

Let’s talk money

The other big problem is that over time relational databases became bloated with “wtf” features, because you really need to be able to do Java inside SQL, right?

The dark shroud of enterprise software obfuscated the qualities of otherwise fine products. That means you will need someone to shepherd the weak through the valley of darkness: a database administrator.

Do you really need more weird people in your organization? I submit you do not.

Which brings us to a topic top management understands very well: storage is an order of magnitude more expensive on relational databases. You see that terabyte drive you can buy for 50 €? Want to add a terabyte to your RDBMS? That will be 5 000 €, thank you very much (I’m not making this up).

This last piece of information probably helps you understand why there is a strong interest for non-relational databases as we enter the yotta world.

When should you go non-relational?

RDBMS are great, but they’re not great at everything and sometimes they truly suck.

That’s why we have non-relational databases, because if you throw away relations and transactions, you can do interesting things in terms of processing, scalability and pure performance.

But do you really want to throw away relations and transactions? Do you need to?

How many gadgets has Batman? We will agree on a number much greater than one. Since you’re probably an order of magnitude below Batman in terms of awesomeness, you will agree that you need to be at least as much prepared as he is. In other words: tool up.

That’s why, if you start a new project, you should definitely consider non-relational databases and include them in your architecture. I guarantee you that you have non-relational data that will be cheap and efficient to store in a post-modern database, and you might even be able to go fully non-relational! Ask us or our beloved competition if you need help.

For existing projects however, we’ve seen that many performance problems can be solved with database tuning and proper caching.

Nevertheless, once you have done that, you may still have performance issues. The thing to understand is that the transition from a relational to a partially (or fully) non-relational schema can be extremely disruptive. One approach is to locate the “hot” data and either duplicate or relocate it into a postmodern database.

I could write a lot more on this topic actually and there’s much to be said.

What I wanted to show is that NoSQL is more about shifting the balance a little less on the relational side than killing RDBMS. Maybe the term AltSQL is a better one as it is a reminder that we’re trying to find new, not trying to demean existing ones.

As for us we will stick to the term postmodern database for now and throw a party for overloading another customer’s network (true story that will be the topic of another post).

On quality and code size

September 26th, 2011

I recently came across a great blog post asserting - based on numerous studies - that size is the best predictor of code quality.

According to my own experience, this is spot on.

I can think of a couple of explanations:

Concise code is generally the work of experienced and skilled developers.
Removing dead code greatly increases code quality.
Refactored code tends to be smaller (Adding features is not refactoring).

Point 2 and 3 result in a smaller code base and are a sign of a continuous code review process ("no code is set in stone" principle). In my experience, re-reading the code regularly is one of the greatest contributors to code quality, much more than unit testing and continuous integration (which are nevertheless required to make frequent refactoring possible).

I know that the best bug fixes I did were generally fixes that reduced the line count. Those bug fixes were generally a case of "Who the hell did something that complex for a problem that simple?! Oh wait, I did...".

In other words if you put efforts into code quality, this will generally result in reduced code size. This, to me, explains more the correlation between size and quality: concision is more an indicator of quality than a cause.

Setting up a high performance wordpress server

April 18th, 2010

Our website isn't critical, in the sense that, if the website goes down, our operations remain unaffected. However, lately, the website was becoming extremely slow and it was turning into bad publicity.

Over the time I grew tired of this slowness until, one day, I decided to address the issue once for all. I bought a dedicated server, installed FreeBSD, nginx and everything needed to have our corporate website and the blog running and decommissioned the old server.

Unfortunately, you may have noticed that this last week, the quality of service was pretty low. This is entirely my fault as I switched to the new server too quickly.

Nevertheless, we're now pretty satisfied with the responsiveness and resource consumtion of the new production environment.

This post is a recollection of the steps I went through last week.

Gearing up

We generally favor a BSD setup for servers. We've had very good experience with Windows Server 2008 and Ubuntu Server Edition but whenever we can, we go BSD because that's what l33t d00dz do.

Ideally we would have opted for OpenBSD, but OpenBSD dedicated hosting is extremely hard to come by, and we wanted to have provider support for the OS. We didn't want to get a random Linux box and bootstrap an OpenBSD installation on it. If ever we got a problem with the box at some point, the support would have answered "please use a supported operating system".

FreeBSD remains an outstanding server operating system, and we happily leased a FreeBSD box with much more disk space and computing power we need.

When considering which webserver to use, this is a no-brainer: you want speed, you chose nginx.

As we use WordPress for the blog, it means we'll also have to install MySQL.

FreeBSD

FreeBSD isn't OpenBSD when it comes to security, but the default security configuration is pretty decent if you opt for a minimal install.

The first good reflex is to set up PF to only allow incoming transmission to the SSH server and the web server. It can also be used to normalize the incoming packets and protect against spoofing.

In case you were wondering what PF is, well, put simply, PF is the best IP filter available. Its strength comes from its clear and unambiguous grammar and its linear complexity. What do I mean by linear complexity? If you have a simple setup, your configuration file will be small and simple. If you have more advanced needs, the grammar enables you to go very deep in the details. It's one of these products you test and never look back.

Without any further ado, here is a minimal PF configuration file with normalization, antispoofing and only ssh and https incoming traffic allowed:

# skip the loopback
set skip on lo
# packet normalization
scrub in
# block everything in
block in
pass out keep state
# antispoofing
antispoof quick for { lo $ext_if }
# allow ssh and web in
pass in on $ext_if proto tcp to ($ext_if) port \
{ ssh, http, https } flags S/SA modulate state

A good security habit - which is not limited to FreeBSD - is to set up a non-root account, disable ssh root login (an option in /etc/ssh/sshd_config) and use sudo and su when privileges are needed (your additional account needs to be in the wheel group for this to work).

SSH login should be done only via a key and the password for all the accounts need to be extremely strong (more than 100-bit of entropy). Using SSH keys for login makes it possible to use extremely strong password.

It is possible to setup the SSH server to listen on a different port than 22, but it might be incompatible with your local firewall policy. When you do that, make sure that you have a way to access your box should something go wrong.

A good sanity check is that when you run the "netstat -an" command, you should only see the SSH server listening to the outside. All other servers must be bound to localhost (or better, an unix socket). I know we set up a firewall to protect against that kind of attacks, but good security is achieved through redundancy.

Recompiling the kernel and getting rid of everything you don't use is a nice final touch. Why have IPV6 when your server is only IPV4? If your machine got rooted via IPV6, would you be able to realize it easily?

Recompiling the kernel on BSD requires creating a configuration file, compiling it with config(5) and then run make. It's a bit less user friendly than in Linux (as far as kernel compilation friendliness goes).

nginx

nginx is an excellent webserver, with a very low memory footprint and very high capability. I've spent a great deal of time inside the bowels of this software and although I regret the choice of old-school C (over template-intensive C++), I really think it's a nice piece of software. It's clean, rigorous and consistent. Our industry needs more software of this quality.

nginx has got an extremely good security history and is delivered secure by default. It doesn't mean the default configuration cannot be improved.

The most obvious is to limit the number of connections per client, set up more aggressive timeouts and limit the buffers' size a client might send. This offers some minimal protection against DoS. Tread carefully however as they can have a bad impact on performance.

You can restrict things further by only allowing GET, HEAD and POST commands, denying all these weird HTTP requests you never use.
Last but not least, you will want to hide the nginx version you are using, no need to make exploit lookup easier, isn't it?

In the http section you will therefore want to add the following:

# hide nginx version
server_tokens off;
# limit connections
limit_zone slimits $binary_remote_addr 5m;
limit_conn slimits 20;
# set up restricting client buffers
client_header_buffer_size 4k;
client_max_body_size 5M;
large_client_header_buffers 2 8k;
# set up aggressive timeouts
client_body_timeout 10;
client_header_timeout 10;
keepalive_timeout 5 5;
send_timeout 10;

and in the server section

# only allow GET, HEAD and POST
if ($request_method !~ ^(GET|HEAD|POST)$ )
{
return 444;
}

Using a dedicated partition is a possibility. It enables you to prohibit features such as setuid bits and even executable permissions at the partition level. It also protects against flooding, should your users be able to upload content.

The drawback of using a dedicated partition is that it makes space (re)allocation more difficult.

MySQL

If you ask me, I'll tell you that relational databases are overused. But we didn't program WordPress, and WordPress only works with MySQL, we therefore need to install it on the box.

Set up a strong password for the MySQL root account and add another account that will have only access to the databases your web server will use. Never use the root account to access your tables from a web server, this is a major security risk.

If for any reason you lose the control of your web server, or someone achieves to extract the login credentials, they will only be able to access your web server database. With the root account, the attacker would - for example - be able to create accounts and even lock you out of your databases!

Additionally, you need to make sure that MySQL only listen to an unix socket (if the MySQL server sits on the same server than nginx), this is done with the option "--skip-networking".

I strongly recommend against installing phpMyAdmin or any equivalent software on production machines.

PHP

On FreeBSD, PHP comes by default with a decent security configuration, including the Suhosin patch, which is much better that the safe mode.

Nevertheless, it's good to disable all the extensions you're not going to use, not only this will increase stability and decrease memory usage, but it will reduce the attack surface.

Install PHP 5 from the ports where you will be able to activate FastCGI, this is important as nginx doesn't have a native PHP support à l'Apache.

Configuring the PHP FastCGI server

The PHP FastCGI server exits after a certain number of requests is done. Since PHP isn't a very stable parser, preventing it from running a long time increases resilience. The drawback is that you have to keep the server running manually.

We will use daemontools to keep the FastCGI server up, but any process monitor will do.

It's advised to use spawn-fcgi to run your PHP FastCGI server, as it will enable you to set finer privileges and it works very well with daemontools.

Here is a run script example for supervise:

#!/bin/sh
export PHP_FCGI_CHILDREN=3
export PHP_FCGI_MAX_REQUESTS=250
exec /usr/local/bin/spawn-fcgi -n -s /tmp/php_fcgi.socket \
-u www -U www -g www \
-- /usr/local/bin/php-cgi

You can see that we bind the FastCGI server to an unix socket instead of a TCP port.

Adding PHP support to nginx

nginx needs to be configured to redirect php parsing to the FastCGI server, you just need to add the following to your server section:

index index.php;

log_not_found off;

location ~ \.php$
{
try_files $uri = 404;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME \
$document_root$fastcgi_script_name;
include fastcgi_params;
fastcgi_pass 127.0.0.1:9000;
}

While we are at it, we will add the required lines to make WordPress work nicely. We need to make sure that WordPress' crafted URLs are not interpreted as 404 by the nginx server:

location /blog
{
try_files $uri $uri/ @wp_blog;
}

location @wp_blog
{
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME \
$document_root/blog/index.php;
fastcgi_param QUERY_STRING q=$uri&$args;
fastcgi_pass unix:/tmp/php_fcgi.socket;
fastcgi_param SCRIPT_NAME /blog/index.php;
}

Note that you need to replace "/blog" with the actual path to your blog and fastcgi_pass will require the hostname and port to your FastCGI server (127.0.0.1:9000 by default, in our case we use an unix socket).

Now that your web server is set up, you might want to be able to monitor its activity. You can do this with nginx-rrd. I won't go too much into the details, just make sure you don't make your status page world readable.

Installing WordPress

Despite the recurring security issues of WordPress, we didn't want to switch to another engine. WordPress is full featured, very well supported and well, Matt already did an incredible layout for the blog...

Making WordPress work with nginx is everything but straightforward. The good news is that we've already done the toughest.

Before installing WordPress, create a dedicated database and user with MySQL, as using root to access a shared database is a very bad idea (bis repetita placent).

Once you have your database credentials working, you can install WordPress in following the infamous 5-Minute install procedure.

WordPress only requires you to edit one configuration file: wp-config.php. You will also see four entries named AUTH_KEY, SECURE_AUTH_KEY, LOGGED_IN_KEY and NONCE_KEY require a passphrase. Generate a cryptographically strong passphrase with the following command:

dd if=/dev/random bs=32 count=1 | sha256

The last line - the hexadecimal number - is your secure passphrase. Now your friends will look at you differently.

For our setup we have only one mandatory plugin: the nginx compatibility one. Should you wish to increase performance, WP Super Cache is highly recommended.

Installing memcached?

If you run a high-traffic web site, you will need to install a cache server.

memcached is supported by WordPress through this plugin. I do not know how it cooperates with WP Super Cache, but I submit the two are mutually exclusive.

At Bureau 14, we don't use memcached for the very simple reason we've developed our own high-performance cache engine that runs faster and scales better. It also comes with more fur. Don't worry, we won't keep this software to ourselves for ever: we will run a closed beta this summer. Should you wish to know more, feel free to subscribe to our mailing list.

Using memcached (or any cache engine) for static files generally yields no performance improvement. The reason is that nginx already caches static files.

Closing words

Although this setup is not as straightforward as the classic Linux/Apache combo, I truly think you will find the trouble to be worth it.

We surely do.

Do you need help tuning and securing your servers? We can help.

Darwinian optimization

March 27th, 2010

This morning was one of these mornings when you wake up with that kind of strange idea only sleep can bring you. It's a concept related to software optimization at the assembly level.

Before I explore this idea with you, I'd like to spend few paragraphs to talk about the vast and inspiring field of assembly optimization.

Optimizing code

Assembly optimization is such a huge topic that I won't even try to be exhaustive. This brief tour should help you understand where the idea comes from. Experts will forgive the generalities and approximations of this quick overview. Should you wish to know more, feel free to read the abundant and excellent documentation provided by Intel.

The natural approach

This first and most straightforward one consists in replacing a set of instructions by an equivalent, but faster, set of instructions.

An example of this is when you write

1	xor eax, eax

instead of

1	mov eax, 0

on the IA32.

These instructions both set the eax register to 0, except that the former is faster (This xor register, register instruction also has got other special powers I will tell you later about...)

Optimizing is in that case just a matter of search and replace, but unfortunately this technique will not lead you very far.

On our heavy-pipelined processors, you will additionally do everything you can to make the most of branch prediction.

Optimizing branches

Branch prediction is a set of heuristics built inside your CPU to avoid flushing the pipeline when branching. It determines the most likely branch when reaching a conditional instruction and preloads the guessed address. Unexplored branches are predicted using static heuristics, explored branches using statistics.

Making the most of branch predictions means disposing conditions in a way favorable for these heuristics, or even avoid branching in the first place through the use of conditional instructions. Unrolling loops is another way to tackle this as it replaces the loop by the equivalent repeated instructions. Unrolling has however to be done reasonably, as excessive use of this technique may result in a larger code that no longer fits in the trace cache.

Putting branches aside, arranging instructions also has got a great impact of performances, because of stalling.

Avoiding stalls

Nota bene : the following example oversimplifies the issue as it overlooks - amongst other things - parallelization on processors featuring multiple treatment units. In other words, rearranging the above instructions might not always yield a performance improvement, but it nevertheless shows the underlying concept.

1
2
3
4

mov eax, 3
mov ebx, 4
add ebx, eax ; addition of 3 + 4
xor ecx, ecx

The addition can only occur when the ebx register is ready, in other words, the addition will have to wait for the register to be fully set before doing anything. That's a stall. If you notice the instruction on ecx, you will see there is no dependence .

If you place the operation on ecx before the add, you have a potential performance gain because the instruction can execute right away without waiting for the previous one to finish. Sometimes dependencies are more subtle and are based on the flags. This is why - for example - it is generally recommended to avoid instructions such as inc and dec that only affect flags partially.

I earlier wrote that the xor register, register instruction has got some special powers. It can indeed be used to break the dependencies we just examined. When the processor encounters this instruction, it will discard all dependencies related to the register you reset, preventing stalls.

And so much more...

Memory layout is another major topic that is very hard to sum up in few words, but I need to at least name it. Just know that aligning instructions and data on cache boundaries can yield a significant performance improvement in optimizing loads and avoiding false sharing (very important for multithreaded code).

And then, you have also the matter of mixing floating point and single instructions multiple data (SIMD) instructions, partial register stalls, vectorization, stack optimization, store forwarding, locality enhancement...

When you realize the amount of techniques that can be used to optimize a code fragment, finding the ideal (or close to ideal) set of parameters for your processor is extremely difficult. This is where we go back to instrumentation, without instrumentation, you (or most likely the compiler) optimize a priori for your given architecture. With instrumentation, you make decisions based on hard evidence.

You need to instrument representative runs of your program to make it worthwhile, and sometimes, there isn't much the optimizer can do to improve the speed of your software.

This morning's idea

What about using natural selection to optimize a routine?

If you inject random optimizations in thousands of instances of a routine, benchmark them, discard the slower ones and keep working on the faster ones, will it eventually output the fastest possible implementation?

Obviously the following requirements would have to be met:

The routine needs to be testable for correctness to a reasonable extent as optimizations might result in modified behavior
It must be possible to generate a representative set of inputs in order to evaluate performance realistically
Enough computing power to mutate, spawn and test the routine as waiting one month for the result might not be acceptable...

The routine mutator would need to know a comprehensive set of optimization techniques for the given architecture, but it could also simply inject random opcodes and let the test phase discard non-working versions.

The approach can also be used to generate the smallest possible program, or any feature that can easily and objectively be measured.

When would that approach be reasonable? Probably for fundamental software bricks, such as an operating system scheduler, the pager or some driver's routine.

That and God of War.

BoostCon 2010

March 16th, 2010

The BoostCon is a Boost centric conference, but it's actually much more than that. It's probably the best C++ conference in the world.

A bold statement?

Contrary to what the name of the conference might imply, it's more C++ centric than Boost centric. A lot of talks revolve around the boost libraries, but there's also content related to the language itself and the challenges we all face.

The advantage of the Boost tropism is that the topic stays on "real world issues". No endless debate about how to phrase a new feature in the C++ standard, no endless theoretical speech about a problem that only exists when you program in a zero gravity environment on a computer that might be invented in 2030 if LSD gets legalized.

I was at BoostCon 2009, not only the content was accurate, advanced and of high quality, but it was extremely relevant.

"Spot on" is actually the term that comes to my mind.

Would it be only for the keynote by Andrei Alexandrescu and the introduction to software transactional memory I would have been happy to do the trip.

You know these conferences where you have some days filled with "so-so" talks, and you go to that talk about optimizing memory usage on a system you've never heard of because all the other talks look horrid, and the guy just read his slides with a vocoder tone, and you start firing up your laptop and suddenly the conference room becomes a very expensive WiFi spot.

Well BoostCon is nothing like it.

What's more bound to happen is that you're going to regret you still don't master mitosis, because you'd really like to learn more about smart pointers and Spirit, and both talks happen at the same time, and they are both given by really great people.

You'll fly back with the head full of new ideas to tackle the issues you've left at home (I'm not talking about how to get the exoskeleton in S.T.A.L.K.E.R.: Зов Припяти, because it's really just a matter of harvesting artifacts like there's no tomorrow), rested and energized by the intense intellectual maelstrom.

The conference is located in Aspen, Colorado.

I'll nitpick and say although the site and the city in itself are really a great, great place to have a conference, it's a bit inconvenient if you live outside the USA. From my European point of view, American conferences located in Boston or New-York are be more convenient.

If you're serious about C++, fly your team to the BoostCon. It's an outstanding investment.

Registration is just one click away