Although we edit NoSQL software, I always disliked calling our product a NoSQL database. I know we say in the third paragraph of our web page that wrpme is a NoSQL database, but see, right after it says that we prefer to call it a postmodern database as proposed by Dr. Richard Hipp.

The obvious reason why we use this fancy word is because we’re French which makes us pretty much the pinnacle of snobbery and pedantry. The other reason is because we weren’t able to come up with a better term.

There’s also the thing that we don’t like NoSQL is because it’s a negative term that suggests our software is here to destroy Relational Database Management Systems (RDBMS) which are the one true evil.

I think I can safely say that no one is trying to destroy anything.

Where I say good things about RDMBS

I guess the NoSQL term originated from someone who was fed up with yet another pervert use of relational databases.

I’ve seen my share, I can relate to that feeling.

The first thing that comes to my mind is any software that uses a RDBMS to serialize data. That's pretty overkill, isn't it? After all, I however think I prefer the “one table to rule them all” scheme. Wait. One table is good, but, I’ve heard primary keys are good too so if we make everything a primary key it must be very good. Oh, and more indexes please, I can’t get enough of them. I know that I will somehow need to research those binary blobs, so index them please. And index the indexes to make the indexes faster.

So yes, it can get very silly.

Let’s not forget one thing though: an incredible amount of intelligence has been poured for more than forty years into RDBMS. People much more intelligent than you, maybe almost as intelligent as us (and that’s saying a lot, see paragraph two), have worked very hard to solve extremely complex problems. And succeeded.

Please, please, please, make sure it dwells very well in your mind: RDBMS work, and they work reliability, and they can adapt to your business case very well, and they when you account everything they do. They have their limits - like everything in the universe - but every time you book a flight, order a book or send money they’re proving to yourself and to the world how dependable they are.

NoSQL engines are for the most part crude, useless and unreliable and as for us, we know we still have a long way to go in terms of flexibility, features and proven reliability.

When you complain that your relational database is too slow, the problem is not the database. The problem is most likely how you use it.

Let’s talk performances

So, am I killing our business? Not really.

RDBMS are fast, but NoSQL postmodern databases can be damn fast. Although you may not need the speed, you may like the fact you need less computing power to handle the same load.

Additionally, to be fast, relational databases have to be used properly. Let’s be realistic for a second, it’s hard to be good at SQL. Non-relational databases are “more obvious” and closer to how the typical programmer thinks and for simple use cases you are most likely to do the right thing with a NoSQL engine than with a relational one.

Ever tried hammering a reasonably sized RDBMS with one thousand distinct clients? A real one? With atomic, consistent, isolated and durable (ACID) transactions? With each client querying the database like there’s no freaking tomorrow? Did it also end up with a database administrator in the air vent with a crossbow aimed at you? I think I made my point.

RDBMS, in certain contexts, can be slow because one of their best features, ACID transactions, come with a hefty price. This is not because RDBMS are poorly done, it’s because to truly ensure that your transaction is atomic, consistent, isolated and durable the database needs to do a lot of work.

And while we’re on the topic of ACID transactions, this important feature is also the reason why they don’t scale very well. Distributing ACID transactions is difficult. You can’t just add commodity servers and expect a linear increase in performance.

Did I say difficult? I wanted to say near-impossible.

To scale a modern relational database, you partition data into buckets and spread the load over buckets (this is an over-simplification, but bear with me). This is called partitioning or sharding, depending on how you split the data. The limit with this approach is that it requires carefully planning the partitions as it’s much more difficult to adjust later. This is not unlike partitioning a hard drive when you install your operating system. This is generally not something you tweak later on production systems, even if you can.

I know, I know. Some new engines are coming out, claiming they can offer the speed of NoSQL and the reliability of ACID transactions.

Are they lying?

Well, we haven’t benchmarked them (yet), but one thing is certain: if they offer truly ACID transactions, they have a performance tax to pay. They can be clever about it and there’s clearly room for achieving great things, but they will always be disadvantaged.

In other words, state of the art relational databases with ACID transactions will always be an order magnitude slower than state of the art non-relational databases without ACID transactions.

Let’s talk money

The other big problem is that over time relational databases became bloated with “wtf” features, because you really need to be able to do Java inside SQL, right?

The dark shroud of enterprise software obfuscated the qualities of otherwise fine products. That means you will need someone to shepherd the weak through the valley of darkness: a database administrator.

Do you really need more weird people in your organization? I submit you do not.

Which brings us to a topic top management understands very well: storage is an order of magnitude more expensive on relational databases. You see that terabyte drive you can buy for 50 €? Want to add a terabyte to your RDBMS? That will be 5 000 €, thank you very much (I’m not making this up).

This last piece of information probably helps you understand why there is a strong interest for non-relational databases as we enter the yotta world.

When should you go non-relational?

RDBMS are great, but they’re not great at everything and sometimes they truly suck.

That’s why we have non-relational databases, because if you throw away relations and transactions, you can do interesting things in terms of processing, scalability and pure performance.

But do you really want to throw away relations and transactions? Do you need to?

How many gadgets has Batman? We will agree on a number much greater than one. Since you’re probably an order of magnitude below Batman in terms of awesomeness, you will agree that you need to be at least as much prepared as he is. In other words: tool up.

That’s why, if you start a new project, you should definitely consider non-relational databases and include them in your architecture. I guarantee you that you have non-relational data that will be cheap and efficient to store in a post-modern database, and you might even be able to go fully non-relational! Ask us or our beloved competition if you need help.

For existing projects however, we’ve seen that many performance problems can be solved with database tuning and proper caching.

Nevertheless, once you have done that, you may still have performance issues. The thing to understand is that the transition from a relational to a partially (or fully) non-relational schema can be extremely disruptive. One approach is to locate the “hot” data and either duplicate or relocate it into a postmodern database.

I could write a lot more on this topic actually and there’s much to be said.

What I wanted to show is that NoSQL is more about shifting the balance a little less on the relational side than killing RDBMS. Maybe the term AltSQL is a better one as it is a reminder that we’re trying to find new, not trying to demean existing ones.

As for us we will stick to the term postmodern database for now and throw a party for overloading another customer’s network (true story that will be the topic of another post).

A couple of weeks ago, a friend of mine had a heart attack. As he felt a strong pain in both arms, he called for help. Less than fifteen minutes later firemen arrived and took him to the hospital. As soon as he arrived he was promptly directed to the operating theater. There, the doctors inserted a needle in his arm, acceded the heart, removed the thrombus and placed a stent. Two days later he was dismissed. Except for his daily medicine intake and a very small scar near the wrist, it’s impossible to know he had a heart attack less than a month ago.

Cars no longer need people to be operated and robots move like humans. If you combine both, it’s not difficult to imagine fully automated delivery services. For example, you could order an item online and have it delivered to you without any human intervention.

And as I type these words, a man-made object is leaving the solar system...

Happy new year 2012. Welcome to the future.

Parrots are wonderful creatures of Earth. How many animals can you actually talk to?

Not that many.

One might retort that parrots only stupidly repeat what they hear.

A valid point, except it's completely wrong.

Parrots are amongst the most intelligent birds on Earth. Some species are even able to associate words with meanings. The African Grey Parrot, in particular, has a degree of intelligence comparable to dolphins and great apes. They understand our language to the point they can form sentences and even display a sense of humor.

Owning a parrot is a huge responsibility as it's likely to outlive you. Overmore, it will be depressed if you are gone missing for too long, is bored or hasn’t been bred properly.

Surely, the last thing you want is to have a bird looking at you with sad eyes as she loses her feathers.

On the other hand, if you have the time, dedication and patience, one day you will be able to walk around with your pet seating on your shoulder.

As she shares witty comments with the audience you will realize one thing: you just made one huge step toward becoming a mighty pirate.

According to my own experience, this is spot on.

I can think of a couple of explanations:

1. Concise code is generally the work of experienced and skilled developers.
2. Removing dead code greatly increases code quality.
3. Refactored code tends to be smaller (Adding features is not refactoring).

Point 2 and 3 result in a smaller code base and are a sign of a continuous code review process ("no code is set in stone" principle). In my experience, re-reading the code regularly is one of the greatest contributors to code quality, much more than unit testing and continuous integration (which are nevertheless required to make frequent refactoring possible).

I know that the best bug fixes I did were generally fixes that reduced the line count. Those bug fixes were generally a case of "Who the hell did something that complex for a problem that simple?! Oh wait, I did...".

In other words if you put efforts into code quality, this will generally result in reduced code size. This, to me, explains more the correlation between size and quality: concision is more an indicator of quality than a cause.

Few are agreeable in conversation, because each thinks more of what he intends to say than of what others are saying and listens no more when he himself has a chance to speak.
-François de la Rochefoucauld

You’ve certainly experienced reunions where you told yourself "This person isn’t saying anything interesting and this meeting is going nowhere.".

In my case it's not really a problem since I'm the boss and I can punch people in the face and get away with it (you probably can't). By the way being allowed to kill people is one of the top reasons why you should start your own startup, provided you keep the death toll within the limits allowed by the law.

But I disgress. What can you do to help the meeting go somewhere?

As strange as it may sound talking less is part of the answer. This is because silence alleviates stress and forces oneself to actually listen. When you listen, you communicate better. Better communication means better meetings.

An experiment

At Bureau 14 we're pretty awesome scientists. And I'm not talking about all the computer science we do all day (C++ is more black magic than science actually), but more like the one that confirmed that interns can't survive without oxygen.

The one you can do is the following: the next meeting you attend, do not talk at all.

It’s highly likely you will notice the following things:

1. Every attendant is saying the same thing over and over with a varying degree of eloquence;
2. The amount and the distribution of information is unchanged by the meeting (unmodified entropy);
3. What you could have said wasn't that useful, intelligent or unique (ouch!).

Problem ? Solution ?

You might tell me: "I'm going to enforce some rule during meetings to fix that":

Unless one has a genuine question, needs information or has something valuable and related to share there is absolutely no reason to speak.

That sounds like a pretty obvious rule on which everybody could easily agree. But that would not change a thing.

Let's be realistic for a minute.

We all think that what we say is pretty clever. Actually what happens is that our overinflated ego makes us believe it's ok to reharse what the person next to us just said because, really, it's important to participate and it's less boring to spend one hour talking than listening and by the way it's probably a good opportunity to disgress and talk about the latest Family Guy episod.

Newsflash: we're not that intelligent and what we have to say doesn't matter that much. Actually most of the time it's completely out of topic.

In other words, we may have the feeling to genuinely contribute when we aren't.

Stop creating rules

Stop believing that every problem must be taken in your own (capable?) hands and needs to be addressed. Stop "managing issues". This is something much more complex and inherent to human nature than just a "problem". It needs education.

The education process is a simple matter of forcing yourself to speak as little as possible, as if you were underwater and you had to keep your precious oxygen for mundane things such as staying alive. Everytime you want to speak, double check that it is relevant and new. Once you've double checked, check again, and again. Listen more, observe.

After a while, you will talk less, be more relaxed and listen more. You'll be a better participant and people will look after your presence. This is the moment when you will realize that contributing to the success of your company is more a matter of listening than a matter of talking.

I thought I took enough care of myself to further postpone this day...

I have a balanced diet. I exercise at least four hours per week. I don’t smoke. I certainly do not fry my brain with illicit substances. I sleep regularly, well and enough. Well except when I have to wake up at three in the morning to catch a plane back to Paris and lose my bag in the taxi. Ah! The stress! That reminds me I’ll have to buy myself a G550 when Bureau 14 rules the world.

For the record, Zeus tried to strike me with lightning several times out of jealousy. I had to climb the Olympus for a private “exhibit” of my Wing Tsun. No wonder I don’t have time for programming with all these delays. For the record the game God of War 3 is based on my story.

Well, guess what? Despite all of this, I’m obviously a brain-dead old man as I reach the extremely advanced age of 33.

What the hell am I talking about? Google Instant! I mean what the Hell?! I feel like I’m using software for hyperactive kids drinking liters of a homemade mix of cocaine and caffeine.

First of all I don’t think the spider is crawling the web according to my search requests, so don’t call it real time search thank you very much. In the finance industry we call the stock streams real-time although they have at least 15 minutes of delay, but we have the right to because we’re bankers and bankers have no soul.

Second Google Instant is extremely confusing. I'm not a designer, but I’m pretty sure it’s a big design mistake to display partial – and potentially unwanted – results in the same place as normal results.

Last but not least, I don’t understand how this is supposed to “save time”. All it does is request searches for items I’m not interested in. Despite me being an old crank, I’m quite able to press the enter key when needed! It’s a waste of bandwidth and CPU time if anything. Visual feedback gives you the illusion of speed, like a progress bar reduces the impression of waiting, but that’s all it is: an illusion.

The heart of the problem is that it’s not just an illusion, it’s also a distraction. It’s not real time search it’s real time advertisement. It feels like watching American TV: there are so many commercials that after a while you forget what you were watching in the first place.

It's pretty clear to me that the purpose of this feature is not to ease my searches it’s to tease me with links. The purpose of Google instant is to increase Google’s revenues.

To that I say: fair enough. Google isn’t a philanthropic institution and as an entrepreneur I fully understand they do everything they can to maximize profits. As an user however I’m dissatisfied.

The feature of Google I like the most is accurate and fast search. As long as they keep improving it that keeps me happy. Although to make me really happy, I’d love them to stop that data-based design nonsense and hire top notch designers to make something of their websites. I'll be 100% honest however and admit that it would only be ice on the cake (That being said, there is to me a market for a web application that does a better job at presenting Google’s results than Google).

In the end, I really don’t understand Google Instant. What am I missing? I am just too old to get it? Or is Google Instant just too nerdy?

When was the last time you laughed at work? Yesterday? Last week? Last month? You don’t even remember?

Do not make the mistake of thinking that a place where you regularly laugh is a place where nothing gets done. Quite the contrary.

Serious work requires – amongst other things – concentration, cohesion and fluent communication. Laughter is an outstanding catalyst for all these things.

If you want a spark of creativity to light a team of competent and intelligent people, mirth is what you need. The thrill and energy that fills you when you shared a good laugher is an incredible creativity booster.

Amusement is very good for bounding as well. Is there a need to demonstrate you like to work with people with whom you have fun?

Finally, humor encourages direct communication. You’ll feel more comfortable to discuss mistakes and errors without unnecessary ado when the mood is joyful.

If we were pirates on a ship we would sing merry songs to achieve this result, but geeks in an office prefer to crack up dark jokes.

When I interview people, I do my best to pin down their ability to get along with the rest of the team and to be honest with you, I test their wit.

Using humor during interviews has got the incredible side effect of detecting flexibility and (some sort of) cleverness. For example, asking a candidate if he tends to stab colleagues with whom he disagrees opens the door to a very unusual but effective test of personality. You might even add that a past of serial killer is generally a recruitment’s deterrent.

We’re looking for competent, bright and insightful people that will resist pressure and are fun to be around. Of course we’re not hiring comedians as we want – first of all – people to be able to write outstanding software.

When the team is under pressure because of an angry customer, that a flurry of bugs magically appeared in the repository (really, it can’t be me!) and everybody is working too much, being around people with whom you can blow off steam is a much better feat than free gourmet food or complimentary massages.

To be completely honest with you, there’s also the thing related to the fact that we can’t afford the free gourmet food.

Invoices are generally useless for individuals, but they are a necessity for companies.

They are required because your local doom kommando fiscal administration will ask for them as a proof you did indeed spend the money. And before they ask for them, your accountant will because if you want him to put his John Hancock on your anual report, you better make sure it's pretty clean of vague, non-business releated entries such as "DB9 leasing" and "Casino losses".

The invoice must have the name of the company, its registration number, an address (including the country), the name of the customer, the name of the customer's company (if any), the items list with a clear and consise description for each - don't scratch your head as long as the word "software" is in the description, the accountant will understand what it's all about - a price and finally the taxes and a grand total. Export that in a printable format (PDF is a de facto standard) and attach it to all your confirmation emails.

What I just said might sound extremely silly, but for ten software licenses I bought I received a valid invoice only three times. Generally speaking I'm more inclined to do business with companies that make my life easier.

Make sure you have all the details of your business process straight.

By the way, Hi! How are you doing? Long time no see! Here, we're going great and are working very hard on our first product!

Over the time I grew tired of this slowness until, one day, I decided to address the issue once for all. I bought a dedicated server, installed FreeBSD, nginx and everything needed to have our corporate website and the blog running and decommissioned the old server.

Unfortunately, you may have noticed that this last week, the quality of service was pretty low. This is entirely my fault as I switched to the new server too quickly.

Nevertheless, we're now pretty satisfied with the responsiveness and resource consumtion of the new production environment.

This post is a recollection of the steps I went through last week.

Gearing up

We generally favor a BSD setup for servers. We've had very good experience with Windows Server 2008 and Ubuntu Server Edition but whenever we can, we go BSD because that's what l33t d00dz do.

Ideally we would have opted for OpenBSD, but OpenBSD dedicated hosting is extremely hard to come by, and we wanted to have provider support for the OS. We didn't want to get a random Linux box and bootstrap an OpenBSD installation on it. If ever we got a problem with the box at some point, the support would have answered "please use a supported operating system".

FreeBSD remains an outstanding server operating system, and we happily leased a FreeBSD box with much more disk space and computing power we need.

When considering which webserver to use, this is a no-brainer: you want speed, you chose nginx.

As we use WordPress for the blog, it means we'll also have to install MySQL.

FreeBSD

FreeBSD isn't OpenBSD when it comes to security, but the default security configuration is pretty decent if you opt for a minimal install.

The first good reflex is to set up PF to only allow incoming transmission to the SSH server and the web server. It can also be used to normalize the incoming packets and protect against spoofing.

In case you were wondering what PF is, well, put simply, PF is the best IP filter available. Its strength comes from its clear and unambiguous grammar and its linear complexity. What do I mean by linear complexity? If you have a simple setup, your configuration file will be small and simple. If you have more advanced needs, the grammar enables you to go very deep in the details. It's one of these products you test and never look back.

Without any further ado, here is a minimal PF configuration file with normalization, antispoofing and only ssh and https incoming traffic allowed:

A good security habit - which is not limited to FreeBSD - is to set up a non-root account, disable ssh root login (an option in /etc/ssh/sshd_config) and use sudo and su when privileges are needed (your additional account needs to be in the wheel group for this to work).

SSH login should be done only via a key and the password for all the accounts need to be extremely strong (more than 100-bit of entropy). Using SSH keys for login makes it possible to use extremely strong password.

It is possible to setup the SSH server to listen on a different port than 22, but it might be incompatible with your local firewall policy. When you do that, make sure that you have a way to access your box should something go wrong.

A good sanity check is that when you run the "netstat -an" command, you should only see the SSH server listening to the outside. All other servers must be bound to localhost (or better, an unix socket). I know we set up a firewall to protect against that kind of attacks, but good security is achieved through redundancy.

Recompiling the kernel and getting rid of everything you don't use is a nice final touch. Why have IPV6 when your server is only IPV4? If your machine got rooted via IPV6, would you be able to realize it easily?

Recompiling the kernel on BSD requires creating a configuration file, compiling it with config(5) and then run make. It's a bit less user friendly than in Linux (as far as kernel compilation friendliness goes).

nginx

nginx is an excellent webserver, with a very low memory footprint and very high capability. I've spent a great deal of time inside the bowels of this software and although I regret the choice of old-school C (over template-intensive C++), I really think it's a nice piece of software. It's clean, rigorous and consistent. Our industry needs more software of this quality.

nginx has got an extremely good security history and is delivered secure by default. It doesn't mean the default configuration cannot be improved.

The most obvious is to limit the number of connections per client, set up more aggressive timeouts and limit the buffers' size a client might send. This offers some minimal protection against DoS. Tread carefully however as they can have a bad impact on performance.

You can restrict things further by only allowing GET, HEAD and POST commands, denying all these weird HTTP requests you never use.
Last but not least, you will want to hide the nginx version you are using, no need to make exploit lookup easier, isn't it?

In the http section you will therefore want to add the following:

and in the server section

Using a dedicated partition is a possibility. It enables you to prohibit features such as setuid bits and even executable permissions at the partition level. It also protects against flooding, should your users be able to upload content.

The drawback of using a dedicated partition is that it makes space (re)allocation more difficult.

MySQL

If you ask me, I'll tell you that relational databases are overused. But we didn't program WordPress, and WordPress only works with MySQL, we therefore need to install it on the box.

Set up a strong password for the MySQL root account and add another account that will have only access to the databases your web server will use. Never use the root account to access your tables from a web server, this is a major security risk.

If for any reason you lose the control of your web server, or someone achieves to extract the login credentials, they will only be able to access your web server database. With the root account, the attacker would - for example - be able to create accounts and even lock you out of your databases!

Additionally, you need to make sure that MySQL only listen to an unix socket (if the MySQL server sits on the same server than nginx), this is done with the option "--skip-networking".

I strongly recommend against installing phpMyAdmin or any equivalent software on production machines.

PHP

On FreeBSD, PHP comes by default with a decent security configuration, including the Suhosin patch, which is much better that the safe mode.

Nevertheless, it's good to disable all the extensions you're not going to use, not only this will increase stability and decrease memory usage, but it will reduce the attack surface.

Install PHP 5 from the ports where you will be able to activate FastCGI, this is important as nginx doesn't have a native PHP support à l'Apache.

Configuring the PHP FastCGI server

The PHP FastCGI server exits after a certain number of requests is done. Since PHP isn't a very stable parser, preventing it from running a long time increases resilience. The drawback is that you have to keep the server running manually.

We will use daemontools to keep the FastCGI server up, but any process monitor will do.

It's advised to use spawn-fcgi to run your PHP FastCGI server, as it will enable you to set finer privileges and it works very well with daemontools.

Here is a run script example for supervise:

You can see that we bind the FastCGI server to an unix socket instead of a TCP port.

Adding PHP support to nginx

nginx needs to be configured to redirect php parsing to the FastCGI server, you just need to add the following to your server section:

While we are at it, we will add the required lines to make WordPress work nicely. We need to make sure that WordPress' crafted URLs are not interpreted as 404 by the nginx server:

Note that you need to replace "/blog" with the actual path to your blog and fastcgi_pass will require the hostname and port to your FastCGI server (127.0.0.1:9000 by default, in our case we use an unix socket).

Now that your web server is set up, you might want to be able to monitor its activity. You can do this with nginx-rrd. I won't go too much into the details, just make sure you don't make your status page world readable.

Installing WordPress

Despite the recurring security issues of WordPress, we didn't want to switch to another engine. WordPress is full featured, very well supported and well, Matt already did an incredible layout for the blog...

Making WordPress work with nginx is everything but straightforward. The good news is that we've already done the toughest.

Before installing WordPress, create a dedicated database and user with MySQL, as using root to access a shared database is a very bad idea (bis repetita placent).

Once you have your database credentials working, you can install WordPress in following the infamous 5-Minute install procedure.

WordPress only requires you to edit one configuration file: wp-config.php. You will also see four entries named AUTH_KEY, SECURE_AUTH_KEY, LOGGED_IN_KEY and NONCE_KEY require a passphrase. Generate a cryptographically strong passphrase with the following command:

The last line - the hexadecimal number - is your secure passphrase. Now your friends will look at you differently.

For our setup we have only one mandatory plugin: the nginx compatibility one. Should you wish to increase performance, WP Super Cache is highly recommended.

Installing memcached?

If you run a high-traffic web site, you will need to install a cache server.

memcached is supported by WordPress through this plugin. I do not know how it cooperates with WP Super Cache, but I submit the two are mutually exclusive.

At Bureau 14, we don't use memcached for the very simple reason we've developed our own high-performance cache engine that runs faster and scales better. It also comes with more fur. Don't worry, we won't keep this software to ourselves for ever: we will run a closed beta this summer. Should you wish to know more, feel free to subscribe to our mailing list.

Using memcached (or any cache engine) for static files generally yields no performance improvement. The reason is that nginx already caches static files.

Closing words

Although this setup is not as straightforward as the classic Linux/Apache combo, I truly think you will find the trouble to be worth it.

We surely do.

Do you need help tuning and securing your servers? We can help.