One of the many qualities of Production Quality Code is to be able to handle all possible errors gracefully and recovering if possible. Even if these error have a one in a million chance of happening, a good production quality code requires programmers to handle the errors.

Error handling code either take the form of a big if-then-else statement or a more graceful exception handling mechanism. Error handling happens when a certain operation fails to perform as expected. For every such operation there are generally many failure reasons. These failure reasons are handled in the else part or the catch block. It may turn out that in the process of handling all error conditions or exceptions, a lot of code has been written and the size of these error handling routines is substantial.

I have wondered many times : "How much of such error handling code and variables are loaded on my RAM ?" The probability of these code+data being used is very rare. They are just sitting on my RAM doing nothing? With 50+ processes running on my home PC, how much of my 1GB RAM is dedicated to holding memory for these error/exception handling routines? But you cannot ignore these code, as you want to handle a bad condition gracefully.

What is needed is a processor architecture that allows programmers to specify via a high level languages these low probability code segments. I am not talking of the likely and the unlikely macros which use the processor's branch prediction capability. They help improve performance but not by saving RAM.

The architecture should not load those less-probable code into memory unless the error or the exception condition is met. While working for Nortel Networks in the beginning of my career, I had come across a similar architecture and there was a possibility in the language (Protel, their in-house proprietary programming language) to do precisely that. I don't remember much now.

I believe that it wont be easy to implement such a thing, but it would be a very much welcome thing, given the fact that almost all computer users today crib about their RAM usage more than the HDD or the CPU.

Thoughts?

After 2 days of struggling with Intel VTune on my Linux VMWare workstation and after trying several versions of VTune, I learned the hard way that you cannot get the Sampling data from VTune if you are running it on VMware.

I wish VTune would give me a better error message then just saying "Failed to create sampling data base. Probably .tb5 files are corrupted or don't exist." and "The Sampling Collector failed to collect data because the selected event(s) did not occur."

Looking at other people who were facing the same problem, the common suggestions were to try a different version of VTune. Yeah, thats where I wasted my time -- the error on .tb5 files was a bit misleading.

Thanks to Intel Forum post, I found and confirmed that Intel VTune is not supported on VMware. Here is another explanation from Intel on this. Having read these, it is understandable why VTune is not supported on any Virtual system. With Virtulization technology becoming so popular among developers, I wish Intel can do a bit more to let know that it is not supported.

The only good news is that you can still generate call graphs using VTune on VMware, but thats not the reason why you want to use VTune.

Today discussion is around programming in C/C++ and file IO.

Almost all programs require reading files from disk to perform their duties. I have been using fread(), read() or C++'s fstream classes to perform the file IO. The way these file IO works is : Data is copied from the disk to a kernel buffer, then the kernel buffer is copied into the process's heap space for use. The data as you see is copied twice. These file IO is always buffered and the amount of buffering differs from one OS's implementation to another.

This works good for small files, files which programs read for configuration details and writing log files, etc.

But what about large data file? A file which is in the order of a few MBs can choke the system. In most cases (maybe except for photo editors, or video programs) it is generally not required to get the entire data for a particular operation. We sometimes need to access random parts of a large file at random times.

Try to solve this problem with the tradition fread/fseek calls and you will end up wasting too much CPU and memory.

The concept of memory mapping of file allows the program to copy the data from the disk straight to the process space. This allows the program to view the contents of the file as an array -- you get the pointer to the first byte of the file. This translates to better usage of CPU and RAM.

In UNIX these can be accomplished by the mmap() function and in Windows its the MapViewOfFile() function. Using these functions is not as easy as using the traditional reads as the programmer may need to take care of certain things:
1. You may need to map only a particular section of the huge file into memory and need to do the alignment with system pagesize.
2. Delicate pointer handling and proper usage of read write permissions.
3. Make sure you dont map a very huge file into your process space, which can lead to too many page faults.
4. Never write pointers into these files. Keep only offsets if you need.

It also allows you to dump serialized objects which you can directly load into memory and cast it to the required type and start working right away.

This technique is only suitable for certain cases and the programmer must make sure that he/she gets measurable advantage using memory mapped files.

Suggested Reading:
1.Sun Developer Network : A Performance Comparison of "read" and "mmap"
2.Pros and Cons of Memory mapped file and when not to use memory mapping.
3. Memory Mapped Files And Shared Memory For C++.

All Unix users are aware of the DAC (-rwxrwxrwx) that we always use to prevent others from accessing files we don't want them to. But there are times when you want more than that.

You may say:
1. I want to prevent myself from accidentally deleting a file despite having -rwxrwxrwx permissions.

2. If you have 000 permissions for a file, then you cant delete it without changing the permissions first. But you can straight away do a "rm -f" .Many people are just used to using the "f" flag(I sometimes wonder if the "f" flag meant "force" or something else). If you are a Linux sys-admin and you are sharing the root password with fellow admins, then you definitely need something to warn your fellow-admins from doing something wrong accidentally. The traditional DAC does not allow you to this. For DAC "root" is as good as God.

3. I want to prevent every body (including root) from deleting the file or writing to file, BUT YOU WANT TO ALLOW THEM TO ADD DATA TO FILE. ie, allow only appending.

A very less know Linux command "chattr" allows you to do precisely these by making the file immutable. This works only for EXT2/ETX3 file-sytems only. If you set the immutable attribute for a file, it means that it cannot be modified, deleted, renamed and no hard links can be created to this file.

Even a root user attempting to delete this file with get the message "Operation not permitted". Only the superuser or a process possessing the CAP_LINUX_IMMUTABLE capability can set or clear this attribute.

chattr +i <filename> sets the immutable attribute of the file.
chattr -i <filename> clears it.
lasttr <filename> will show the status of the attributes.

This not-so-popular command has got a bunch of other attributes that you can set either to improve the security of your system or even to increase performance.

Using the +a attribute you can make the file to be opened only in append mode. (-a to remove the attribute.

For a security sensitive file-system you may want to set the "+s" attribute which will make sure that when you delete the file, the contents in the hard disk are all zeroed out.

Read the man pages of lsattr and chattr for more details and limitations/bugs. The full features of chattr are not yet available.

As promised in my post on Thoughts on password management here is a step by step guide on how someone can capture your login id and password while you login into a website that does not support HTTPS (HTTP over SSL).

A little basics first. HTTP protocol is based on clear text interchange of data. It is an application layer protocol with TCP and IP in the underlying layer. It means that anybody who can capture a HTTP packet traveling over an Ethernet network can read all information you sent.

When you type http://blog.netotto.com/index.php, your browser sends the following request:

    GET / HTTP/1.1\r\n
        Request Method: GET
        Request URI: /index.php
        Request Version: HTTP/1.1
    Accept: */*\r\n
    Accept-Language: en-us\r\n
    Accept-Encoding: gzip, deflate\r\n
    User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)\r\n
    Host: blog.netotto.com\r\n
    Connection: Keep-Alive\r\n

This is a HTTP GET request, the important part is just the GET and the Host lines. The server responds to this again in clear text, that your browser can understand. Here is how it looks:

   HTTP/1.1 200 OK\r\n
        Request Version: HTTP/1.1
        Response Code: 200
    Date: Sat, 15 Sep 2007 10:41:22 GMT\r\n
    Server: Apache/2.0.54 (Unix) PHP/4.4.7 mod_ssl/2.0.54 OpenSSL/0.9.7e mod_fastcgi/2.4.2 DAV/2 SVN/1.4.2\r\n
    X-Powered-By: PHP/5.2.3\r\n
    Expires: Thu, 19 Nov 1981 08:52:00 GMT\r\n
    Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0\r\n
    Pragma: no-cache\r\n
    Vary: Accept-Encoding\r\n
    Content-Encoding: gzip\r\n
    Content-Length: 6611\r\n
    Keep-Alive: timeout=2, max=100\r\n
    Connection: Keep-Alive\r\n
    Content-Type: text/html\r\n
    \r\n

    Content-encoded entity body (gzip): 6611 bytes -> 26607 bytes

This is followed by the HTML page (index.html) in clear text.

If you are at the login page of a non-secure web page your login and passwords can easily be read by somebody on the network. All you need is a network packet capturing tool.

On an Ethernet, the electrical signals (to which your HTTP packet finally resolves to while traveling on the wire) is actually readable by all the computers in your Ethernet LAN. The network card in normal operation mode (called non-promiscuous mode) does not read packets that are destined for somebody else.

A packet capturing tool like Ethereal can ask your network card to be in promiscuous mode and read all packets. These includes packets destined for other machines in your LAN.

You can download and install Ethereal for Windows from the Ethereal Download Page and install it and follow the following steps to convince yourself that sending data over a non-secure channel (a non HTTPS website for example) is very un-safe.

Lets jump straight into the steps:

In this example, I have installed Ethereal on the same computer from where I will be attempting to login to a non-ssl website like www.youtube.com . You can however run the Browser and Ethereal on different computers in your LAN which are in the same subnet (Eg: Browser on your computer at office and Ethereal on your colleague's computer).

1. Open the youtube login page (http://youtube.com/login?next=/index) in your browser. Note: its http and not https. I type in "shuva" and password as "testpassword"

2. Start Ethereal (I am using version 0.99.0)

3. Lets configure it so that it captures only HTTP packets. We dont want to capture hundreds of other packets. From the menu bar, select Capture--->Options. In the Options dialog box, choose the correct n/w interface if you have more than one. I have something like "Intel Pro/100 VE". In the capture filter type in "tcp port http". This means we will capture only http packets.

4. Click the start button to start the capture.

5. Click on the login button on your browser.

6. Back to Ethreal, click the stop button on the popped up dialog box.

The Ethereal window is divided into 3 parts,

the upper part listing the packets,

the middle part showing the selected packet in human readable form and properly divided into the layers (HTTP/TCP/IP/Ethernet/Frame) and

the lower part showing the raw bytes of the selected packet.

In the upper part, where you many lines with lots of IP addresses, search for the line that has "HTTP" and "POST /login?"

In the middle part click on "+" sign that says "Line-based text data". There is your login and the password. Just above the "Line-based text data" is the HTTP request that you sent to youtube.

Here is a screen shot:



Below is the HTTP request that went through the network:

    POST /login?next=/index HTTP/1.1\r\n
        Request Method: POST
        Request URI: /login?next=/index
        Request Version: HTTP/1.1
    Host: youtube.com\r\n
    User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6\r\n
    Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5\r\n
    Accept-Language: en-us,en;q=0.5\r\n
    Accept-Encoding: gzip,deflate\r\n
    Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n
    Keep-Alive: 300\r\n
    Connection: keep-alive\r\n
    Referer: http://youtube.com/login?next=/index\r\n
    Cookie: dkv=; GEO=a78121630c1e9382b3cf8d384cd0d399cxYAAABJTixhcCxoeWRlcmFiYWQsLCwsLC0x; LOCALE_PREFERENCE=86d1d09eefe6b79b4068000ce05518a4dAUAAABlbl9VUw==; LOGIN_INFO=; VISITOR_INFO1_LIVE=D6Jub46OfWo; use_hitbox=72c46ff6cbcdb7c5585c36411b6
    Content-Type: application/x-www-form-urlencoded\r\n
    Content-Length: 93\r\n
    \r\n
current_form=loginForm&next=%2Findex&username=shuva&password=testpassword&action_login=Log+In

Disclaimer: My selection of youtube.com for this demo is not to prove that youtube.com is unsafe. I just wanted a very well known website. My intention of writing this article is not to encourage people to read somebody's else packet from the network but to demonstrate to users that visits to non-secure sites are not safe and you should therefore never select a password for such sites which you are reusing for a secure site like your email or banking sites.

Relevant posts:
1. Thoughts on password management.
2. Storing all your password safely.