Byte Order

Part two of Writing Cross-Platform Software: Getting Started

This is an attempt to stop a war. I hope it is not too late and that somehow, magically perhaps, peace will prevail again.
The latecomers into the arena believe that the issue is: "What is the proper byte order in messages?".

I'm For Hire!

Do you need a Cross-Platform Software Engineer?

The root of the conflict lies much deeper than that. It is the question of which bit should travel first, the bit from the little end of the word, or the bit from the big end of the word? The followers of the former approach are called the Little-Endians, and the followers of the latter are called the Big-Endians. The details of the holy war between the Little-Endians and the Big-Endians are documented in [6][1] and described in brief, in the Appendix. I recommend you read it at this point.

-- Danny Cohen ON HOLY WARS AND A PLEA FOR PEACE, Internet Experiment Note 137

There is a lot to know about cross-platform development so I am going to start with the basic low-level details and over the course of a few columns work towards larger architectural issues.

Most Linux users use the x86 instruction set, which uses little-endian byte order. What this means is that the most significant byte of a multiple byte value occurs at the lowest address. Little-endian byte order became established early, I understand, because it is easer to implement in hardware; that makes little difference with today's manufacturing technology but is so prevalent because it was used in both Vaxen and the 8086 processors of the first IBM PC's.

If we have:

short example[ 2 ] = { 0x0001, 0x0302 };

then expressed as hexadecimal with addresses reading from left to right, the data in memory is:

01 00 02 03

PowerPC and Sparc chips on the other hand are big-endian; the most significant byte is at the lower address. A little bit of extra circuitry is required to implement this in a microprocessor but this is not significant in today's processors in comparison to such things as speculative execution and cache control. The biggest advantage is that it is much easier to understand the data when viewing it in a low-level debugger or file hex dump or network sniffer display.

For big-endian, the example above becomes:

00 01 03 02

(Because raw data is much easier to debug in big-endian format I suggest using big-endian file formats when you have the opportunity to design new ones regardless of the processor architecture.)

I will leave it to the gentle reader's horrified imagination to consider what other byte orders could be and in fact have been used on older processors. We can be grateful that the two arrangements in current use are in strictly monotonic order. There are only two ways to rearrange a two-byte integer, but there are lots of ways you could express a four-byte value. As far as I know, the less obvious formats are no longer in use. Also note that the internet protocol standards are written in terms of "octets" of data rather than "bytes" because a byte is not always eight bits.

There is a trick in common use in little-endian code that is a serious cross-platform no-no; that is casting a pointer to a long to a pointer to a char and assuming that the least-significant byte will be at the address pointed to:

unsigned long value = 0x03020100 unsigned long *ptr = &value; unsigned char charVal; charVal = *(unsigned char*)ptr;

On a little-endian processor charVal will become 0. On a big-endian processor it will become 3. Clearly you do not want to use such code in your program but it is very common, and in old code written for one little-endian platform it is one of the hardest things to find and root out.

To accomplish the same thing in a portable way we use a temporary variable:

unsigned long temp = *ptr; charVal = (unsigned char)temp;

The second line above will take its value from the least significant byte on every architecture whether it is at the high or low end of the temporary; the compiler handles the details for you.

It is important to consider byte order carefully when reading and writing data to files or to the network. In general, you should not blindly read or write a multiple-byte value to a file or socket without preprocessing it in some way. What "in some way" means depends on the file format and architecture of your program. Do not do this:

long val = 1; int result = write( fileDes, &val, sizeof( val ) );

Because when the result is read back in, possibly on a different machine across the network, or maybe years down the road on some machine reading a file that has been burned into a CD:

long valRead; int result = read( fileDes, &valRead, sizeof( valRead ) );

The contents of valRead will be 1 if the machines have the same byte order, but 0x01000000 if they are different (but both are 32 bit machines).

(A note on using ints as I do in my sample code - they are generally to be avoided. Choose shorts or longs, or better still use a typedef to the desired type in a header file so it can be redefined for new platforms. I use ints here because the Unix system call API's are defined to return ints. You cannot in general count on an int having the same size even between different compilers on the same machine. You should only use them if an API is defined to use them, or for variables like loop counters that are best left to be the "natural machine word" as defined by the compiler programmer, and where you are sure the range will not exceed the smallest range that your code could conceivably encounter.)

There are some choices as to how you handle byte order. One is to decide on a certain byte order for all external data and to require swapping of all values written and read on a machine with a different order. This is done by the TCP/IP protocols, for example - all packet header data is big-endian (even if the payload data is little endian) so it happens that since the vast majority of hosts connected to the Internet are little-endian x86 machines, most Internet packets get their headers swapped during transmission and receipt.

Another is to allow data to be read and written in any order, which allows an application to write only in its native order, which allows some simplification and a little bit of extra efficiency during writing (but usually not that much). The Tagged Image File Format specifies this - TIFF readers have to accept data in either byte order. The first two bytes of a TIFF file are two ASCII characters, either "MM" or "II" to indicate Motorola or Intel byte order. The next four bytes are the ASCII characters "TIFF" expressed in the given byte order.

Fancier TIFF format writers will allow you to save in either order but the free libtiff library written by Sam Leffler saves in native byte order and reads in either order. Interestingly, the byte order of the host machine is not a normal configuration option but is discovered by the configure script which spits a short c program out to a file, compiles it and checks the result of running it:

CheckForBigEndian() { echo 'main() { int one = 1; char* cp = (char*)&one; exit(*cp!=0); }'>t.c capture cat t.c runMake t "t:; ${CCOMPILER} ${ENVOPTS} t.c" && ./a.out }

I suggest you use such a trick in your own code to ease configuration and reduce user error. It is troublesome to depend on a script to configure your program - not all platforms have shell scripts or run them the same way. You can discover the byte order at runtime perhaps:

bool gIsBigEndian; void InitializeEndianFlag() { short one = 1; char *cp = (char*)&one; if ( *cp == 0 ) gIsBigEndian = true; else gIsBigEndian = false; return; }

Does byte-swapping affect efficiency? In most real-world situations, the answer is no. The code required to swap a two or four byte value is just a few instructions and is easily done entirely in the registers. If you have a lot of data to swap, such as the pixel format for an image, all of the code will fit in a small loop that takes up just a couple cache lines, and the data can be fetched sequentially from the data cache, which is very efficient.

The vast majority of your program's time will be spent doing more complex things. Be more concerned with getting your data format correct and portable than getting it efficient; there will be better places to devote labor to optimizing your code.

(It is my experience that many programmers do not know how to optimize well, and labor greatly to tighten their code in ways that have little effect. It is best to use profilers to find hot spots where you should concentrate your effort, and to improve your algorithms, rather than fussing with low level details. The exceptions to this would come in such cases as image blitting routines for video games, where it probably is worthwhile to write tightly coded special-case code devoted to each platform in use.)

Notes

[1] Cohen's reference [6] is Gulliver's Travels by Jonathon Swift, which describes a war between those who crack their boiled eggs from the little end vs. those who prefer to crack the big end.

Voting for GoingWare at The Programming Pages will encourage more people to read these articles.