Difference between pages "Awk by Example, Part 1" and "POSIX Threads Explained, Part 1"

(Difference between pages)
m (add highlighting ##i##)
 
 
Line 1: Line 1:
 
{{Article
 
{{Article
 +
|Summary=POSIX (Portable Operating System Interface) threads are a great way to increase the responsiveness and performance of your code. In this series, Daniel Robbins shows you exactly how to use threads in your code. A lot of behind-the-scenes details are covered, so by the end of this series you'll really be ready to create your own multithreaded programs.
 
|Author=Drobbins
 
|Author=Drobbins
|Next in Series=Awk by Example, Part 2
 
 
}}
 
}}
{{WikiArticle}}
+
== Threads are fun ==
  
== An intro to the great language with the strange name ==
+
Knowing how to properly use threads should be part of every good programmer's repertoire. Threads are similar to processes. Threads, like processes, are time-sliced by the kernel. On uniprocessor systems the kernel uses time slicing to simulate simultaneous execution of threads in much the same way it uses time slicing with processes. And, on multiprocessor systems, threads are actually able to run simultaneously, just like two or more processes can.
  
=== In defense of awk ===
+
So why is multithreading preferable to multiple independent processes for most cooperative tasks? Well, threads share the same memory space. Independent threads can access the same variables in memory. So all of your program's threads can read or write the declared global integers. If you've ever programmed any non-trivial code that uses fork(), you'll recognize the importance of this tool. Why? While fork() allows you to create multiple processes, it also creates the following communication problem: how to get multiple processes, each with their own independent memory space, to communicate. There is no one simple answer to this problem. While there are many different kinds of local IPC (inter-process communication), they all suffer from two important drawbacks:
In this series of articles, I'm going to turn you into a proficient awk coder. I'll admit, awk doesn't have a very pretty or particularly "hip" name, and the GNU version of awk, called gawk, sounds downright weird. Those unfamiliar with the language may hear "awk" and think of a mess of code so backwards and antiquated that it's capable of driving even the most knowledgeable UNIX guru to the brink of insanity (causing him to repeatedly yelp "kill -9!" as he runs for coffee machine).
+
  
Sure, awk doesn't have a great name. But it is a great language. Awk is geared toward text processing and report generation, yet features many well-designed features that allow for serious programming. And, unlike some languages, awk's syntax is familiar, and borrows some of the best parts of languages like C, python, and bash (although, technically, awk was created before both python and bash). Awk is one of those languages that, once learned, will become a key part of your strategic coding arsenal.
+
* They impose some form of additional kernel overhead, lowering performance.
 +
* In almost all situations, IPC is not a "natural" extension of your code. It often dramatically increases the complexity of your program.
  
=== The first awk ===
 
Let's go ahead and start playing around with awk to see how it works. At the command line, enter the following command:
 
  
<console>$##i## awk '{ print }' /etc/passwd</console>
+
Double bummer: overhead and complication aren't good things. If you've ever had to make massive modifications to one of your programs so that it supports IPC, you'll really appreciate the simple memory-sharing approach that threads provide. POSIX threads don't need to make expensive and complicated long-distance calls because all our threads happen to live in the same house. With a little synchronization, all your threads can read and modify your program's existing data structures. You don't have to pump the data through a file descriptor or squeeze it into a tight, shared memory space. For this reason alone you should consider the one process/multithread model rather than the multiprocess/single-thread model.
  
You should see the contents of your /etc/passwd file appear before your eyes. Now, for an explanation of what awk did. When we called awk, we specified /etc/passwd as our input file. When we executed awk, it evaluated the print command for each line in /etc/passwd, in order. All output is sent to stdout, and we get a result identical to catting /etc/passwd.
+
== Threads are nimble ==
  
Now, for an explanation of the { print } code block. In awk, curly braces are used to group blocks of code together, similar to C. Inside our block of code, we have a single print command. In awk, when a print command appears by itself, the full contents of the current line are printed.
+
But there's more. Threads also happen to be extremely nimble. Compared to a standard fork(), they carry a lot less overhead. The kernel does not need to make a new independent copy of the process memory space, file descriptors, etc. That saves a lot of CPU time, making thread creation ten to a hundred times faster than new process creation. Because of this, you can use a whole bunch of threads and not worry too much about the CPU and memory overhead incurred. You don't have a big CPU hit the way you do with fork(). This means you can generally create threads whenever it makes sense in your program.
  
Here is another awk example that does exactly the same thing:
+
Of course, just like processes, threads will take advantage of multiple CPUs. This is a really great feature if your software is designed to be used on a multiprocessor machine (if the software is open source, it will probably end up running on quite a few of these). The performance of certain kinds of threaded programs (CPU-intensive ones in particular) will scale almost linearly with the number of processors in the system. If you're writing a program that is very CPU-intensive, you'll definitely want to find ways to use multiple threads in your code. Once you're adept at writing threaded code, you'll also be able to approach coding challenges in new and creative ways without a lot of IPC red tape and miscellaneous mumbo-jumbo. All these benefits work synergistically to make multithreaded programming fun, fast, and flexible.
  
<console>$##i## awk '{ print $0 }' /etc/passwd</console>
+
== I think I'm a clone now ==
  
In awk, the $0 variable represents the entire current line, so print and print $0 do exactly the same thing. If you'd like, you can create an awk program that will output data totally unrelated to the input data. Here's an example:
+
If you've been in the Linux programming world for a while, you may know about the <code>__clone()</code> system call. <code>__clone()</code> is similar to <code>fork()</code>, but allows you to do lots of things that threads can do. For example, with {{c|__clone()}} you can selectively share parts of your parent's execution context (memory space, file descriptors, etc.) with a new child process. That's a good thing. But there is also a not-so-good thing about {{c|__clone()}}. As the {{c|__clone()}} man page states:
  
<console>$##i## awk '{ print "" }' /etc/passwd</console>
+
<blockquote>
 +
"The __clone call is Linux-specific and should not be used in programs
 +
intended to be portable. For programming threaded applications (multiple
 +
threads of control in the same memory space), it is better to use a library
 +
implementing the POSIX 1003.1c thread API, such as the Linux-Threads
 +
library. See pthread_create(3thr)."
 +
</blockquote>
  
Whenever you pass the "" string to the print command, it prints a blank line. If you test this script, you'll find that awk outputs one blank line for every line in your /etc/passwd file. Again, this is because awk executes your script for every line in the input file. Here's another example:
+
So, while {{c|__clone()}} offers many of the benefits of threads, it is not portable. That doesn't mean you shouldn't use it in your code. But you should weigh this fact when you are considering using {{c|__clone()}} in your software. Fortunately, as the __clone() man page states, there's a better alternative: POSIX threads. When you want to write portable multithreaded code, code that works under Solaris, FreeBSD, Linux, and more, POSIX threads are the way to go.
  
<console>$##i## awk '{ print "hiya" }' /etc/passwd</console>
+
== Beginning threads ==
  
Running this script will fill your screen with hiya's. :)
+
Here's a simple example program that uses POSIX threads:
  
=== Multiple fields ===
+
{{file|name=thread1.c|lang=c|desc=Sample program using POSIX threads|body=
Awk is really good at handling text that has been broken into multiple logical fields, and allows you to effortlessly reference each individual field from inside your awk script. The following script will print out a list of all user accounts on your system:
+
#include <pthread.h>
 +
#include <stdlib.h>
 +
#include <unistd.h>
  
<console>$##i## awk -F":" '{ print $1 }' /etc/passwd</console>
+
void *thread_function(void *arg) {
 +
  int i;
 +
  for ( i=0; i<20; i++ ) {
 +
    printf("Thread says hi!\n");
 +
    sleep(1);
 +
  }
 +
  return NULL;
 +
}
  
Above, when we called awk, we use the -F option to specify ":" as the field separator. When awk processes the print $1 command, it will print out the first field that appears on each line in the input file. Here's another example:
+
int main(void) {
  
<console>$##i## awk -F":" '{ print $1 $3 }' /etc/passwd</console>
+
  pthread_t mythread;
 +
 
 +
  if ( pthread_create( &mythread, NULL, thread_function, NULL) ) {
 +
    printf("error creating thread.");
 +
    abort();
 +
  }
  
Here's an excerpt of the output from this script:
+
  if ( pthread_join ( mythread, NULL ) ) {
<pre>
+
    printf("error joining thread.");
halt7
+
    abort();
operator11
+
  }
root0
+
shutdown6
+
sync5
+
bin1
+
....etc.
+
</pre>
+
As you can see, awk prints out the first and third fields of the /etc/passwd file, which happen to be the username and uid fields respectively. Now, while the script did work, it's not perfect -- there aren't any spaces between the two output fields! If you're used to programming in bash or python, you may have expected the print $1 $3 command to insert a space between the two fields. However, when two strings appear next to each other in an awk program, awk concatenates them without adding an intermediate space. The following command will insert a space between both fields:
+
  
<console>$##i## awk -F":" '{ print $1 " " $3 }' /etc/passwd</console>
+
  exit(0);
  
When you call print this way, it'll concatenate $1, " ", and $3, creating readable output. Of course, we can also insert some text labels if needed:
+
}
 +
}}
  
<console>$##i## awk -F":" '{ print "username: " $1 "\t\tuid:" $3 }' /etc/passwd</console>
+
To compile this program, simply save this program as {{c|thread1.c}}, and type:
  
This will cause the output to be:
+
<console>
<pre>
+
$ ##i##gcc thread1.c -o thread1 -lpthread
username: halt    uid:7
+
</console>
username: operator uid:11
+
username: root    uid:0
+
username: shutdown uid:6
+
username: sync    uid:5
+
username: bin      uid:1
+
....etc.  
+
</pre>
+
  
=== External Scripts ===
+
Run it by typing:
Passing your scripts to awk as a command line argument can be very handy for small one-liners, but when it comes to complex, multi-line programs, you'll definitely want to compose your script in an external file. Awk can then be told to source this script file by passing it the -f option:
+
  
<console>$##i## awk -f myscript.awk myfile.in </console>
+
<console>
 +
$ ##i##./thread1
 +
</console>
  
Putting your scripts in their own text files also allows you to take advantage of additional awk features. For example, this multi-line script does the same thing as one of our earlier one-liners, printing out the first field of each line in /etc/passwd:
+
== Understanding thread1.c ==
<pre>
+
BEGIN {
+
        FS=":"
+
}
+
{ print $1 }
+
</pre>
+
The difference between these two methods has to do with how we set the field separator. In this script, the field separator is specified within the code itself (by setting the FS variable), while our previous example set FS by passing the -F":" option to awk on the command line. It's generally best to set the field separator inside the script itself, simply because it means you have one less command line argument to remember to type. We'll cover the FS variable in more detail later in this article.
+
  
It is also possible to make the script directly executable, by placing a "#!/usr/bin/awk -f" at the top of the file, as follows:
+
{{c|thread1.c}} is an extremely simple threaded program. While it doesn't do anything useful, it'll help you understand how threads work. Let's take a step-by-step look at what this program does. In {{c|main()}} we first declare a variable called {{c|mythread,}} which has a type of {{c|pthread_t}}. The pthread_t type, defined in pthread.h, is often called a "thread id" (commonly abbreviated as "tid"). Think of it as a kind of thread handle.
<pre>
+
#!/usr/bin/awk -f
+
BEGIN {
+
FS=":"
+
}
+
{ print $1 }
+
</pre>
+
Next, the script must be made executable by setting the script file's execute bit:
+
  
<console>$##i## chmod +x myscript.awk</console>
+
After mythread is declared (remember that mythread is just a "tid", or a handle to the thread we are about to create), we call the {{c|pthread_create}} function to create a real, living thread. Don't be confused by the fact that pthread_create() is inside an "if" statement. Since pthread_create() returns zero on success and a non-zero value on failure, placing the function call in an if() is just an elegant way of detecting a failed pthread_create() call. Let's look at the arguments to pthread_create. The first one is a pointer to mythread, {{c|&mythread}}. The second argument, currently set to NULL, can be used to define certain attributes for our thread. Because the default thread attributes work for us, we simply set it to NULL.
  
Now, you should be able to execute the script as follows:
+
Our third argument is the name of a function that the new thread will execute when it starts. In this case the name of the function is thread_function(). When this thread_function() returns, our new thread will terminate. In this example our thread function doesn't do anything remarkable. It just prints out "Thread says hi!" 20 times and then exits. Notice that thread_function() accepts a void * as an argument and also returns a void * as a return value. This shows us that it's possible to use a void * to pass an arbitrary piece of data to our new thread, and that our new thread can return an arbitrary piece of data when it finishes. Now, how do we pass our thread an arbitrary argument? Easy. We use the fourth argument to the pthread_create() call. In this example, we set it to NULL because we don't need to pass any data into our trivial thread_function().
  
<console>$##i## ./myscript.awk myfile.in</console>
+
As you may have guessed, the program will consist of two threads after pthread_create() successfully returns. Wait a minute, two threads? Didn't we just create a single thread? Yes, we did. But our main program is also considered a thread. Think of it this way: if you write a program and don't use POSIX threads at all, the program will be single-threaded (this single thread is called the "main" thread). By creating a new thread we now have a total of two threads in our program.
  
=== The BEGIN and END blocks ===
+
I imagine you have at least two important questions at this point. The first is what the main thread does after the new thread is created. It keeps on going and executes the next line of our program in sequence (in this case that line is "if ( pthread_join(...))"). The second question you may be wondering about is what happens to our new thread when it exits. It stops and waits to be merged or "joined" with another thread as part of its cleanup process.
Normally, awk executes each block of your script's code once for each input line. However, there are many programming situations where you may need to execute initialization code before awk begins processing the text from the input file. For such situations, awk allows you to define a BEGIN block. We used a BEGIN block in the previous example. Because the BEGIN block is evaluated before awk starts processing the input file, it's an excellent place to initialize the FS (field separator) variable, print a heading, or initialize other global variables that you'll reference later in the program.
+
  
Awk also provides another special block, called the END block. Awk executes this block after all lines in the input file have been processed. Typically, the END block is used to perform final calculations or print summaries that should appear at the end of the output stream.
+
OK, now on to {{c|pthread_join()}}. Just as pthread_create() split our single thread into two threads, pthread_join() merges two threads into a single thread. The first argument to pthread_join() is our tid mythread. The second argument is a pointer to a void pointer. If the void pointer is non-NULL, pthread_join will place our thread's void * return value at the location we specify. Since we don't care about thread_function()'s return value, we set it to NULL.
  
=== Regular expressions and blocks ===
+
You'll notice that our thread_function() takes 20 seconds to complete. Long before thread_function() completes, our main thread has already called pthread_join(). When this occurs our main thread will block (go to sleep) and wait for thread_function() to complete. When thread_function() completes, pthread_join() will return. Now our program has one main thread again. When our program exits, all new threads have been pthread_join()ed. This is exactly how you should deal with every new thread you create in your programs. If a new thread isn't joined it will still count against your system's maximum thread limit. This means that not doing the proper cleanup will eventually cause new pthread_create() calls to fail.
Awk allows the use of regular expressions to selectively execute an individual block of code, depending on whether or not the regular expression matches the current line. Here's an example script that outputs only those lines that contain the character sequence foo:
+
  
<pre>/foo/ { print }</pre>
+
== No parents, no children ==
  
Of course, you can use more complicated regular expressions. Here's a script that will print only lines that contain a floating point number:
+
If you've used the fork() system call you're probably familiar with the concept of parent and child processes. When a process creates another new process, using fork(), the new process is considered the child and the original process is considered the parent. This creates a hierarchical relationship that can be handy, especially when waiting for child processes to terminate. The {{c|waitpid()}} function, for example, will tell the current process to wait for the termination of any child processes. waitpid() is used to implement a simple cleanup routine in your parent process.
  
<pre>/[0-9]+\.[0-9]*/ { print }</pre>
+
Things are a little more interesting with POSIX threads. You may have noticed that I have intentionally avoided using the terms "parent thread" and "child thread" so far. That's because with POSIX threads this hierarchical relationship doesn't exist. While a main thread may create a new thread, and that new thread may create an additional new thread, the POSIX threads standard views all your threads as a single pool of equals. So the concept of waiting for a child thread to exit doesn't make sense. The POSIX threads standard does not record any "family" information. This lack of genealogy has one major implication: if you want to wait for a thread to terminate, you need to specify which one you are waiting for by passing the proper tid to pthread_join(). The threads library can't figure it out for you.
  
=== Expressions and blocks ===
+
To many people this isn't very good news because it can complicate programs that consist of more than two threads. Don't let it worry you. The POSIX threads standard provides all the tools you need to manage multiple threads gracefully. Actually, the fact that there is no parent/child relationship opens up some creative ways to use threads in our programs. For example, if we have a thread called thread 1, and thread 1 creates a thread called thread 2, it's not necessary for thread 1 itself to call pthread_join() for thread 2. Any other thread in the program can do that. This allows for interesting possibilities when you write heavily multithreaded code. You can, for example, create a global "dead list" that contains all stopped threads and then have a special cleanup thread that simply waits for an item to be added to the list. The cleanup thread calls pthread_join() to merge it with itself. Now your entire cleanup will be handled neatly and efficiently in a single thread.
There are many other ways to selectively execute a block of code. We can place any kind of boolean expression before a code block to control when a particular block is executed. Awk will execute a code block only if the preceding boolean expression evaluates to true. The following example script will output the third field of all lines that have a first field equal to fred. If the first field of the current line is not equal to fred, awk will continue processing the file and will not execute the print statement for the current line:
+
  
<pre>$1 == "fred" { print $3 }</pre>
+
== Synchronized swimming ==
  
Awk offers a full selection of comparison operators, including the usual "==", "<", ">", "<=", ">=", and "!=". In addition, awk provides the "~" and "!~" operators, which mean "matches" and "does not match". They're used by specifying a variable on the left side of the operator, and a regular expression on the right side. Here's an example that will print only the third field on the line if the fifth field on the same line contains the character sequence root:
+
Now it's time to take a look at some code that does something a little unexpected. Here's thread2.c:
  
<pre>$5 ~ /root/ { print $3 }</pre>
+
{{file|name=thread2.c|lang=c|body=
 +
#include <pthread.h>
 +
#include <stdlib.h>
 +
#include <unistd.h>
 +
#include <stdio.h>
  
=== Conditional statements ===
+
int myglobal;
Awk also offers very nice C-like if statements. If you'd like, you could rewrite the previous script using an if statement:
+
<pre>
+
{
+
    if ( $5 ~ /root/ ) {
+
        print $3
+
    }
+
}
+
</pre>
+
Both scripts function identically. In the first example, the boolean expression is placed outside the block, while in the second example, the block is executed for every input line, and we selectively perform the print command by using an if statement. Both methods are available, and you can choose the one that best meshes with the other parts of your script.
+
  
Here's a more complicated example of an awk if statement. As you can see, even with complex, nested conditionals, if statements look identical to their C counterparts:
+
void *thread_function(void *arg) {
<pre>
+
  int i,j;
{
+
  for ( i=0; i<20; i++ ) {
     if ( $1 == "foo" ) {
+
     j=myglobal;
        if ( $2 == "foo" ) {
+
    j=j+1;
            print "uno"
+
    printf(".");
        } else {
+
     fflush(stdout);
            print "one"
+
  sleep(1);
        }
+
    myglobal=j;
     } else if ($1 == "bar" ) {
+
  }
        print "two"
+
  return NULL;
    } else {
+
        print "three"
+
    }
+
 
}
 
}
</pre>
 
Using if statements, we can also transform this code:
 
<pre>
 
! /matchme/ { print $1 $3 $4 }
 
</pre>
 
to this:
 
<pre>
 
{
 
    if ( $0 !~ /matchme/ ) {
 
        print $1 $3 $4
 
    }
 
}
 
</pre>
 
Both scripts will output only those lines that don't contain a matchme character sequence. Again, you can choose the method that works best for your code. They both do the same thing.
 
  
Awk also allows the use of boolean operators "||" (for "logical or") and "&&"(for "logical and") to allow the creation of more complex boolean expressions:
+
int main(void) {
<pre>
+
( $1 == "foo" ) && ( $2 == "bar" ) { print }
+
</pre>
+
This example will print only those lines where field one equals foo and field two equals bar.
+
  
=== Numeric variables! ===
+
  pthread_t mythread;
So far, we've either printed strings, the entire line, or specific fields. However, awk also allows us to perform both integer and floating point math. Using mathematical expressions, it's very easy to write a script that counts the number of blank lines in a file. Here's one that does just that:
+
   int i;
<pre>
+
BEGIN { x=0 }
+
/^$/  { x=x+1 }
+
END   { print "I found " x " blank lines. :)" }
+
</pre>
+
In the BEGIN block, we initialize our integer variable x to zero. Then, each time awk encounters a blank line, awk will execute the x=x+1 statement, incrementing x. After all the lines have been processed, the END block will execute, and awk will print out a final summary, specifying the number of blank lines it found.
+
  
=== Stringy variables ===
+
  if ( pthread_create( &mythread, NULL, thread_function, NULL) ) {
One of the neat things about awk variables is that they are "simple and stringy." I consider awk variables "stringy" because all awk variables are stored internally as strings. At the same time, awk variables are "simple" because you can perform mathematical operations on a variable, and as long as it contains a valid numeric string, awk automatically takes care of the string-to-number conversion steps. To see what I mean, check out this example:
+
    printf("error creating thread.");
<pre>
+
    abort();
x="1.01"  
+
  }
# We just set x to contain the *string* "1.01"
+
x=x+1
+
# We just added one to a *string*
+
print x
+
# Incidentally, these are comments :)  
+
</pre>
+
Awk will output:
+
<pre>
+
2.01
+
</pre>
+
Interesting! Although we assigned the string value 1.01 to the variable x, we were still able to add one to it. We wouldn't be able to do this in bash or python. First of all, bash doesn't support floating point arithmetic. And, while bash has "stringy" variables, they aren't "simple"; to perform any mathematical operations, bash requires that we enclose our math in an ugly $( ) construct. If we were using python, we would have to explicitly convert our 1.01 string to a floating point value before performing any arithmetic on it. While this isn't difficult, it's still an additional step. With awk, it's all automatic, and that makes our code nice and clean. If we wanted to square and add one to the first field in each input line, we would use this script:
+
<pre>
+
{ print ($1^2)+1 }
+
</pre>
+
If you do a little experimenting, you'll find that if a particular variable doesn't contain a valid number, awk will treat that variable as a numerical zero when it evaluates your mathematical expression.
+
  
=== Lots of operators ===
+
  for ( i=0; i<20; i++) {
Another nice thing about awk is its full complement of mathematical operators. In addition to standard addition, subtraction, multiplication, and division, awk allows us to use the previously demonstrated exponent operator "^", the modulo (remainder) operator "%", and a bunch of other handy assignment operators borrowed from C.
+
    myglobal=myglobal+1;
 +
    printf("o");
 +
    fflush(stdout);
 +
    sleep(1);
 +
  }
  
These include pre- and post-increment/decrement ( i++, --foo ), add/sub/mult/div assign operators ( a+=3, b*=2, c/=2.2, d-=6.2 ). But that's not all -- we also get handy modulo/exponent assign ops as well ( a^=2, b%=4 ).
+
  if ( pthread_join ( mythread, NULL ) ) {
 +
    printf("error joining thread.");
 +
    abort();
 +
  }
  
=== Field separators ===
+
  printf("\nmyglobal equals %d\n",myglobal);
Awk has its own complement of special variables. Some of them allow you to fine-tune how awk functions, while others can be read to glean valuable information about the input. We've already touched on one of these special variables, FS. As mentioned earlier, this variable allows you to set the character sequence that awk expects to find between fields. When we were using /etc/passwd as input, FS was set to ":". While this did the trick, FS allows us even more flexibility.
+
  
The FS value is not limited to a single character; it can also be set to a regular expression, specifying a character pattern of any length. If you're processing fields separated by one or more tabs, you'll want to set FS like so:
+
  exit(0);
<pre>
+
FS="\t+"
+
</pre>
+
Above, we use the special "+" regular expression character, which means "one or more of the previous character".
+
  
If your fields are separated by whitespace (one or more spaces or tabs), you may be tempted to set FS to the following regular expression:
+
}
<pre>
+
}}
FS="[[:space:]]+"
+
</pre>
+
While this assignment will do the trick, it's not necessary. Why? Because by default, FS is set to a single space character, which awk interprets to mean "one or more spaces or tabs." In this particular example, the default FS setting was exactly what you wanted in the first place!
+
  
Complex regular expressions are no problem. Even if your records are separated by the word "foo," followed by three digits, the following regular expression will allow your data to be parsed properly:
+
== Understanding thread2.c ==
<pre>
+
FS="foo[0-9][0-9][0-9]"
+
</pre>
+
  
=== Number of fields ===
+
This program, like our first one, creates a new thread. Both the main thread and the new thread then increment a global variable called myglobal 20 times. But the program itself produces some unexpected results. Compile it by typing:
The next two variables we're going to cover are not normally intended to be written to, but are normally read and used to gain useful information about the input. The first is the NF variable, also called the "number of fields" variable. Awk will automatically set this variable to the number of fields in the current record. You can use the NF variable to display only certain input lines:
+
<pre>
+
NF == 3 { print "this particular record has three fields: " $0 }
+
</pre>
+
Of course, you can also use the NF variable in conditional statements, as follows:
+
<pre>
+
{
+
    if ( NF > 2 ) {
+
        print $1 " " $2 ":" $3
+
    }
+
}
+
</pre>
+
  
=== Record number ===
+
<console>
The record number (NR) is another handy variable. It will always contain the number of the current record (awk counts the first record as record number 1). Up until now, we've been dealing with input files that contain one record per line. For these situations, NR will also tell you the current line number. However, when we start to process multi-line records later in the series, this will no longer be the case, so be careful! NR can be used like the NF variable to print only certain lines of the input:
+
$ ##i##gcc thread2.c -o thread2 -lpthread
<pre>
+
</console>
(NR < 10 ) || (NR > 100) { print "We are on record number 1-9 or 101+" }
+
 
</pre>
+
And run it:
<pre>
+
 
{
+
<console>
    #skip header
+
$ ##i##./thread2
    if ( NR > 10 ) {
+
..o.o.o.o.oo.o.o.o.o.o.o.o.o.o..o.o.o.o.o
        print "ok, now for the real information!"
+
myglobal equals 21
    }
+
</console>
}
+
 
</pre>
+
Quite unexpected! Since myglobal starts at zero, and both the main thread and the new thread each increment it by 20, we should see myglobal equaling 40 at the end of the program. Since myglobal equals 21, we know that something fishy is going on here. But what is it?
Awk provides additional variables that can be used for a variety of purposes. We'll cover more of these variables in later articles.
+
 
 +
Give up? OK, I'll show you why it happens. Take a look at thread_function(). Notice how we copy myglobal into a local variable called "j"? And see how we increment j, then sleep for one second, and only then copy our new j value into myglobal? That's the key. Imagine what happens if our main thread increments myglobal just after our new thread copies the value of myglobal into j. When thread_function() writes the value of j back to myglobal, it will overwrite the modification that our main thread made.
  
We've come to the end of our initial exploration of awk. As the series continues, I'll demonstrate more advanced awk functionality, and we'll end the series with a real-world awk application.
+
When writing threaded programs, you want to avoid useless side effects like the one we just looked at because they're a waste of time (except when you are writing articles on POSIX threads, of course :). Now, what can we do to eliminate this hassle?
  
== Resources ==
+
Since the problem occurs because we copy myglobal to j and hold it there for one second before writing it back, we could try to avoid using a temporary local variable and incrementing myglobal directly. While this solution will probably work in this particular example, it is not correct. And if we were performing a relatively complex mathematical operation on myglobal instead of just incrementing it, it would certainly fail. But why?
  
* Read Daniel's other awk articles on Funtoo: Awk By Example, [[Awk by example, Part2 |Part 2]] and [[Awk by example, Part3 |Part 3]].
+
To understand the problem, you need to remember that threads run simultaneously. Even on a uniprocessor machine (where the kernel uses time slicing to simulate true multitasking) we can, from a programmer's perspective, imagine both threads as executing at the same time. thread2.c has problems because the code in thread_function() relies on the fact that myglobal will not be modified during the ~1 second before it is incremented. We need some way for one thread to tell the other to "hold off" while it's making changes to myglobal. I'll show you exactly how to do this in my next article. See you then.
* If you'd like a good old-fashioned book, [http://www.oreilly.com/catalog/sed2/ O'Reilly's sed & awk, 2nd Edition] is a wonderful choice.
+
* Be sure to check out the [http://www.faqs.org/faqs/computer-lang/awk/faq/ comp.lang.awk FAQ]. It also contains lots of additional awk links.
+
* Patrick Hartigan's [http://sparky.rice.edu/~hartigan/awk.html awk tutorial] is packed with handy awk scripts.
+
* [http://www.tasoft.com/tawk.html Thompson's TAWK Compiler] compiles awk scripts into fast binary executables. Versions are available for Windows, OS/2, DOS, and UNIX.
+
* [http://www.gnu.org/software/gawk/manual/gawk.html The GNU Awk User's Guide] is available for online reference.
+
* [http://www.folkstalk.com/2011/12/good-examples-of-awk-command-in-unix.html Awk Command] daily useful examples.
+
[[Category:Linux Core Concepts]]
+
[[Category:Articles]]
+
 
{{ArticleFooter}}
 
{{ArticleFooter}}

Revision as of 07:36, December 31, 2014

POSIX (Portable Operating System Interface) threads are a great way to increase the responsiveness and performance of your code. In this series, Daniel Robbins shows you exactly how to use threads in your code. A lot of behind-the-scenes details are covered, so by the end of this series you'll really be ready to create your own multithreaded programs.

Support Funtoo and help us grow! Donate $15 per month and get a free SSD-based Funtoo Virtual Container.
Looking for people interested in testing and documenting Docker support! Contact Daniel Robbins for more info.

Threads are fun

Knowing how to properly use threads should be part of every good programmer's repertoire. Threads are similar to processes. Threads, like processes, are time-sliced by the kernel. On uniprocessor systems the kernel uses time slicing to simulate simultaneous execution of threads in much the same way it uses time slicing with processes. And, on multiprocessor systems, threads are actually able to run simultaneously, just like two or more processes can.

So why is multithreading preferable to multiple independent processes for most cooperative tasks? Well, threads share the same memory space. Independent threads can access the same variables in memory. So all of your program's threads can read or write the declared global integers. If you've ever programmed any non-trivial code that uses fork(), you'll recognize the importance of this tool. Why? While fork() allows you to create multiple processes, it also creates the following communication problem: how to get multiple processes, each with their own independent memory space, to communicate. There is no one simple answer to this problem. While there are many different kinds of local IPC (inter-process communication), they all suffer from two important drawbacks:

  • They impose some form of additional kernel overhead, lowering performance.
  • In almost all situations, IPC is not a "natural" extension of your code. It often dramatically increases the complexity of your program.


Double bummer: overhead and complication aren't good things. If you've ever had to make massive modifications to one of your programs so that it supports IPC, you'll really appreciate the simple memory-sharing approach that threads provide. POSIX threads don't need to make expensive and complicated long-distance calls because all our threads happen to live in the same house. With a little synchronization, all your threads can read and modify your program's existing data structures. You don't have to pump the data through a file descriptor or squeeze it into a tight, shared memory space. For this reason alone you should consider the one process/multithread model rather than the multiprocess/single-thread model.

Threads are nimble

But there's more. Threads also happen to be extremely nimble. Compared to a standard fork(), they carry a lot less overhead. The kernel does not need to make a new independent copy of the process memory space, file descriptors, etc. That saves a lot of CPU time, making thread creation ten to a hundred times faster than new process creation. Because of this, you can use a whole bunch of threads and not worry too much about the CPU and memory overhead incurred. You don't have a big CPU hit the way you do with fork(). This means you can generally create threads whenever it makes sense in your program.

Of course, just like processes, threads will take advantage of multiple CPUs. This is a really great feature if your software is designed to be used on a multiprocessor machine (if the software is open source, it will probably end up running on quite a few of these). The performance of certain kinds of threaded programs (CPU-intensive ones in particular) will scale almost linearly with the number of processors in the system. If you're writing a program that is very CPU-intensive, you'll definitely want to find ways to use multiple threads in your code. Once you're adept at writing threaded code, you'll also be able to approach coding challenges in new and creative ways without a lot of IPC red tape and miscellaneous mumbo-jumbo. All these benefits work synergistically to make multithreaded programming fun, fast, and flexible.

I think I'm a clone now

If you've been in the Linux programming world for a while, you may know about the __clone() system call. __clone() is similar to fork(), but allows you to do lots of things that threads can do. For example, with __clone() you can selectively share parts of your parent's execution context (memory space, file descriptors, etc.) with a new child process. That's a good thing. But there is also a not-so-good thing about __clone(). As the __clone() man page states:

"The __clone call is Linux-specific and should not be used in programs intended to be portable. For programming threaded applications (multiple threads of control in the same memory space), it is better to use a library implementing the POSIX 1003.1c thread API, such as the Linux-Threads library. See pthread_create(3thr)."

So, while __clone() offers many of the benefits of threads, it is not portable. That doesn't mean you shouldn't use it in your code. But you should weigh this fact when you are considering using __clone() in your software. Fortunately, as the __clone() man page states, there's a better alternative: POSIX threads. When you want to write portable multithreaded code, code that works under Solaris, FreeBSD, Linux, and more, POSIX threads are the way to go.

Beginning threads

Here's a simple example program that uses POSIX threads:

thread1.c (c source code) - Sample program using POSIX threads
#include <pthread.h>
#include <stdlib.h>
#include <unistd.h>
 
void *thread_function(void *arg) {
  int i;
  for ( i=0; i<20; i++ ) {
    printf("Thread says hi!\n");
    sleep(1);
  }
  return NULL;
}
 
int main(void) {
 
  pthread_t mythread;
 
  if ( pthread_create( &mythread, NULL, thread_function, NULL) ) {
    printf("error creating thread.");
    abort();
  }
 
  if ( pthread_join ( mythread, NULL ) ) {
    printf("error joining thread.");
    abort();
  }
 
  exit(0);
 
}

To compile this program, simply save this program as thread1.c, and type:

$ gcc thread1.c -o thread1 -lpthread

Run it by typing:

$ ./thread1

Understanding thread1.c

thread1.c is an extremely simple threaded program. While it doesn't do anything useful, it'll help you understand how threads work. Let's take a step-by-step look at what this program does. In main() we first declare a variable called mythread, which has a type of pthread_t. The pthread_t type, defined in pthread.h, is often called a "thread id" (commonly abbreviated as "tid"). Think of it as a kind of thread handle.

After mythread is declared (remember that mythread is just a "tid", or a handle to the thread we are about to create), we call the pthread_create function to create a real, living thread. Don't be confused by the fact that pthread_create() is inside an "if" statement. Since pthread_create() returns zero on success and a non-zero value on failure, placing the function call in an if() is just an elegant way of detecting a failed pthread_create() call. Let's look at the arguments to pthread_create. The first one is a pointer to mythread, &mythread. The second argument, currently set to NULL, can be used to define certain attributes for our thread. Because the default thread attributes work for us, we simply set it to NULL.

Our third argument is the name of a function that the new thread will execute when it starts. In this case the name of the function is thread_function(). When this thread_function() returns, our new thread will terminate. In this example our thread function doesn't do anything remarkable. It just prints out "Thread says hi!" 20 times and then exits. Notice that thread_function() accepts a void * as an argument and also returns a void * as a return value. This shows us that it's possible to use a void * to pass an arbitrary piece of data to our new thread, and that our new thread can return an arbitrary piece of data when it finishes. Now, how do we pass our thread an arbitrary argument? Easy. We use the fourth argument to the pthread_create() call. In this example, we set it to NULL because we don't need to pass any data into our trivial thread_function().

As you may have guessed, the program will consist of two threads after pthread_create() successfully returns. Wait a minute, two threads? Didn't we just create a single thread? Yes, we did. But our main program is also considered a thread. Think of it this way: if you write a program and don't use POSIX threads at all, the program will be single-threaded (this single thread is called the "main" thread). By creating a new thread we now have a total of two threads in our program.

I imagine you have at least two important questions at this point. The first is what the main thread does after the new thread is created. It keeps on going and executes the next line of our program in sequence (in this case that line is "if ( pthread_join(...))"). The second question you may be wondering about is what happens to our new thread when it exits. It stops and waits to be merged or "joined" with another thread as part of its cleanup process.

OK, now on to pthread_join(). Just as pthread_create() split our single thread into two threads, pthread_join() merges two threads into a single thread. The first argument to pthread_join() is our tid mythread. The second argument is a pointer to a void pointer. If the void pointer is non-NULL, pthread_join will place our thread's void * return value at the location we specify. Since we don't care about thread_function()'s return value, we set it to NULL.

You'll notice that our thread_function() takes 20 seconds to complete. Long before thread_function() completes, our main thread has already called pthread_join(). When this occurs our main thread will block (go to sleep) and wait for thread_function() to complete. When thread_function() completes, pthread_join() will return. Now our program has one main thread again. When our program exits, all new threads have been pthread_join()ed. This is exactly how you should deal with every new thread you create in your programs. If a new thread isn't joined it will still count against your system's maximum thread limit. This means that not doing the proper cleanup will eventually cause new pthread_create() calls to fail.

No parents, no children

If you've used the fork() system call you're probably familiar with the concept of parent and child processes. When a process creates another new process, using fork(), the new process is considered the child and the original process is considered the parent. This creates a hierarchical relationship that can be handy, especially when waiting for child processes to terminate. The waitpid() function, for example, will tell the current process to wait for the termination of any child processes. waitpid() is used to implement a simple cleanup routine in your parent process.

Things are a little more interesting with POSIX threads. You may have noticed that I have intentionally avoided using the terms "parent thread" and "child thread" so far. That's because with POSIX threads this hierarchical relationship doesn't exist. While a main thread may create a new thread, and that new thread may create an additional new thread, the POSIX threads standard views all your threads as a single pool of equals. So the concept of waiting for a child thread to exit doesn't make sense. The POSIX threads standard does not record any "family" information. This lack of genealogy has one major implication: if you want to wait for a thread to terminate, you need to specify which one you are waiting for by passing the proper tid to pthread_join(). The threads library can't figure it out for you.

To many people this isn't very good news because it can complicate programs that consist of more than two threads. Don't let it worry you. The POSIX threads standard provides all the tools you need to manage multiple threads gracefully. Actually, the fact that there is no parent/child relationship opens up some creative ways to use threads in our programs. For example, if we have a thread called thread 1, and thread 1 creates a thread called thread 2, it's not necessary for thread 1 itself to call pthread_join() for thread 2. Any other thread in the program can do that. This allows for interesting possibilities when you write heavily multithreaded code. You can, for example, create a global "dead list" that contains all stopped threads and then have a special cleanup thread that simply waits for an item to be added to the list. The cleanup thread calls pthread_join() to merge it with itself. Now your entire cleanup will be handled neatly and efficiently in a single thread.

Synchronized swimming

Now it's time to take a look at some code that does something a little unexpected. Here's thread2.c:

thread2.c (c source code)
#include <pthread.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>
 
int myglobal;
 
 void *thread_function(void *arg) {
  int i,j;
  for ( i=0; i<20; i++ ) {
    j=myglobal;
    j=j+1;
    printf(".");
    fflush(stdout);
   sleep(1);
    myglobal=j;
  }
  return NULL;
}
 
int main(void) {
 
  pthread_t mythread;
  int i;
 
  if ( pthread_create( &mythread, NULL, thread_function, NULL) ) {
    printf("error creating thread.");
    abort();
  }
 
  for ( i=0; i<20; i++) {
    myglobal=myglobal+1;
    printf("o");
    fflush(stdout);
    sleep(1);
  }
 
  if ( pthread_join ( mythread, NULL ) ) {
    printf("error joining thread.");
    abort();
  }
 
  printf("\nmyglobal equals %d\n",myglobal);
 
  exit(0);
 
}

Understanding thread2.c

This program, like our first one, creates a new thread. Both the main thread and the new thread then increment a global variable called myglobal 20 times. But the program itself produces some unexpected results. Compile it by typing:

$ gcc thread2.c -o thread2 -lpthread

And run it:

$ ./thread2
..o.o.o.o.oo.o.o.o.o.o.o.o.o.o..o.o.o.o.o
myglobal equals 21

Quite unexpected! Since myglobal starts at zero, and both the main thread and the new thread each increment it by 20, we should see myglobal equaling 40 at the end of the program. Since myglobal equals 21, we know that something fishy is going on here. But what is it?

Give up? OK, I'll show you why it happens. Take a look at thread_function(). Notice how we copy myglobal into a local variable called "j"? And see how we increment j, then sleep for one second, and only then copy our new j value into myglobal? That's the key. Imagine what happens if our main thread increments myglobal just after our new thread copies the value of myglobal into j. When thread_function() writes the value of j back to myglobal, it will overwrite the modification that our main thread made.

When writing threaded programs, you want to avoid useless side effects like the one we just looked at because they're a waste of time (except when you are writing articles on POSIX threads, of course :). Now, what can we do to eliminate this hassle?

Since the problem occurs because we copy myglobal to j and hold it there for one second before writing it back, we could try to avoid using a temporary local variable and incrementing myglobal directly. While this solution will probably work in this particular example, it is not correct. And if we were performing a relatively complex mathematical operation on myglobal instead of just incrementing it, it would certainly fail. But why?

To understand the problem, you need to remember that threads run simultaneously. Even on a uniprocessor machine (where the kernel uses time slicing to simulate true multitasking) we can, from a programmer's perspective, imagine both threads as executing at the same time. thread2.c has problems because the code in thread_function() relies on the fact that myglobal will not be modified during the ~1 second before it is incremented. We need some way for one thread to tell the other to "hold off" while it's making changes to myglobal. I'll show you exactly how to do this in my next article. See you then.

Next >>>

Read the next article in this series: POSIX Threads Explained, Part 2

Support Funtoo and help us grow! Donate $15 per month and get a free SSD-based Funtoo Virtual Container.
Looking for people interested in testing and documenting Docker support! Contact Daniel Robbins for more info.

Got Funtoo?

Have you installed Funtoo Linux yet? Discover the power of a from-source meta-distribution optimized for your hardware! See our installation instructions and browse our CPU-optimized builds.

Funtoo News

Drobbins

Pre-built kernels!

Funtoo stage3's are now starting to offer pre-built kernels for ease of install. read more....
12 May 2015 by Drobbins
Drobbins

Better Experiences: Ego and Vim

Info on Funtoo's new personality tool called 'ego', and user-focused updates to vim's defaults.
27 April 2015 by Drobbins
Drobbins

How We're Keeping You At the Center of the Funtoo Universe

Read about recent developments that keep you, our users, at the forefront of our focus as Funtoo moves forward.
10 April 2015 by Drobbins
View More News...

More Articles

Browse all our Linux-related articles, below:

A

B

F

G

K

L

M

O

P

S

T

W

X

Z