Difference between pages "POSIX Threads Explained, Part 2" and "POSIX Threads Explained, Part 3"

From Funtoo
(Difference between pages)
Jump to navigation Jump to search
 
(Created page with "{{Article |Subtitle=Improve efficiency with condition variables |Summary=In this article, the last of a three-part series on POSIX threads, Daniel takes a good look at how to...")
 
Line 1: Line 1:
{{Article
{{Article
|Summary=POSIX threads are a great way to increase the responsiveness and performance of your code. In this second article of a three-part series, Daniel Robbins shows you how to protect the integrity of shared data structures in your threaded code by using nifty little things called mutexes.
|Subtitle=Improve efficiency with condition variables
|Summary=In this article, the last of a three-part series on POSIX threads, Daniel takes a good look at how to use condition variables. Condition variables are POSIX thread structures that allow you to "wake up" threads when certain conditions are met. You can think of them as a thread-safe form of signalling. Daniel wraps up the article by using all that you've learned so far to implement a multi-threaded work crew application.
|Author=Drobbins
|Author=Drobbins
|Previous in Series=POSIX Threads Explained, Part 1
|Previous in Series=POSIX Threads Explained, Part 2
|Next in Series=POSIX Threads Explained, Part 3
}}
}}
== Mutex me! ==
== Condition variables explained ==


In my previous article, I talked about threaded code that did unusual and unexpected things. Two threads each incremented a global variable twenty times. The variable was supposed to end up with a value of 40, but ended up with a value of 21 instead. What happened? The problem occurred because one thread repeatedly "cancelled out" the increment performed by the other thread. Let's take a look at some corrected code that uses a mutex to solve the problem:
I ended my previous article by describing a particular dilemma how does a thread deal with a situation where it is waiting for a specific condition to become true? It could repeatedly lock and unlock a mutex, each time checking a shared data structure for a certain value. But this is a waste of time and resources, and this form of busy polling is extremely inefficient. The best way to do this is to use the {{c|pthread_cond_wait()}} call to wait on a particular condition to become true.


{{file|name=thread3.c|lang=c|body=
It's important to understand what pthread_cond_wait() does -- it's the heart of the POSIX threads signalling system, and also the hardest part to understand.
#include <pthread.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>


int myglobal;
First, let's consider a scenario where a thread has locked a mutex, in order to take a look at a linked list, and the list happens to be empty. This particular thread can't do anything -- it's designed to remove a node from the list, and there are no nodes available. So, this is what it does.
pthread_mutex_t mymutex=PTHREAD_MUTEX_INITIALIZER;


void *thread_function(void *arg) {
While still holding the mutex lock, our thread will call {{c|pthread_cond_wait(&mycond,&mymutex)}}. The pthread_cond_wait() call is rather complex, so we'll step through each of its operations one at a time.
  int i,j;
  for ( i=0; i<20; i++ ) {
    pthread_mutex_lock(&mymutex);
    j=myglobal;
    j=j+1;
    printf(".");
    fflush(stdout);
  sleep(1);
    myglobal=j;
    pthread_mutex_unlock(&mymutex);
  }
  return NULL;
}


int main(void) {
The first thing pthread_cond_wait() does is simultaneously unlock the mutex mymutex (so that other threads can modify the linked list) and wait on the condition mycond (so that pthread_cond_wait() will wake up when it is "signalled" by another thread). Now that the mutex is unlocked, other threads can access and modify the linked list, possibly adding items.


  pthread_t mythread;
At this point, the pthread_cond_wait() call has not yet returned. Unlocking the mutex happens immediately, but waiting on the condition mycond is normally a blocking operation, meaning that our thread will go to sleep, consuming no CPU cycles until it is woken up. This is exactly what we want to happen. Our thread is sleeping, waiting for a particular condition to become true, without performing any kind of busy polling that would waste CPU time. From our thread's perspective, it's simply waiting for the pthread_cond_wait() call to return.
  int i;


  if ( pthread_create( &mythread, NULL, thread_function, NULL) ) {
Now, to continue the explanation, let's say that another thread (call it "thread 2") locks mymutex and adds an item to our linked list. Immediately after unlocking the mutex, thread 2 calls the function pthread_cond_broadcast(&mycond). By doing so, thread 2 will cause all threads waiting on the mycond condition variable to immediately wake up. This means that our first thread (which is in the middle of a pthread_cond_wait() call) will now wake up.
    printf("error creating thread.");
    bort();
  }


  for ( i=0; i<20; i++) {
Now, let's take a look at what happens to our first thread. After thread 2 called pthread_cond_broadcast(&mymutex) you might think that thread 1's pthread_cond_wait() will immediately return. Not so! Instead, pthread_cond_wait() will perform one last operation: relock mymutex. Once pthread_cond_wait() has the lock, it will then return and allow thread 1 to continue execution. At that point, it can immediately check the list for any interesting changes.
    pthread_mutex_lock(&mymutex);
    myglobal=myglobal+1;
    pthread_mutex_unlock(&mymutex);
    printf("o");
    fflush(stdout);
    sleep(1);
  }


  if ( pthread_join ( mythread, NULL ) ) {
== Stop and review! ==
    printf("error joining thread.");
    abort();
  }


   printf("\nmyglobal equals %d\n",myglobal);
{{file|name=queue.h|lang=c|body=
/* queue.h
** Copyright 2000 Daniel Robbins, Gentoo Technologies, Inc.
** Author: Daniel Robbins
** Date: 16 Jun 2000
*/
typedef struct node {
  struct node *next;
} node;
typedef struct queue {
   node *head, *tail;
} queue;
void queue_init(queue *myroot);
void queue_put(queue *myroot, node *mynode);
node *queue_get(queue *myroot);


  exit(0);
Code Listing 1.2: queue.c


/* queue.c
** Copyright 2000 Daniel Robbins, Gentoo Technologies, Inc.
** Author: Daniel Robbins
** Date: 16 Jun 2000
**
** This set of queue functions was originally thread-aware.  I
** redesigned the code to make this set of queue routines
** thread-ignorant (just a generic, boring yet very fast set of queue
** routines).  Why the change?  Because it makes more sense to have
** the thread support as an optional add-on.  Consider a situation
** where you want to add 5 nodes to the queue.  With the
** thread-enabled version, each call to queue_put() would
** automatically lock and unlock the queue mutex 5 times -- that's a
** lot of unnecessary overhead.  However, by moving the thread stuff
** out of the queue routines, the caller can lock the mutex once at
** the beginning, then insert 5 items, and then unlock at the end.
** Moving the lock/unlock code out of the queue functions allows for
** optimizations that aren't possible otherwise.  It also makes this
** code useful for non-threaded applications.
**
** We can easily thread-enable this data structure by using the
** data_control type defined in control.c and control.h. */
#include <stdio.h>
#include "queue.h"
void queue_init(queue *myroot) {
  myroot->head=NULL;
  myroot->tail=NULL;
}
void queue_put(queue *myroot,node *mynode) {
  mynode->next=NULL;
  if (myroot->tail!=NULL)
    myroot->tail->next=mynode;
  myroot->tail=mynode;
  if (myroot->head==NULL)
    myroot->head=mynode;
}
node *queue_get(queue *myroot) {
  //get from root
  node *mynode;
  mynode=myroot->head;
  if (myroot->head!=NULL)
    myroot->head=myroot->head->next;
  return mynode;
}
}
}}
}}
== Comprehension time ==


If you compare this code to the version in my previous article, you'll notice the addition of the calls {{c|pthread_mutex_lock()}} and {{c|pthread_mutex_unlock()}}. These calls perform a much-needed function in threaded programs. They provide a means of mutual exclusion (hence the name). No two threads can have the same mutex locked at the same time.
{{file|name=control.h|lang=c|body=
#include <pthread.h>
typedef struct data_control {
  pthread_mutex_t mutex;
  pthread_cond_t cond;
  int active;
} data_control;
}}


This is how mutexes work. If thread "a" tries to lock a mutex while thread "b" has the same mutex locked, thread "a" goes to sleep. As soon as thread "b" releases the mutex (via a pthread_mutex_unlock() call), thread "a" will be able to lock the mutex (in other words, it will return from the pthread_mutex_lock() call with the mutex locked). Likewise, if thread "c" tries to lock the mutex while thread "a" is holding it, thread "c" will also be put to sleep temporarily. All threads that go to sleep from calling pthread_mutex_lock() on an already-locked mutex will "queue up" for access to that mutex.
{{file|name=control.c|lang=c|body=
 
/* control.c
pthread_mutex_lock() and pthread_mutex_unlock() are normally used to protect data structures. That is, you make sure that only one thread at a time can access a certain data structure by locking and unlocking it. As you may have guessed, the POSIX threads library will grant a lock without having put the thread to sleep at all if a thread tries to lock an unlocked mutex.
** Copyright 2000 Daniel Robbins, Gentoo Technologies, Inc.
 
** Author: Daniel Robbins
<div style="margin: 10px;">[[File:L-posix-mutex.gif|class=img-responsive|frame|For your enjoyment, four znurts re-enact a scene from recent pthread_mutex_lock() calls]]</div>
** Date: 16 Jun 2000
 
**
The thread in this image that has the mutex locked gets to access the complex data structure without worrying about having other threads mess with it at the same time. The data structure is in effect "frozen" until the mutex is unlocked. It's as if the pthread_mutex_lock() and pthread_mutex_unlock() calls are "under construction" signs that surround a particular piece of shared data that's being modified or read. The calls act as a warning to other threads to go to sleep and wait their turn for the mutex lock. Of course this is only true if your surround every read and write to a particular data structure with calls to pthread_mutex_lock() and pthread_mutex_unlock().
** These routines provide an easy way to make any type of
 
** data-structure thread-aware. Simply associate a data_control
== Why mutex at all? ==
** structure with the data structure (by creating a new struct, for
 
** example). Then, simply lock and unlock the mutex, or
Sounds interesting, but why exactly do we want to put our threads to sleep? After all, isn't the main advantage of threads their ability to work independently and in many cases simultaneously? Yes, that's completely true. However, every non-trivial threads program will require at least some use of mutexes. Let's refer to our example program again to understand why.
** wait/signal/broadcast on the condition variable in the data_control
 
** structure as needed.
If you take a look at thread_function(), you'll notice that the mutex is locked at the beginning of the loop and released at the very end. In this example program, mymutex is used to protect the value of myglobal. If you look carefully at thread_function() you'll notice that the increment code copies myglobal to a local variable, increments the local variable, sleeps for one second, and only then copies the local value back to myglobal. Without the mutex, thread_function() will overwrite the incremented value when it wakes up if our main thread increments myglobal during thread_function()'s one-second nap. Using a mutex ensures that this doesn't happen. (In case you're wondering, I added the one-second delay to trigger a flawed result. There is no real reason for thread_function() to go to sleep for one second before writing the local value back to myglobal.) Our new program using mutex produces the desired result:
**
 
** data_control structs contain an int called "active". This int is
<console>
** intended to be used for a specific kind of multithreaded design,
$ ##i##./thread3
** where each thread checks the state of "active" every time it locks
o..o..o.o..o..o.o.o.o.o..o..o..o.ooooooo
** the mutex.  If active is 0, the thread knows that instead of doing
myglobal equals 40
** its normal routine, it should stop itself. If active is 1, it
</console>
** should continue as normal. So, by setting active to 0, a
 
** controlling thread can easily inform a thread work crew to shut
To further explore this extremely important concept, let's take a look at the increment code from our program:
** down instead of processing new jobs.  Use the control_activate()
 
** and control_deactivate() functions, which will also broadcast on
{{file|lang=c|desc=Increment code|body=
** the data_control struct's condition variable, so that all threads
/* thread_function() increment code: */
** stuck in pthread_cond_wait() will wake up, have an opportunity to
 
** notice the change, and then terminate.
  j=myglobal;
*/
    j=j+1;
#include "control.h"
    printf(".");
int control_init(data_control *mycontrol) {
    fflush(stdout);
  int mystatus;
    sleep(1);
  if (pthread_mutex_init(&(mycontrol->mutex),NULL))
     myglobal=j;
    return 1;
 
  if (pthread_cond_init(&(mycontrol->cond),NULL))
/* main thread increment code: */
    return 1;
    myglobal=myglobal+1;
  mycontrol->active=0;
  return 0;
}
int control_destroy(data_control *mycontrol) {
  int mystatus;
  if (pthread_cond_destroy(&(mycontrol->cond)))
    return 1;
  if (pthread_mutex_destroy(&(mycontrol->cond)))
    return 1;
  mycontrol->active=0;
  return 0;
}
int control_activate(data_control *mycontrol) {
  int mystatus;
  if (pthread_mutex_lock(&(mycontrol->mutex)))
    return 0;
  mycontrol->active=1;
  pthread_mutex_unlock(&(mycontrol->mutex));
  pthread_cond_broadcast(&(mycontrol->cond));
  return 1;
}
int control_deactivate(data_control *mycontrol) {
  int mystatus;
  if (pthread_mutex_lock(&(mycontrol->mutex)))
     return 0;
  mycontrol->active=0;
  pthread_mutex_unlock(&(mycontrol->mutex));
  pthread_cond_broadcast(&(mycontrol->cond));
  return 1;
}
}}
}}


If this code were in a single-threaded program we'd expect the thread_function() code to execute in its entirety. The execution would then be followed by the main thread code (or the other way around). In a threaded program without mutexes, the code can (and often will, thanks to the sleep() call) end up executing in the following order:
== Debug time ==


Code Listing 1.4: Executing order
One more miscellaneous file before we get to the biggie. Here's dbug.h:


{{TableStart}}
{{file|name=dbug.h|lang=C|body=
<tr><td class="active">{{c|thread_function()}} thread</td><td class="active">main thread</td></tr>
#define dabort() \
<tr><td>
  { printf("Aborting at line %d in source file %s\n",__LINE__,__FILE__); abort(); }
  j=myglobal;
  j=j+1;
  printf(".");
  fflush(stdout);
  sleep(1);                       
  myglobal=j;
</td><td><pre>
 
   
   
  myglobal=myglobal+1;
</pre></td></tr>
{{TableEnd}}
 
When the code executes in this particular order, the main thread's modification to myglobal gets overwritten. We then end up with an incorrect value at the end of our program. If we were manipulating pointers we'd probably end up with a segfault. Notice that our thread_function() thread executes all its instructions in order. It's not as if thread_function() does anything out of order. The problem is that we have another thread performing the other modifications to the same data structure effectively at the same time.
 
== Inside threads 1 ==
 
Before I explain how to figure out where to use mutexes, I'll offer a little insight into the internal working of threads. Here's our first example:
 
Let's say you have a main thread that creates three new threads: threads "a", "b", and "c". Let's say that thread "a" is created first, thread "b", second and thread "c" last.
 
{{file|lang=c|desc=Thread creating order|body=
  pthread_create( &thread_a, NULL, thread_function, NULL);
  pthread_create( &thread_b, NULL, thread_function, NULL);
  pthread_create( &thread_c, NULL, thread_function, NULL);
}}
}}


After the first pthread_create() call completes, you can assume either that thread "a" exists or that it has finished and is now stopped. After the second pthread_create() call, both the main thread and thread "b" can assume that thread "a" exists (or has stopped).
We use this code to handle unrecoverable errors in our work crew code.


However, immediately after the second create() call returns, the main thread can't assume which thread (a or b) will actually start running first. Although both threads exist it's up to the kernel and threads library to give them a slice of CPU time. And there is no hard rule as to who goes first. Now, it's very likely that thread "a" will start executing before thread "b", but it isn't guaranteed. This is especially true on a multi-processor machine. If you write code that assumes that thread "a" will actually start executing its code before thread "b" starts, you'll end up with a program that works 99% of the time. Or worse, a program that works 100% of the time on your machine but 0% of the time on your client's quad-processor server.
== The work crew code ==


Another thing that we can learn from this example is that the threads library preserves the order of code execution for each individual thread. In other words, those three pthread_create() calls will in fact execute in the order that they appear. From the main thread's perspective, all the code is executing in order. Sometimes we can take advantage of this to optimize parts of our threaded programs. For instance, in the above example, thread "c" can assume that thread "a" and "b" are running or have already terminated. It does not have to worry about the possibility that threads "a" and "b" haven't been created yet. You can use this logic to optimize your threaded programs.
Speaking of the work crew code, here it is:


== Inside threads 2 ==
Code Listing 1.6: workcrew.c>
 
{{file|name=workcrew.c|lang=c|body=
OK, now for another hypothetical example. Let's say we have a bunch of threads that are executing the following code:
#include <stdio.h>
 
#include <stdlib.h>
{{file|lang=c|body=
#include "control.h"
   myglobal=myglobal+1;
#include "queue.h"
#include "dbug.h"
/* the work_queue holds tasks for the various threads to complete.*/
struct work_queue {
  data_control control;
  queue work;
} wq;
/* I added a job number to the work node.  Normally, the work node
  would contain additional data that needed to be processed. */
typedef struct work_node {
  struct node *next;
  int jobnum;
} wnode;
/* the cleanup queue holds stopped threads.  Before a thread
  terminates, it adds itself to this list.  Since the main thread is
  waiting for changes in this list, it will then wake up and clean up
  the newly terminated thread. */
struct cleanup_queue {
  data_control control;
  queue cleanup;
} cq;
/* I added a thread number (for debugging/instructional purposes) and
  a thread id to the cleanup node.  The cleanup node gets passed to
  the new thread on startup, and just before the thread stops, it
  attaches the cleanup node to the cleanup queue.  The main thread
  monitors the cleanup queue and is the one that performs the
  necessary cleanup. */
typedef struct cleanup_node {
  struct node *next;
  int threadnum;
  pthread_t tid;
} cnode;
void *threadfunc(void *myarg) {
  wnode *mywork;
  cnode *mynode;
  mynode=(cnode *) myarg;
  pthread_mutex_lock(&wq.control.mutex);
  while (wq.control.active) {
    while (wq.work.head==NULL && wq.control.active) {
      pthread_cond_wait(&wq.control.cond, &wq.control.mutex);
    }
    if (!wq.control.active)
      break;
    //we got something!
    mywork=(wnode *) queue_get(&wq.work);
    pthread_mutex_unlock(&wq.control.mutex);
    //perform processing...
    printf("Thread number %d processing job %d\n",mynode->threadnum,mywork->jobnum);
    free(mywork);
    pthread_mutex_lock(&wq.control.mutex);
  }
  pthread_mutex_unlock(&wq.control.mutex);
  pthread_mutex_lock(&cq.control.mutex);
  queue_put(&cq.cleanup,(node *) mynode);
  pthread_mutex_unlock(&cq.control.mutex);
  pthread_cond_signal(&cq.control.cond);
  printf("thread %d shutting down...\n",mynode->threadnum);
  return NULL;
 
}
#define NUM_WORKERS 4
int numthreads;
void join_threads(void) {
  cnode *curnode;
  printf("joining threads...\n");
  while (numthreads) {
    pthread_mutex_lock(&cq.control.mutex);
    /* below, we sleep until there really is a new cleanup node. This
      takes care of any false wakeups... even if we break out of
      pthread_cond_wait(), we don't make any assumptions that the
      condition we were waiting for is true.  */
    while (cq.cleanup.head==NULL) {
      pthread_cond_wait(&cq.control.cond,&cq.control.mutex);
    }
    /* at this point, we hold the mutex and there is an item in the
      list that we need to process. First, we remove the node from
      the queue.  Then, we call pthread_join() on the tid stored in
      the node.  When pthread_join() returns, we have cleaned up
      after a thread.  Only then do we free() the node, decrement the
      number of additional threads we need to wait for and repeat the
      entire process, if necessary */
      curnode = (cnode *) queue_get(&cq.cleanup);
      pthread_mutex_unlock(&cq.control.mutex);
      pthread_join(curnode->tid,NULL);
      printf("joined with thread %d\n",curnode->threadnum);
      free(curnode);
      numthreads--;
  }
}
int create_threads(void) {
  int x;
  cnode *curnode;
  for (x=0; x<NUM_WORKERS; x++) {
    curnode=malloc(sizeof(cnode));
    if (!curnode)
      return 1;
    curnode->threadnum=x;
    if (pthread_create(&curnode->tid, NULL, threadfunc, (void *) curnode))
      return 1;
    printf("created thread %d\n",x);
    numthreads++;
  }
  return 0;
}
void initialize_structs(void) {
  numthreads=0;
  if (control_init(&wq.control))
    dabort();
  queue_init(&wq.work);
  if (control_init(&cq.control)) {
    control_destroy(&wq.control);
    dabort();
  }
  queue_init(&wq.work);
   control_activate(&wq.control);
}
void cleanup_structs(void) {
  control_destroy(&cq.control);
  control_destroy(&wq.control);
}
int main(void) {
  int x;
  wnode *mywork;
  initialize_structs();
  /* CREATION */
 
  if (create_threads()) {
    printf("Error starting threads... cleaning up.\n");
    join_threads();
    dabort();
  }
  pthread_mutex_lock(&wq.control.mutex);
  for (x=0; x<16000; x++) {
    mywork=malloc(sizeof(wnode));
    if (!mywork) {
      printf("ouch! can't malloc!\n");
      break;
    }
    mywork->jobnum=x;
    queue_put(&wq.work,(node *) mywork);
  }
  pthread_mutex_unlock(&wq.control.mutex);
  pthread_cond_broadcast(&wq.control.cond);
  printf("sleeping...\n");
  sleep(2);
  printf("deactivating work queue...\n");
  control_deactivate(&wq.control);
  /* CLEANUP  */
  join_threads();
  cleanup_structs();
}
}}
}}
Do we need to lock and unlock the mutex before and after the increment respectively? Some of you may say "no". The compiler will after all very likely compile the above assignment into a single machine instruction. As you know a single machine instruction cannot be interrupted "mid-stream". Even hardware interrupts will respect the atomicity of machine instructions. Because of this tendency it's tempting to leave out the pthread_mutex_lock() and pthread_mutex_unlock() calls entirely. Don't do it.
Am I being a wimp? Not really. First, you shouldn't assume that the above assignment would be compiled as a single machine instruction unless you personally verify the machine code yourself. Even if you insert some inline assembly to ensure that the increment happens atomically -- or even if you wrote the compiler yourself! -- you may still have problems.


Here's why. Using a single inline assembly opcode will probably work wonderfully on a uni-processor machine. Each increment will happen atomically, and chances are you'll get the desired result. But a multi-processor machine is another story. On a multi-CPU machine, you can have two separate processors executing the above assignment at nearly (or at times exactly) the same time. And remember that this memory modification will need to trickle down from L1 to L2 cache, and then to main memory. (An SMP machine is not just an additional processor; it also has special hardware that arbitrates access to RAM.) In the end, you'll really have no idea which CPU "wins" the race of writing to main memory. To produce predictable code, you'll want to use mutexes. Mutexes will insert a "memory barrier," which ensures that the writes to main memory occur in the order the threads lock the mutex.
== Code walkthrough ==


Consider an SMP architecture that updates main memory in 32-bit blocks. If you are incrementing a 64-bit integer without mutexes, the uppermost 4 bytes might come from one CPU and the other four might come from another. Bummer! Worst of all, using poor technique will probably make your program bomb out once in a blue moon or at 3 AM on an important client's system. David R. Butenhof covers the possible permutations of not using mutexes in his book, Programming with POSIX Threads (see Resources at the end of this article).
Now it's time for a quick code walkthrough. The first struct defined is called {{c|wq}}, and contains a data_control and a queue header. The data_control structure will be used to arbitrate access to the entire queue, including the nodes in the queue. Our next job is to define the actual work nodes. To keep the code lean to fit in this article, all that's contained here is a job number.


== Many mutexes ==
Next, we create the cleanup queue. The comments explain how this works. OK, now let's skip the threadfunc(), join_threads(), create_threads() and initialize_structs() calls, and jump down to main(). The first thing we do is initialize our structures -- this includes initializing our data_controls and queues, as well as activating our work queue.


If you place too many mutexes, your code won't have any kind of concurrency and will run slower than a single-threaded solution. If you place too few, your code will have weird and embarrassing bugs. Fortunately, there is a middle ground. First of all, mutexes are used to serialize access to *shared data*. Don't use them for non-shared data, and don't use them if your program's logic ensures that only one thread is accessing a particular data structure at a single time.
== Cleanup special ==


Second of all, if you are using shared data, use mutexes for both reading and writing. Surround your read and write sections with pthread_mutex_lock() and pthread_mutex_unlock(), or use them any time a program's invariant is temporarily broken. Learn to view your code from the perspective of a single thread and make sure each individual thread in your program has a consistent, friendly view of memory. It'll probably take you several hours of writing your own code to get the hang of mutexes, but they will soon become second nature and you'll be able to use them properly without thinking too much.
Now it's time to initialize our threads. If you look at our create_threads() call, everything will look pretty normal -- except for one thing. Notice that we are allocating a cleanup node, and initializing its threadnum and TID components. We also pass a cleanup node to each new worker thread as an initial argument. Why do we do this?


== Using the calls: initialization ==
Because when a worker thread exits, it'll attach its cleanup node to the cleanup queue, and terminate. Then, our main thread will detect this addition to the cleanup queue (by use of a condition variable) and dequeue the node. Because the TID (thread id) is stored in the cleanup node, our main thread will know exactly which thread terminated. Then, our main thread will call pthread_join(tid), and join with the appropriate worker thread. If we didn't perform such bookkeeping, our main thread would need to join with worker threads in an arbitrary order, possibly in the order that they were created. Because the threads may not necessarily terminate in this order, our main thread could be waiting to join with one thread while it could have been joining with ten others. Can you see how this design decision can really speed up our shutdown code, especially if we were to use hundreds of worker threads?


OK, now it's time to see all the different ways to use mutexes. First, we'll start with initialization. In our thread3.c example, we used a static initialization method. This involved declaring a pthread_mutex_t variable and assigning it the constant PTHREAD_MUTEX_INITIALIZER:
== Creating work ==


{{file|lang=c|body=
Now that we've started our worker threads (and they're off performing their threadfunc(), which we'll get to in a bit), our main thread begins inserting items into the work queue. First, it locks wq's control mutex, and then allocates 16000 work packets, inserting them into the queue one-by-one. After this is done, pthread_cond_broadcast() is called, so that any sleeping threads are woken up and able to do the work. Then, our main thread sleeps for two seconds, and then deactivates the work queue, telling our worker threads to terminate. Then, our main thread calls the join_threads() function to clean up all the worker threads.
pthread_mutex_t mymutex=PTHREAD_MUTEX_INITIALIZER;
}}
 
That's pretty easy. But you can also create a mutex dynamically. Use this dynamic method whenever your code allocates a new mutex using malloc(). In this case, the static initialization method won't do, and the routine pthread_mutex_init() should be used:
 
{{file|lang=c|desc=Dynamic way of creating mutex|body=
int pthread_mutex_init( pthread_mutex_t *mymutex, const pthread_mutexattr_t*attr)
}}


As you can see, pthread_mutex_init accepts a pointer to an already-allocated region of memory to initialize as a mutex. As a second argument, it can also accept an optional pthread_mutexattr_t pointer. This structure can be used to set various mutex attributes. But usually these attributes are not needed, so specifying NULL is the norm.
== threadfunc() ==


Whenever you initialize a mutex using pthread_mutex_init(), it should be destroyed using pthread_mutex_destroy(). pthread_mutex_destroy() accepts a single pointer to a pthread_mutex_t and frees any resources allocated to the mutex when it was created. Be sure to note that pthread_mutex_destroy() does not free the memory used to store the pthread_mutex_t. It's up to you to free() your own memory. Also remember that both pthread_mutex_init() and pthread_mutex_destroy() return zero on success.
Time to look at threadfunc(), the code that each worker thread executes. When a worker thread starts, it immediately locks the work queue mutex, gets one work node (if available) and processes it. If no work is available, pthread_cond_wait() is called. You'll notice that it's called in a very tight while() loop, and this is very important. When you wake up from a pthread_cond_wait() call, you should never assume that your condition is definitely true -- it will probably be true, but it may not. The while loop will cause pthread_cond_wait() to be called again if it so happens that the thread was mistakenly woken up and the list is empty.


== Using the calls: locking ==
If there's a work node, we simply print out its job number, free it, and exit. Real code would do something more substantial. At the end of the while() loop, we lock the mutex so we can check the active variable as well as checking for new work nodes at the top of the loop. If you follow the code through, you'll find that if wq.control.active is 0, the while loop will be terminated and the cleanup code at the end of threadfunc() will begin.


;{{c|pthread_mutex_lock(pthread_mutex_t *mutex)}}:  pthread_mutex_lock() accepts a single pointer to a mutex to lock. If the mutex already happens to be locked, the caller will go to sleep. When the function returns, the caller will be woken up (obviously), and the caller will also now hold the lock. This call either returns zero on success or a non-zero error code on failure.
The worker thread's part of the cleanup code is pretty interesting. First, it unlocks the work_queue, since pthread_cond_wait() returns with the mutex locked. Then, it gets a lock on the cleanup queue, adds our cleanup node (containing our TID, which the main thread will use for its pthread_join() call), and then it unlocks the cleanup queue. After that, it signals any cq waiters (pthread_cond_signal(&cq.control.cond)) so that the main thread will know that there's a new node to process. We don't use pthread_cond_broadcast() because it's not necessary -- only one thread (the main thread) is waiting for new entries in the cleanup queue. Our worker thread prints a shutdown message, and then terminates, waiting to be pthread_joined() by the main thread when it calls join_threads().


; {{c|pthread_mutex_unlock(pthread_mutex_t *mutex)}}: pthread_mutex_unlock() complements pthread_mutex_lock() and unlocks a mutex that the thread has already locked. You should always unlock a mutex that you've locked as soon as safely possible (to increase performance). And you should never unlock a mutex that you don't hold a lock for (otherwise, the pthread_mutex_unlock() call will fail with a non-zero EPERM return value).
== join_threads() ==


; {{c|pthread_mutex_trylock(pthread_mutex_t *mutex)}}: This call is handy when you want to lock a mutex while your thread is doing something else (because the mutex is currently locked). When you call pthread_mutex_trylock() you'll attempt to lock the mutex. If the mutex is currently unlocked you'll get the lock, and this function will return zero. However, if the mutex is locked this call won't block. Rather, it will return a non-zero EBUSY error value. Then you can go about your business and try to lock at a later time.
If you want to see a simple example of how condition variables should be used, take a look at the join_threads() function. While we still have worker threads in existence, join_threads() loops, waiting for new cleanup nodes in our cleanup queue. If there is a new node, we dequeue the node, unlock the cleanup queue (so that other cleanup nodes can be added by our worker threads), join with our new thread (using the TID stored in the cleanup node), free the cleanup node, decrement the number of threads "out there", and continue.


== Waiting on conditions ==
== Wrapping it up ==


Mutexes are necessary tools for threaded programs, but they can't do everything. What happens, for instance, if your thread is waiting for a certain condition to appear in shared data? Your code could repeatedly lock and unlock the mutex, checking for any changes to the value. At the same time it will also quickly unlock the mutex so that others can make any necessary changes. But this is a horrible approach because this thread will need to busy-loop to detect a change in a reasonable time frame.
We've reached the end of the "POSIX threads explained" series, and I hope that you're now ready to begin adding multithreaded code to your own applications. For more information, please see the Resources section, which also contains a tarball of all the sources used in this article. I'll see you next series!


You could put the calling thread to sleep for a little bit, say three seconds between each check, but then your threaded code wouldn't be optimally responsive. What you really need is a way to put a thread to sleep while it waits for some condition to be met. Once the condition is met you need a method to wake up the thread(s) that are waiting for that particular condition to become true. If you can do this, your threaded code will be really efficient and it won't tie up valuable mutex locks. That's precisely what POSIX condition variables can do for you!


And POSIX condition variables are the subject of my next article, where I'll show you exactly how to use them. Then you'll have all the resources to create sophisticated threaded programs that model work crews, assembly lines, and more. I'm going to pick up the pace in the next article now that you're getting more familiar with threads. I'm hoping this will allow me to squeeze in a reasonably sophisticated threaded program at the end of the next article. And speaking of waiting on conditions, I'll see you then!
{{ArticleFooter}}
{{ArticleFooter}}

Revision as of 08:08, December 31, 2014

Improve efficiency with condition variables

In this article, the last of a three-part series on POSIX threads, Daniel takes a good look at how to use condition variables. Condition variables are POSIX thread structures that allow you to "wake up" threads when certain conditions are met. You can think of them as a thread-safe form of signalling. Daniel wraps up the article by using all that you've learned so far to implement a multi-threaded work crew application.
   Support Funtoo!
Get an awesome Funtoo container and support Funtoo! See Funtoo Containers for more information.

Condition variables explained

I ended my previous article by describing a particular dilemma how does a thread deal with a situation where it is waiting for a specific condition to become true? It could repeatedly lock and unlock a mutex, each time checking a shared data structure for a certain value. But this is a waste of time and resources, and this form of busy polling is extremely inefficient. The best way to do this is to use the pthread_cond_wait() call to wait on a particular condition to become true.

It's important to understand what pthread_cond_wait() does -- it's the heart of the POSIX threads signalling system, and also the hardest part to understand.

First, let's consider a scenario where a thread has locked a mutex, in order to take a look at a linked list, and the list happens to be empty. This particular thread can't do anything -- it's designed to remove a node from the list, and there are no nodes available. So, this is what it does.

While still holding the mutex lock, our thread will call pthread_cond_wait(&mycond,&mymutex). The pthread_cond_wait() call is rather complex, so we'll step through each of its operations one at a time.

The first thing pthread_cond_wait() does is simultaneously unlock the mutex mymutex (so that other threads can modify the linked list) and wait on the condition mycond (so that pthread_cond_wait() will wake up when it is "signalled" by another thread). Now that the mutex is unlocked, other threads can access and modify the linked list, possibly adding items.

At this point, the pthread_cond_wait() call has not yet returned. Unlocking the mutex happens immediately, but waiting on the condition mycond is normally a blocking operation, meaning that our thread will go to sleep, consuming no CPU cycles until it is woken up. This is exactly what we want to happen. Our thread is sleeping, waiting for a particular condition to become true, without performing any kind of busy polling that would waste CPU time. From our thread's perspective, it's simply waiting for the pthread_cond_wait() call to return.

Now, to continue the explanation, let's say that another thread (call it "thread 2") locks mymutex and adds an item to our linked list. Immediately after unlocking the mutex, thread 2 calls the function pthread_cond_broadcast(&mycond). By doing so, thread 2 will cause all threads waiting on the mycond condition variable to immediately wake up. This means that our first thread (which is in the middle of a pthread_cond_wait() call) will now wake up.

Now, let's take a look at what happens to our first thread. After thread 2 called pthread_cond_broadcast(&mymutex) you might think that thread 1's pthread_cond_wait() will immediately return. Not so! Instead, pthread_cond_wait() will perform one last operation: relock mymutex. Once pthread_cond_wait() has the lock, it will then return and allow thread 1 to continue execution. At that point, it can immediately check the list for any interesting changes.

Stop and review!

   queue.h (c source code)
/* queue.h
** Copyright 2000 Daniel Robbins, Gentoo Technologies, Inc.
** Author: Daniel Robbins
** Date: 16 Jun 2000
*/
typedef struct node {
  struct node *next;
} node;
typedef struct queue {
  node *head, *tail; 
} queue;
void queue_init(queue *myroot);
void queue_put(queue *myroot, node *mynode);
node *queue_get(queue *myroot);

Code Listing 1.2: queue.c

/* queue.c
** Copyright 2000 Daniel Robbins, Gentoo Technologies, Inc.
** Author: Daniel Robbins
** Date: 16 Jun 2000
**
** This set of queue functions was originally thread-aware.  I
** redesigned the code to make this set of queue routines
** thread-ignorant (just a generic, boring yet very fast set of queue
** routines).  Why the change?  Because it makes more sense to have
** the thread support as an optional add-on.  Consider a situation
** where you want to add 5 nodes to the queue.  With the
** thread-enabled version, each call to queue_put() would
** automatically lock and unlock the queue mutex 5 times -- that's a
** lot of unnecessary overhead.  However, by moving the thread stuff
** out of the queue routines, the caller can lock the mutex once at
** the beginning, then insert 5 items, and then unlock at the end.
** Moving the lock/unlock code out of the queue functions allows for
** optimizations that aren't possible otherwise.  It also makes this
** code useful for non-threaded applications.
**
** We can easily thread-enable this data structure by using the
** data_control type defined in control.c and control.h. */
#include <stdio.h>
#include "queue.h"
void queue_init(queue *myroot) {
  myroot->head=NULL;
  myroot->tail=NULL;
}
void queue_put(queue *myroot,node *mynode) {
  mynode->next=NULL;
  if (myroot->tail!=NULL)
    myroot->tail->next=mynode;
  myroot->tail=mynode;
  if (myroot->head==NULL)
    myroot->head=mynode;
}
node *queue_get(queue *myroot) {
  //get from root
  node *mynode;
  mynode=myroot->head;
  if (myroot->head!=NULL)
    myroot->head=myroot->head->next;
  return mynode;
}
   control.h (c source code)
#include <pthread.h>
typedef struct data_control {
  pthread_mutex_t mutex;
  pthread_cond_t cond;
  int active;
} data_control;
   control.c (c source code)
/* control.c
** Copyright 2000 Daniel Robbins, Gentoo Technologies, Inc.
** Author: Daniel Robbins
** Date: 16 Jun 2000
**
** These routines provide an easy way to make any type of
** data-structure thread-aware.  Simply associate a data_control
** structure with the data structure (by creating a new struct, for
** example).  Then, simply lock and unlock the mutex, or
** wait/signal/broadcast on the condition variable in the data_control
** structure as needed.
**
** data_control structs contain an int called "active".  This int is
** intended to be used for a specific kind of multithreaded design,
** where each thread checks the state of "active" every time it locks
** the mutex.  If active is 0, the thread knows that instead of doing
** its normal routine, it should stop itself.  If active is 1, it
** should continue as normal.  So, by setting active to 0, a
** controlling thread can easily inform a thread work crew to shut
** down instead of processing new jobs.  Use the control_activate()
** and control_deactivate() functions, which will also broadcast on
** the data_control struct's condition variable, so that all threads
** stuck in pthread_cond_wait() will wake up, have an opportunity to
** notice the change, and then terminate.
*/
#include "control.h"
int control_init(data_control *mycontrol) {
  int mystatus;
  if (pthread_mutex_init(&(mycontrol->mutex),NULL))
    return 1;
  if (pthread_cond_init(&(mycontrol->cond),NULL))
    return 1;
  mycontrol->active=0;
  return 0;
}
int control_destroy(data_control *mycontrol) {
  int mystatus;
  if (pthread_cond_destroy(&(mycontrol->cond)))
    return 1;
  if (pthread_mutex_destroy(&(mycontrol->cond)))
    return 1;
  mycontrol->active=0;
  return 0;
}
int control_activate(data_control *mycontrol) {
  int mystatus;
  if (pthread_mutex_lock(&(mycontrol->mutex)))
    return 0;
  mycontrol->active=1;
  pthread_mutex_unlock(&(mycontrol->mutex));
  pthread_cond_broadcast(&(mycontrol->cond));
  return 1;
}
int control_deactivate(data_control *mycontrol) {
  int mystatus;
  if (pthread_mutex_lock(&(mycontrol->mutex)))
    return 0;
  mycontrol->active=0;
  pthread_mutex_unlock(&(mycontrol->mutex));
  pthread_cond_broadcast(&(mycontrol->cond));
  return 1;
}

Debug time

One more miscellaneous file before we get to the biggie. Here's dbug.h:

   dbug.h (C source code)
#define dabort() \
 {  printf("Aborting at line %d in source file %s\n",__LINE__,__FILE__);  abort(); }

We use this code to handle unrecoverable errors in our work crew code.

The work crew code

Speaking of the work crew code, here it is:

Code Listing 1.6: workcrew.c>

   workcrew.c (c source code)
#include <stdio.h>
#include <stdlib.h>
#include "control.h"
#include "queue.h"
#include "dbug.h"
/* the work_queue holds tasks for the various threads to complete.*/
struct work_queue {
  data_control control;
  queue work;
} wq;
/* I added a job number to the work node.  Normally, the work node
   would contain additional data that needed to be processed. */
typedef struct work_node {
  struct node *next;
  int jobnum;
} wnode;
/* the cleanup queue holds stopped threads.  Before a thread
   terminates, it adds itself to this list.  Since the main thread is
   waiting for changes in this list, it will then wake up and clean up
   the newly terminated thread. */
struct cleanup_queue {
  data_control control;
  queue cleanup;
} cq;
/* I added a thread number (for debugging/instructional purposes) and
   a thread id to the cleanup node.  The cleanup node gets passed to
   the new thread on startup, and just before the thread stops, it
   attaches the cleanup node to the cleanup queue.  The main thread
   monitors the cleanup queue and is the one that performs the
   necessary cleanup. */
typedef struct cleanup_node {
  struct node *next;
  int threadnum;
  pthread_t tid;
} cnode;
void *threadfunc(void *myarg) {
  wnode *mywork;
  cnode *mynode;
  mynode=(cnode *) myarg;
  pthread_mutex_lock(&wq.control.mutex);
  while (wq.control.active) {
    while (wq.work.head==NULL && wq.control.active) {
      pthread_cond_wait(&wq.control.cond, &wq.control.mutex);
    }
    if (!wq.control.active) 
      break;
    //we got something!
    mywork=(wnode *) queue_get(&wq.work);
    pthread_mutex_unlock(&wq.control.mutex);
    //perform processing...
    printf("Thread number %d processing job %d\n",mynode->threadnum,mywork->jobnum);
    free(mywork);
    pthread_mutex_lock(&wq.control.mutex);
  }
  pthread_mutex_unlock(&wq.control.mutex);
  pthread_mutex_lock(&cq.control.mutex);
  queue_put(&cq.cleanup,(node *) mynode);
  pthread_mutex_unlock(&cq.control.mutex);
  pthread_cond_signal(&cq.control.cond);
  printf("thread %d shutting down...\n",mynode->threadnum);
  return NULL;
  
}
#define NUM_WORKERS 4
int numthreads;
void join_threads(void) {
  cnode *curnode;
  printf("joining threads...\n");
  while (numthreads) {
    pthread_mutex_lock(&cq.control.mutex);
    /* below, we sleep until there really is a new cleanup node. This
       takes care of any false wakeups... even if we break out of
       pthread_cond_wait(), we don't make any assumptions that the
       condition we were waiting for is true.  */
    while (cq.cleanup.head==NULL) {
      pthread_cond_wait(&cq.control.cond,&cq.control.mutex);
    }
    /* at this point, we hold the mutex and there is an item in the
       list that we need to process.  First, we remove the node from
       the queue.  Then, we call pthread_join() on the tid stored in
       the node.  When pthread_join() returns, we have cleaned up
       after a thread.  Only then do we free() the node, decrement the
       number of additional threads we need to wait for and repeat the
       entire process, if necessary */
      curnode = (cnode *) queue_get(&cq.cleanup);
      pthread_mutex_unlock(&cq.control.mutex);
      pthread_join(curnode->tid,NULL);
      printf("joined with thread %d\n",curnode->threadnum);
      free(curnode);
      numthreads--;
  }
}
int create_threads(void) {
  int x;
  cnode *curnode;
  for (x=0; x<NUM_WORKERS; x++) {
    curnode=malloc(sizeof(cnode));
    if (!curnode)
      return 1;
    curnode->threadnum=x;
    if (pthread_create(&curnode->tid, NULL, threadfunc, (void *) curnode))
      return 1;
    printf("created thread %d\n",x);
    numthreads++;
  }
  return 0;
}
void initialize_structs(void) {
  numthreads=0;
  if (control_init(&wq.control))
    dabort();
  queue_init(&wq.work);
  if (control_init(&cq.control)) {
    control_destroy(&wq.control);
    dabort();
  }
  queue_init(&wq.work);
  control_activate(&wq.control);
}
void cleanup_structs(void) {
  control_destroy(&cq.control);
  control_destroy(&wq.control);
}
int main(void) {
  int x;
  wnode *mywork;
  initialize_structs();
  /* CREATION */
  
  if (create_threads()) {
    printf("Error starting threads... cleaning up.\n");
    join_threads();
    dabort();
  }
  pthread_mutex_lock(&wq.control.mutex);
  for (x=0; x<16000; x++) {
    mywork=malloc(sizeof(wnode));
    if (!mywork) {
      printf("ouch! can't malloc!\n");
      break;
    }
    mywork->jobnum=x;
    queue_put(&wq.work,(node *) mywork);
  }
  pthread_mutex_unlock(&wq.control.mutex);
  pthread_cond_broadcast(&wq.control.cond);
  printf("sleeping...\n");
  sleep(2);
  printf("deactivating work queue...\n");
  control_deactivate(&wq.control);
  /* CLEANUP  */
  join_threads();
  cleanup_structs();
}

Code walkthrough

Now it's time for a quick code walkthrough. The first struct defined is called wq, and contains a data_control and a queue header. The data_control structure will be used to arbitrate access to the entire queue, including the nodes in the queue. Our next job is to define the actual work nodes. To keep the code lean to fit in this article, all that's contained here is a job number.

Next, we create the cleanup queue. The comments explain how this works. OK, now let's skip the threadfunc(), join_threads(), create_threads() and initialize_structs() calls, and jump down to main(). The first thing we do is initialize our structures -- this includes initializing our data_controls and queues, as well as activating our work queue.

Cleanup special

Now it's time to initialize our threads. If you look at our create_threads() call, everything will look pretty normal -- except for one thing. Notice that we are allocating a cleanup node, and initializing its threadnum and TID components. We also pass a cleanup node to each new worker thread as an initial argument. Why do we do this?

Because when a worker thread exits, it'll attach its cleanup node to the cleanup queue, and terminate. Then, our main thread will detect this addition to the cleanup queue (by use of a condition variable) and dequeue the node. Because the TID (thread id) is stored in the cleanup node, our main thread will know exactly which thread terminated. Then, our main thread will call pthread_join(tid), and join with the appropriate worker thread. If we didn't perform such bookkeeping, our main thread would need to join with worker threads in an arbitrary order, possibly in the order that they were created. Because the threads may not necessarily terminate in this order, our main thread could be waiting to join with one thread while it could have been joining with ten others. Can you see how this design decision can really speed up our shutdown code, especially if we were to use hundreds of worker threads?

Creating work

Now that we've started our worker threads (and they're off performing their threadfunc(), which we'll get to in a bit), our main thread begins inserting items into the work queue. First, it locks wq's control mutex, and then allocates 16000 work packets, inserting them into the queue one-by-one. After this is done, pthread_cond_broadcast() is called, so that any sleeping threads are woken up and able to do the work. Then, our main thread sleeps for two seconds, and then deactivates the work queue, telling our worker threads to terminate. Then, our main thread calls the join_threads() function to clean up all the worker threads.

threadfunc()

Time to look at threadfunc(), the code that each worker thread executes. When a worker thread starts, it immediately locks the work queue mutex, gets one work node (if available) and processes it. If no work is available, pthread_cond_wait() is called. You'll notice that it's called in a very tight while() loop, and this is very important. When you wake up from a pthread_cond_wait() call, you should never assume that your condition is definitely true -- it will probably be true, but it may not. The while loop will cause pthread_cond_wait() to be called again if it so happens that the thread was mistakenly woken up and the list is empty.

If there's a work node, we simply print out its job number, free it, and exit. Real code would do something more substantial. At the end of the while() loop, we lock the mutex so we can check the active variable as well as checking for new work nodes at the top of the loop. If you follow the code through, you'll find that if wq.control.active is 0, the while loop will be terminated and the cleanup code at the end of threadfunc() will begin.

The worker thread's part of the cleanup code is pretty interesting. First, it unlocks the work_queue, since pthread_cond_wait() returns with the mutex locked. Then, it gets a lock on the cleanup queue, adds our cleanup node (containing our TID, which the main thread will use for its pthread_join() call), and then it unlocks the cleanup queue. After that, it signals any cq waiters (pthread_cond_signal(&cq.control.cond)) so that the main thread will know that there's a new node to process. We don't use pthread_cond_broadcast() because it's not necessary -- only one thread (the main thread) is waiting for new entries in the cleanup queue. Our worker thread prints a shutdown message, and then terminates, waiting to be pthread_joined() by the main thread when it calls join_threads().

join_threads()

If you want to see a simple example of how condition variables should be used, take a look at the join_threads() function. While we still have worker threads in existence, join_threads() loops, waiting for new cleanup nodes in our cleanup queue. If there is a new node, we dequeue the node, unlock the cleanup queue (so that other cleanup nodes can be added by our worker threads), join with our new thread (using the TID stored in the cleanup node), free the cleanup node, decrement the number of threads "out there", and continue.

Wrapping it up

We've reached the end of the "POSIX threads explained" series, and I hope that you're now ready to begin adding multithreaded code to your own applications. For more information, please see the Resources section, which also contains a tarball of all the sources used in this article. I'll see you next series!



   Note

Browse all our available articles below. Use the search field to search for topics and keywords in real-time.

Article Subtitle
Article Subtitle
Awk by Example, Part 1 An intro to the great language with the strange name
Awk by Example, Part 2 Records, loops, and arrays
Awk by Example, Part 3 String functions and ... checkbooks?
Bash by Example, Part 1 Fundamental programming in the Bourne again shell (bash)
Bash by Example, Part 2 More bash programming fundamentals
Bash by Example, Part 3 Exploring the ebuild system
BTRFS Fun
Funtoo Filesystem Guide, Part 1 Journaling and ReiserFS
Funtoo Filesystem Guide, Part 2 Using ReiserFS and Linux
Funtoo Filesystem Guide, Part 3 Tmpfs and Bind Mounts
Funtoo Filesystem Guide, Part 4 Introducing Ext3
Funtoo Filesystem Guide, Part 5 Ext3 in Action
GUID Booting Guide
Learning Linux LVM, Part 1 Storage management magic with Logical Volume Management
Learning Linux LVM, Part 2 The cvs.gentoo.org upgrade
Libvirt
Linux Fundamentals, Part 1
Linux Fundamentals, Part 2
Linux Fundamentals, Part 3
Linux Fundamentals, Part 4
LVM Fun
Making the Distribution, Part 1
Making the Distribution, Part 2
Making the Distribution, Part 3
Maximum Swappage Getting the most out of swap
On screen annotation Write on top of apps on your screen
OpenSSH Key Management, Part 1 Understanding RSA/DSA Authentication
OpenSSH Key Management, Part 2 Introducing ssh-agent and keychain
OpenSSH Key Management, Part 3 Agent Forwarding
Partition Planning Tips Keeping things organized on disk
Partitioning in Action, Part 1 Moving /home
Partitioning in Action, Part 2 Consolidating data
POSIX Threads Explained, Part 1 A simple and nimble tool for memory sharing
POSIX Threads Explained, Part 2
POSIX Threads Explained, Part 3 Improve efficiency with condition variables
Sed by Example, Part 1
Sed by Example, Part 2
Sed by Example, Part 3
Successful booting with UUID Guide to use UUID for consistent booting.
The Gentoo.org Redesign, Part 1 A site reborn
The Gentoo.org Redesign, Part 2 The Documentation System
The Gentoo.org Redesign, Part 3 The New Main Pages
The Gentoo.org Redesign, Part 4 The Final Touch of XML
Traffic Control
Windows 10 Virtualization with KVM