Difference between pages "Awk by Example, Part 1" and "Funtoo:Keychain"

From Funtoo
(Difference between pages)
Jump to navigation Jump to search
 
 
Line 1: Line 1:
{{Article
{{Article
|Subtitle=An intro to the great language with the strange name
|Subtitle=Official Project Page
|Keywords=command,unix,variables,print,space
|Summary=Keychain helps you to manage SSH and GPG keys in a convenient and secure manner. Download and learn how to use Keychain on your Linux, Unix or MacOS system.
|Keywords=keychain,ssh,rsa,dsa,gpg,linux,gentoo,macos,download,source code
|Author=Drobbins
|Author=Drobbins
|Next in Series=Awk by Example, Part 2
}}
}}
=== In defense of awk ===
<tt>Keychain</tt> helps you to manage SSH and GPG keys in a convenient and secure manner. It acts as a frontend to <tt>ssh-agent</tt> and <tt>ssh-add</tt>, but allows you to easily have one long running <tt>ssh-agent</tt> process per system, rather than the norm of one <tt>ssh-agent</tt> per login session.
In this series of articles, I'm going to turn you into a proficient awk coder. I'll admit, awk doesn't have a very pretty or particularly "hip" name, and the GNU version of awk, called gawk, sounds downright weird. Those unfamiliar with the language may hear "awk" and think of a mess of code so backwards and antiquated that it's capable of driving even the most knowledgeable UNIX guru to the brink of insanity (causing him to repeatedly yelp "kill -9!" as he runs for coffee machine).


Sure, awk doesn't have a great name. But it is a great language. Awk is geared toward text processing and report generation, yet features many well-designed features that allow for serious programming. And, unlike some languages, awk's syntax is familiar, and borrows some of the best parts of languages like C, python, and bash (although, technically, awk was created before both python and bash). Awk is one of those languages that, once learned, will become a key part of your strategic coding arsenal.
This dramatically reduces the number of times you need to enter your passphrase. With <tt>keychain</tt>, you only need to enter a passphrase once every time your local machine is rebooted. <tt>Keychain</tt> also makes it easy for remote cron jobs to securely "hook in" to a long-running <tt>ssh-agent</tt> process, allowing your scripts to take advantage of key-based logins.


=== The first awk ===
Those who are new to OpenSSH and the use of public/private keys for authentication may want to check out the following articles by Daniel Robbins, which will provide a gentle introduction to the concepts used by Keychain:
Let's go ahead and start playing around with awk to see how it works. At the command line, enter the following command:
* [[OpenSSH Key Management, Part_1]]
* [[OpenSSH Key Management, Part_2]]
* [[OpenSSH Key Management, Part_3]]


<console>$##i## awk '{ print }' /etc/passwd</console>
== Download and Resources ==


You should see the contents of your /etc/passwd file appear before your eyes. Now, for an explanation of what awk did. When we called awk, we specified /etc/passwd as our input file. When we executed awk, it evaluated the print command for each line in /etc/passwd, in order. All output is sent to stdout, and we get a result identical to catting /etc/passwd.
The latest release of keychain is version <tt>2.7.2_beta1</tt>, and was released on July 7, 2014. The current version of keychain supports <tt>gpg-agent</tt> as well as <tt>ssh-agent</tt>.


Now, for an explanation of the { print } code block. In awk, curly braces are used to group blocks of code together, similar to C. Inside our block of code, we have a single print command. In awk, when a print command appears by itself, the full contents of the current line are printed.
Keychain is compatible with many operating systems, including <tt>AIX</tt>, <tt>*BSD</tt>, <tt>Cygwin</tt>, <tt>MacOS X</tt>, <tt>Linux</tt>, <tt>HP/UX</tt>, <tt>Tru64 UNIX</tt>, <tt>IRIX</tt>, <tt>Solaris</tt> and <tt>GNU Hurd</tt>.


Here is another awk example that does exactly the same thing:
=== Download ===


<console>$##i## awk '{ print $0 }' /etc/passwd</console>
* ''Release Archive''
** [http://www.funtoo.org/distfiles/keychain/keychain-2.7.2_beta1.tar.bz2 keychain 2.7.2_beta1]
** [http://www.funtoo.org/distfiles/keychain/keychain-2.7.1.tar.bz2 keychain 2.7.1]


In awk, the $0 variable represents the entire current line, so print and print $0 do exactly the same thing. If you'd like, you can create an awk program that will output data totally unrelated to the input data. Here's an example:
* ''Apple MacOS X Packages''
** [http://www.funtoo.org/distfiles/keychain/keychain-2.7.1-macosx.tar.gz keychain 2.7.1 MacOS X package]


<console>$##i## awk '{ print "" }' /etc/passwd</console>
Keychain development sources can be found in the [http://www.github.com/funtoo/keychain keychain git repository]. Please use the [https://bugs.funtoo.org Funtoo Linux bug tracker] and [irc://irc.freenode.net/funtoo #funtoo irc channel] for keychain support questions as well as bug reports.


Whenever you pass the "" string to the print command, it prints a blank line. If you test this script, you'll find that awk outputs one blank line for every line in your /etc/passwd file. Again, this is because awk executes your script for every line in the input file. Here's another example:
=== Project History ===


<console>$##i## awk '{ print "hiya" }' /etc/passwd</console>
Daniel Robbins originally wrote <tt>keychain</tt> 1.0 through 2.0.3. 1.0 was written around June 2001, and 2.0.3 was released in late August, 2002.


Running this script will fill your screen with hiya's. :)
After 2.0.3, <tt>keychain</tt> was maintained by various Gentoo developers, including Seth Chandler, Mike Frysinger and Robin H. Johnson, through July 3, 2003.


=== Multiple fields ===
On April 21, 2004, Aron Griffis committed a major rewrite of <tt>keychain</tt> which was released as 2.2.0. Aron continued to actively maintain and improve <tt>keychain</tt> through October 2006 and the <tt>keychain</tt> 2.6.8 release. He also made a few commits after that date, up through mid-July, 2007. At this point, <tt>keychain</tt> had reached a point of maturity.
Awk is really good at handling text that has been broken into multiple logical fields, and allows you to effortlessly reference each individual field from inside your awk script. The following script will print out a list of all user accounts on your system:


<console>$##i## awk -F":" '{ print $1 }' /etc/passwd</console>  
In mid-July, 2009, Daniel Robbins migrated Aron's mercurial repository to git and set up a new project page on funtoo.org, and made a few bug fix commits to the git repo that had been collecting in [http://bugs.gentoo.org bugs.gentoo.org]. Daniel continues to maintain <tt>keychain</tt> and supporting documentation on funtoo.org, and plans to make regular maintenance releases of <tt>keychain</tt> as needed.


Above, when we called awk, we use the -F option to specify ":" as the field separator. When awk processes the print $1 command, it will print out the first field that appears on each line in the input file. Here's another example:
== Quick Setup ==


<console>$##i## awk -F":" '{ print $1 $3 }' /etc/passwd</console>
=== Linux ===


Here's an excerpt of the output from this script:
To install under Gentoo or Funtoo Linux, type
<pre>
<console>
halt7
###i## emerge keychain
operator11
</console>
root0
shutdown6
sync5
bin1
....etc.
</pre>
As you can see, awk prints out the first and third fields of the /etc/passwd file, which happen to be the username and uid fields respectively. Now, while the script did work, it's not perfect -- there aren't any spaces between the two output fields! If you're used to programming in bash or python, you may have expected the print $1 $3 command to insert a space between the two fields. However, when two strings appear next to each other in an awk program, awk concatenates them without adding an intermediate space. The following command will insert a space between both fields:


<console>$##i## awk -F":" '{ print $1 " " $3 }' /etc/passwd</console>
For other Linux distributions, use your distribution's package manager, or download and install using the source tarball above. Then generate RSA/DSA keys if necessary. The quick install docs assume you have a DSA key pair named <tt>id_dsa</tt> and <tt>id_dsa.pub</tt> in your <tt>~/.ssh/</tt> directory. Add the following to your <tt>~/.bash_profile</tt>:


When you call print this way, it'll concatenate $1, " ", and $3, creating readable output. Of course, we can also insert some text labels if needed:
{{file|name=~/.bash_profile|body=
eval `keychain --eval --agents ssh id_rsa`
}}
 
If you want to take advantage of GPG functionality, ensure that GNU Privacy Guard is installed and omit the <tt>--agents ssh</tt> option above.
 
=== Apple MacOS X ===
 
To install under MacOS X, install the MacOS X package for keychain. Assuming you have an <tt>id_dsa</tt> and <tt>id_dsa.pub</tt> key pair in your <tt>~/.ssh/</tt> directory, add the following to your <tt>~/.bash_profile</tt>:
 
{{file|name=~/.bash_profile|body=
eval `keychain --eval --agents ssh --inherit any id_dsa`
}}
 
{{Fancynote|The <tt>--inherit any</tt> option above causes keychain to inherit any ssh key passphrases stored in your Apple MacOS Keychain. If you would prefer for this to not happen, then this option can be omitted.}}
 
== Background ==
 
You're probably familiar with <tt>ssh</tt>, which has become a secure replacement for the venerable <tt>telnet</tt> and <tt>rsh</tt> commands.
 
Typically, when one uses <tt>ssh</tt> to connect to a remote system, one supplies a secret passphrase to <tt>ssh</tt>, which is then passed in encrypted form over the network to the remote server. This passphrase is used by the remote <tt>sshd</tt> server to determine if you should be granted access to the system.
 
However, OpenSSH and nearly all other SSH clients and servers have the ability to perform another type of authentication, called asymmetric public key authentication, using the RSA or DSA authentication algorithms. They are very useful, but can also be complicated to use. <tt>keychain</tt> has been designed to make it easy to take advantage of the benefits of RSA and DSA authentication.
 
== Generating a Key Pair ==
 
To use RSA and DSA authentication, first you use a program called <tt>ssh-keygen</tt> (included with OpenSSH) to generate a ''key pair'' -- two small files. One of the files is the ''public key''. The other small file contains the ''private key''. <tt>ssh-keygen</tt> will ask you for a passphrase, and this passphrase will be used to encrypt your private key. You will need to supply this passphrase to use your private key. If you wanted to generate a DSA key pair, you would do this:
 
<console># ##i##ssh-keygen -t dsa
Generating public/private dsa key pair.</console>
You would then be prompted for a location to store your key pair. If you do not have one currently stored in <tt>~/.ssh</tt>, it is fine to accept the default location:


<console>$##i## awk -F":" '{ print "username: " $1 "\t\tuid:" $3 }' /etc/passwd</console>
<console>Enter file in which to save the key (/root/.ssh/id_dsa): </console>
Then, you are prompted for a passphrase. This passphrase is used to encrypt the ''private key'' on disk, so even if it is stolen, it will be difficult for someone else to use it to successfully authenticate as you with any accounts that have been configured to recognize your public key.


This will cause the output to be:
Note that conversely, if you '''do not''' provide a passphrase for your private key file, then your private key file '''will not''' be encrypted. This means that if someone steals your private key file, ''they will have the full ability to authenticate with any remote accounts that are set up with your public key.''
<pre>
username: halt    uid:7
username: operator uid:11
username: root    uid:0
username: shutdown uid:6
username: sync    uid:5
username: bin      uid:1
....etc.  
</pre>


=== External Scripts ===
Below, I have supplied a passphrase so that my private key file will be encrypted on disk:
Passing your scripts to awk as a command line argument can be very handy for small one-liners, but when it comes to complex, multi-line programs, you'll definitely want to compose your script in an external file. Awk can then be told to source this script file by passing it the -f option:


<console>$##i## awk -f myscript.awk myfile.in </console>
<console>Enter passphrase (empty for no passphrase): ##i#########
Enter same passphrase again: ##i#########
Your identification has been saved in /var/tmp/id_dsa.
Your public key has been saved in /var/tmp/id_dsa.pub.
The key fingerprint is:
5c:13:ff:46:7d:b3:bf:0e:37:1e:5e:8c:7b:a3:88:f4 root@devbox-ve
The key's randomart image is:
+--[ DSA 1024]----+
|          .      |
|          o  . |
|          o . ..o|
|      . . . o  +|
|        S    o. |
|            . o.|
|        .  ..++|
|        . o . =o*|
|        . E .+*.|
+-----------------+</console>


Putting your scripts in their own text files also allows you to take advantage of additional awk features. For example, this multi-line script does the same thing as one of our earlier one-liners, printing out the first field of each line in /etc/passwd:
== Setting up Authentication ==
<pre>
BEGIN {
        FS=":"
}
{ print $1 }
</pre>
The difference between these two methods has to do with how we set the field separator. In this script, the field separator is specified within the code itself (by setting the FS variable), while our previous example set FS by passing the -F":" option to awk on the command line. It's generally best to set the field separator inside the script itself, simply because it means you have one less command line argument to remember to type. We'll cover the FS variable in more detail later in this article.


It is also possible to make the script directly executable, by placing a "#!/usr/bin/awk -f" at the top of the file, as follows:
Here's how you use these files to authenticate with a remote server. On the remote server, you would append the contents of your ''public key'' to the <tt>~.ssh/authorized_keys</tt> file, if such a file exists. If it doesn't exist, you can simply create a new <tt>authorized_keys</tt> file in the remote account's <tt>~/.ssh</tt> directory that contains the contents of your local <tt>id_dsa.pub</tt> file.
<pre>
#!/usr/bin/awk -f
BEGIN {
FS=":"
}
{ print $1 }
</pre>
Next, the script must be made executable by setting the script file's execute bit:


<console>$##i## chmod +x myscript.awk</console>
Then, if you weren't going to use <tt>keychain</tt>, you'd perform the following steps. On your local client, you would start a program called <tt>ssh-agent</tt>, which runs in the background. Then you would use a program called <tt>ssh-add</tt> to tell <tt>ssh-agent</tt> about your secret private key. Then, if you've set up your environment properly, the next time you run <tt>ssh</tt>, it will find <tt>ssh-agent</tt> running, grab the private key that you added to <tt>ssh-agent</tt> using <tt>ssh-add</tt>, and use this key to authenticate with the remote server.


Now, you should be able to execute the script as follows:
Again, the steps in the previous paragraph is what you'd do if <tt>keychain</tt> wasn't around to help. If you are using <tt>keychain</tt>, and I hope you are, you would simply add the following line to your <tt>~/.bash_profile</tt> or if a regular user to<tt>~/.bashrc</tt> :


<console>$##i## ./myscript.awk myfile.in</console>
{{file|name=~/.bash_profile|body=
eval `keychain --eval id_dsa`
}}


=== The BEGIN and END blocks ===
The next time you log in or source your <tt>~/.bash_profile</tt> or if you use <tt>~/.bashrc</tt>, <tt>keychain</tt> will start, start <tt>ssh-agent</tt> for you if it has not yet been started, use <tt>ssh-add</tt> to add your <tt>id_dsa</tt> private key file to <tt>ssh-agent</tt>, and set up your shell environment so that <tt>ssh</tt> will be able to find <tt>ssh-agent</tt>. If <tt>ssh-agent</tt> is already running, <tt>keychain</tt> will ensure that your <tt>id_dsa</tt> private key has been added to <tt>ssh-agent</tt> and then set up your environment so that <tt>ssh</tt> can find the already-running <tt>ssh-agent</tt>. It will look something like this:
Normally, awk executes each block of your script's code once for each input line. However, there are many programming situations where you may need to execute initialization code before awk begins processing the text from the input file. For such situations, awk allows you to define a BEGIN block. We used a BEGIN block in the previous example. Because the BEGIN block is evaluated before awk starts processing the input file, it's an excellent place to initialize the FS (field separator) variable, print a heading, or initialize other global variables that you'll reference later in the program.


Awk also provides another special block, called the END block. Awk executes this block after all lines in the input file have been processed. Typically, the END block is used to perform final calculations or print summaries that should appear at the end of the output stream.
Note that when <tt>keychain</tt> runs for the first time after your local system has booted, you will be prompted for a passphrase for your private key file if it is encrypted. But here's the nice thing about using <tt>keychain</tt> -- even if you are using an encrypted private key file, you will only need to enter your passphrase when your system first boots (or in the case of a server, when you first log in.) After that, <tt>ssh-agent</tt> is already running and has your decrypted private key cached in memory. So if you open a new shell, you will see something like this:


=== Regular expressions and blocks ===
This means that you can now <tt>ssh</tt> to your heart's content, without supplying a passphrase.
Awk allows the use of regular expressions to selectively execute an individual block of code, depending on whether or not the regular expression matches the current line. Here's an example script that outputs only those lines that contain the character sequence foo:


<pre>/foo/ { print }</pre>
You can also execute batch <tt>cron</tt> jobs and scripts that need to use <tt>ssh</tt> or <tt>scp</tt>, and they can take advantage of passwordless RSA/DSA authentication as well. To do this, you would add the following line to the top of a bash script:


Of course, you can use more complicated regular expressions. Here's a script that will print only lines that contain a floating point number:
{{file|name=example-script.sh|body=
eval `keychain --noask --eval id_dsa` || exit 1
}}
 
The extra <tt>--noask</tt> option tells <tt>keychain</tt> that it should not prompt for a passphrase if one is needed. Since it is not running interactively, it is better for the script to fail if the decrypted private key isn't cached in memory via <tt>ssh-agent</tt>.
 
== Keychain Options ==
 
=== Specifying Agents ===
 
In the images above, you will note that <tt>keychain</tt> starts <tt>ssh-agent</tt>, but also starts <tt>gpg-agent</tt>. Modern versions of <tt>keychain</tt> also support caching decrypted GPG keys via use of <tt>gpg-agent</tt>, and will start <tt>gpg-agent</tt> by default if it is available on your system. To avoid this behavior and only start <tt>ssh-agent</tt>, modify your <tt>~/.bash_profile</tt> as follows:
 
{{file|name=~/.bash_profile|body=
eval `keychain --agents ssh --eval id_dsa` || exit 1
}}


<pre>/[0-9]+\.[0-9]*/ { print }</pre>
The additional <tt>--agents ssh</tt> option tells <tt>keychain</tt> just to manage <tt>ssh-agent</tt>, and ignore <tt>gpg-agent</tt> even if it is available.


=== Expressions and blocks ===
=== Clearing Keys ===
There are many other ways to selectively execute a block of code. We can place any kind of boolean expression before a code block to control when a particular block is executed. Awk will execute a code block only if the preceding boolean expression evaluates to true. The following example script will output the third field of all lines that have a first field equal to fred. If the first field of the current line is not equal to fred, awk will continue processing the file and will not execute the print statement for the current line:


<pre>$1 == "fred" { print $3 }</pre>
Sometimes, it might be necessary to flush all cached keys in memory. To do this, type:


Awk offers a full selection of comparison operators, including the usual "==", "<", ">", "<=", ">=", and "!=". In addition, awk provides the "~" and "!~" operators, which mean "matches" and "does not match". They're used by specifying a variable on the left side of the operator, and a regular expression on the right side. Here's an example that will print only the third field on the line if the fifth field on the same line contains the character sequence root:
<console># ##i##keychain --clear</console>
Any agent(s) will continue to run.


<pre>$5 ~ /root/ { print $3 }</pre>
=== Improving Security ===


=== Conditional statements ===
To improve the security of <tt>keychain</tt>, some people add the <tt>--clear</tt> option to their <tt>~/.bash_profile</tt> <tt>keychain</tt> invocation. The rationale behind this is that any user logging in should be assumed to be an intruder until proven otherwise. This means that you will need to re-enter any passphrases when you log in, but cron jobs will still be able to run when you log out.
Awk also offers very nice C-like if statements. If you'd like, you could rewrite the previous script using an if statement:
<pre>
{
    if ( $5 ~ /root/ ) {
        print $3
    }
}
</pre>
Both scripts function identically. In the first example, the boolean expression is placed outside the block, while in the second example, the block is executed for every input line, and we selectively perform the print command by using an if statement. Both methods are available, and you can choose the one that best meshes with the other parts of your script.


Here's a more complicated example of an awk if statement. As you can see, even with complex, nested conditionals, if statements look identical to their C counterparts:
=== Stopping Agents ===
<pre>
{
    if ( $1 == "foo" ) {
        if ( $2 == "foo" ) {
            print "uno"
        } else {
            print "one"
        }
    } else if ($1 == "bar" ) {
        print "two"
    } else {
        print "three"
    }
}
</pre>
Using if statements, we can also transform this code:
<pre>
! /matchme/ { print $1 $3 $4 }
</pre>
to this:
<pre>
{
    if ( $0 !~ /matchme/ ) {
        print $1 $3 $4
    }
}
</pre>
Both scripts will output only those lines that don't contain a matchme character sequence. Again, you can choose the method that works best for your code. They both do the same thing.


Awk also allows the use of boolean operators "||" (for "logical or") and "&&"(for "logical and") to allow the creation of more complex boolean expressions:
If you want to stop all agents, which will also of course cause your keys/identities to be flushed from memory, you can do this as follows:
<pre>
( $1 == "foo" ) && ( $2 == "bar" ) { print }
</pre>
This example will print only those lines where field one equals foo and field two equals bar.


=== Numeric variables! ===
<console># ##i##keychain -k all</console>
So far, we've either printed strings, the entire line, or specific fields. However, awk also allows us to perform both integer and floating point math. Using mathematical expressions, it's very easy to write a script that counts the number of blank lines in a file. Here's one that does just that:
If you have other agents running under your user account, you can also tell <tt>keychain</tt> to just stop only the agents that <tt>keychain</tt> started:
<pre>
BEGIN { x=0 }
/^$/  { x=x+1 }
END  { print "I found " x " blank lines. :)" }
</pre>
In the BEGIN block, we initialize our integer variable x to zero. Then, each time awk encounters a blank line, awk will execute the x=x+1 statement, incrementing x. After all the lines have been processed, the END block will execute, and awk will print out a final summary, specifying the number of blank lines it found.


=== Stringy variables ===
<console># ##i##keychain -k mine</console>
One of the neat things about awk variables is that they are "simple and stringy." I consider awk variables "stringy" because all awk variables are stored internally as strings. At the same time, awk variables are "simple" because you can perform mathematical operations on a variable, and as long as it contains a valid numeric string, awk automatically takes care of the string-to-number conversion steps. To see what I mean, check out this example:
<pre>
x="1.01"
# We just set x to contain the *string* "1.01"
x=x+1
# We just added one to a *string*
print x
# Incidentally, these are comments :)
</pre>
Awk will output:
<pre>
2.01
</pre>
Interesting! Although we assigned the string value 1.01 to the variable x, we were still able to add one to it. We wouldn't be able to do this in bash or python. First of all, bash doesn't support floating point arithmetic. And, while bash has "stringy" variables, they aren't "simple"; to perform any mathematical operations, bash requires that we enclose our math in an ugly $( ) construct. If we were using python, we would have to explicitly convert our 1.01 string to a floating point value before performing any arithmetic on it. While this isn't difficult, it's still an additional step. With awk, it's all automatic, and that makes our code nice and clean. If we wanted to square and add one to the first field in each input line, we would use this script:
<pre>
{ print ($1^2)+1 }
</pre>
If you do a little experimenting, you'll find that if a particular variable doesn't contain a valid number, awk will treat that variable as a numerical zero when it evaluates your mathematical expression.


=== Lots of operators ===
=== GPG ===
Another nice thing about awk is its full complement of mathematical operators. In addition to standard addition, subtraction, multiplication, and division, awk allows us to use the previously demonstrated exponent operator "^", the modulo (remainder) operator "%", and a bunch of other handy assignment operators borrowed from C.


These include pre- and post-increment/decrement ( i++, --foo ), add/sub/mult/div assign operators ( a+=3, b*=2, c/=2.2, d-=6.2 ). But that's not all -- we also get handy modulo/exponent assign ops as well ( a^=2, b%=4 ).
Keychain can ask you for your GPG passphrase if you provide it the GPG key ID. To find it out:
<console>
$##i## gpg -k
pub  2048R/DEADBEEF 2012-08-16
uid                  Name (Comment) <email@host.tld>
sub   2048R/86D2FAC6 2012-08-16
</console>


=== Field separators ===
Note the '''DEADBEEF''' above is the ID. Then, in your login script, do your usual
Awk has its own complement of special variables. Some of them allow you to fine-tune how awk functions, while others can be read to glean valuable information about the input. We've already touched on one of these special variables, FS. As mentioned earlier, this variable allows you to set the character sequence that awk expects to find between fields. When we were using /etc/passwd as input, FS was set to ":". While this did the trick, FS allows us even more flexibility.


The FS value is not limited to a single character; it can also be set to a regular expression, specifying a character pattern of any length. If you're processing fields separated by one or more tabs, you'll want to set FS like so:
<console>
<pre>
$##i## keychain --dir ~/.ssh/.keychain ~/.ssh/id_rsa DEADBEEF
FS="\t+"
$##i## source ~/.ssh/.keychain/$HOST-sh
</pre>
$##i## source ~/.ssh/.keychain/$HOST-sh-gpg
Above, we use the special "+" regular expression character, which means "one or more of the previous character".
</console>


If your fields are separated by whitespace (one or more spaces or tabs), you may be tempted to set FS to the following regular expression:
=== Learning More ===
<pre>
FS="[[:space:]]+"
</pre>
While this assignment will do the trick, it's not necessary. Why? Because by default, FS is set to a single space character, which awk interprets to mean "one or more spaces or tabs." In this particular example, the default FS setting was exactly what you wanted in the first place!


Complex regular expressions are no problem. Even if your records are separated by the word "foo," followed by three digits, the following regular expression will allow your data to be parsed properly:
The instructions above will work on any system that uses <tt>bash</tt> as its default shell, such as most Linux systems and Mac OS X.
<pre>
FS="foo[0-9][0-9][0-9]"
</pre>


=== Number of fields ===
To learn more about the many things that <tt>keychain</tt> can do, including alternate shell support, consult the keychain man page, or type <tt>keychain --help | less</tt> for a full list of command options.
The next two variables we're going to cover are not normally intended to be written to, but are normally read and used to gain useful information about the input. The first is the NF variable, also called the "number of fields" variable. Awk will automatically set this variable to the number of fields in the current record. You can use the NF variable to display only certain input lines:
<pre>
NF == 3 { print "this particular record has three fields: " $0 }
</pre>
Of course, you can also use the NF variable in conditional statements, as follows:
<pre>
{
    if ( NF > 2 ) {
        print $1 " " $2 ":" $3
    }
}
</pre>


=== Record number ===
I also recommend you read my original series of articles about [http://www.openssh.com OpenSSH] that I wrote for IBM developerWorks, called <tt>OpenSSH Key Management</tt>. Please note that <tt>keychain</tt> 1.0 was released along with Part 2 of this article, which was written in 2001. <tt>keychain</tt> has changed quite a bit since then. In other words, read these articles for the conceptual and [http://www.openssh.com OpenSSH] information, but consult the <tt>keychain</tt> man page for command-line options and usage instructions :)
The record number (NR) is another handy variable. It will always contain the number of the current record (awk counts the first record as record number 1). Up until now, we've been dealing with input files that contain one record per line. For these situations, NR will also tell you the current line number. However, when we start to process multi-line records later in the series, this will no longer be the case, so be careful! NR can be used like the NF variable to print only certain lines of the input:
<pre>
(NR < 10 ) || (NR > 100) { print "We are on record number 1-9 or 101+" }
</pre>
<pre>
{
    #skip header
    if ( NR > 10 ) {
        print "ok, now for the real information!"
    }
}
</pre>
Awk provides additional variables that can be used for a variety of purposes. We'll cover more of these variables in later articles.


We've come to the end of our initial exploration of awk. As the series continues, I'll demonstrate more advanced awk functionality, and we'll end the series with a real-world awk application.
* [http://www.ibm.com/developerworks/library/l-keyc.html Common Threads: OpenSSH key management, Part 1] - Understanding RSA/DSA Authentication
* [http://www.ibm.com/developerworks/library/l-keyc2/ Common Threads: OpenSSH key management, Part 2] - Introducing <tt>ssh-agent</tt> and <tt>keychain</tt>
* [http://www.ibm.com/developerworks/library/l-keyc3/ Common Threads: OpenSSH key management, Part 3] - Agent forwarding and <tt>keychain</tt> improvements


== Resources ==
As mentioned at the top of the page, <tt>keychain</tt> development sources can be found in the [http://www.github.com/funtoo/keychain keychain git repository]. Please use the [http://groups.google.com/group/funtoo-dev funtoo-dev mailing list] and [irc://irc.freenode.net/funtoo #funtoo irc channel] for keychain support questions as well as bug reports.


* Read Daniel's other awk articles on Funtoo: Awk By Example, [[Awk by example, Part2 |Part 2]] and [[Awk by example, Part3 |Part 3]].
[[Category:HOWTO]]
* If you'd like a good old-fashioned book, [http://www.oreilly.com/catalog/sed2/ O'Reilly's sed & awk, 2nd Edition] is a wonderful choice.
[[Category:Projects]]
* Be sure to check out the [http://www.faqs.org/faqs/computer-lang/awk/faq/ comp.lang.awk FAQ]. It also contains lots of additional awk links.
[[Category:First Steps]]
* Patrick Hartigan's [http://sparky.rice.edu/~hartigan/awk.html awk tutorial] is packed with handy awk scripts.
* [http://www.tasoft.com/tawk.html Thompson's TAWK Compiler] compiles awk scripts into fast binary executables. Versions are available for Windows, OS/2, DOS, and UNIX.
* [http://www.gnu.org/software/gawk/manual/gawk.html The GNU Awk User's Guide] is available for online reference.
* [http://www.folkstalk.com/2011/12/good-examples-of-awk-command-in-unix.html Awk Command] daily useful examples.
[[Category:Linux Core Concepts]]
[[Category:Articles]]
[[Category:Articles]]
{{ArticleFooter}}
{{ArticleFooter}}

Revision as of 16:22, January 5, 2015

Official Project Page

Keychain helps you to manage SSH and GPG keys in a convenient and secure manner. Download and learn how to use Keychain on your Linux, Unix or MacOS system.
   Support Funtoo!
Get an awesome Funtoo container and support Funtoo! See Funtoo Containers for more information.

Keychain helps you to manage SSH and GPG keys in a convenient and secure manner. It acts as a frontend to ssh-agent and ssh-add, but allows you to easily have one long running ssh-agent process per system, rather than the norm of one ssh-agent per login session.

This dramatically reduces the number of times you need to enter your passphrase. With keychain, you only need to enter a passphrase once every time your local machine is rebooted. Keychain also makes it easy for remote cron jobs to securely "hook in" to a long-running ssh-agent process, allowing your scripts to take advantage of key-based logins.

Those who are new to OpenSSH and the use of public/private keys for authentication may want to check out the following articles by Daniel Robbins, which will provide a gentle introduction to the concepts used by Keychain:

Download and Resources

The latest release of keychain is version 2.7.2_beta1, and was released on July 7, 2014. The current version of keychain supports gpg-agent as well as ssh-agent.

Keychain is compatible with many operating systems, including AIX, *BSD, Cygwin, MacOS X, Linux, HP/UX, Tru64 UNIX, IRIX, Solaris and GNU Hurd.

Download

Keychain development sources can be found in the keychain git repository. Please use the Funtoo Linux bug tracker and #funtoo irc channel for keychain support questions as well as bug reports.

Project History

Daniel Robbins originally wrote keychain 1.0 through 2.0.3. 1.0 was written around June 2001, and 2.0.3 was released in late August, 2002.

After 2.0.3, keychain was maintained by various Gentoo developers, including Seth Chandler, Mike Frysinger and Robin H. Johnson, through July 3, 2003.

On April 21, 2004, Aron Griffis committed a major rewrite of keychain which was released as 2.2.0. Aron continued to actively maintain and improve keychain through October 2006 and the keychain 2.6.8 release. He also made a few commits after that date, up through mid-July, 2007. At this point, keychain had reached a point of maturity.

In mid-July, 2009, Daniel Robbins migrated Aron's mercurial repository to git and set up a new project page on funtoo.org, and made a few bug fix commits to the git repo that had been collecting in bugs.gentoo.org. Daniel continues to maintain keychain and supporting documentation on funtoo.org, and plans to make regular maintenance releases of keychain as needed.

Quick Setup

Linux

To install under Gentoo or Funtoo Linux, type

root # emerge keychain

For other Linux distributions, use your distribution's package manager, or download and install using the source tarball above. Then generate RSA/DSA keys if necessary. The quick install docs assume you have a DSA key pair named id_dsa and id_dsa.pub in your ~/.ssh/ directory. Add the following to your ~/.bash_profile:

   ~/.bash_profile
eval `keychain --eval --agents ssh id_rsa`

If you want to take advantage of GPG functionality, ensure that GNU Privacy Guard is installed and omit the --agents ssh option above.

Apple MacOS X

To install under MacOS X, install the MacOS X package for keychain. Assuming you have an id_dsa and id_dsa.pub key pair in your ~/.ssh/ directory, add the following to your ~/.bash_profile:

   ~/.bash_profile
eval `keychain --eval --agents ssh --inherit any id_dsa`
   Note

The --inherit any option above causes keychain to inherit any ssh key passphrases stored in your Apple MacOS Keychain. If you would prefer for this to not happen, then this option can be omitted.

Background

You're probably familiar with ssh, which has become a secure replacement for the venerable telnet and rsh commands.

Typically, when one uses ssh to connect to a remote system, one supplies a secret passphrase to ssh, which is then passed in encrypted form over the network to the remote server. This passphrase is used by the remote sshd server to determine if you should be granted access to the system.

However, OpenSSH and nearly all other SSH clients and servers have the ability to perform another type of authentication, called asymmetric public key authentication, using the RSA or DSA authentication algorithms. They are very useful, but can also be complicated to use. keychain has been designed to make it easy to take advantage of the benefits of RSA and DSA authentication.

Generating a Key Pair

To use RSA and DSA authentication, first you use a program called ssh-keygen (included with OpenSSH) to generate a key pair -- two small files. One of the files is the public key. The other small file contains the private key. ssh-keygen will ask you for a passphrase, and this passphrase will be used to encrypt your private key. You will need to supply this passphrase to use your private key. If you wanted to generate a DSA key pair, you would do this:

root # ssh-keygen -t dsa
Generating public/private dsa key pair.

You would then be prompted for a location to store your key pair. If you do not have one currently stored in ~/.ssh, it is fine to accept the default location:

Enter file in which to save the key (/root/.ssh/id_dsa): 

Then, you are prompted for a passphrase. This passphrase is used to encrypt the private key on disk, so even if it is stolen, it will be difficult for someone else to use it to successfully authenticate as you with any accounts that have been configured to recognize your public key.

Note that conversely, if you do not provide a passphrase for your private key file, then your private key file will not be encrypted. This means that if someone steals your private key file, they will have the full ability to authenticate with any remote accounts that are set up with your public key.

Below, I have supplied a passphrase so that my private key file will be encrypted on disk:

Enter passphrase (empty for no passphrase): #######
Enter same passphrase again: #######
Your identification has been saved in /var/tmp/id_dsa.
Your public key has been saved in /var/tmp/id_dsa.pub.
The key fingerprint is:
5c:13:ff:46:7d:b3:bf:0e:37:1e:5e:8c:7b:a3:88:f4 root@devbox-ve
The key's randomart image is:
+--[ DSA 1024]----+
|          .      |
|           o   . |
|          o . ..o|
|       . . . o  +|
|        S     o. |
|             . o.|
|         .   ..++|
|        . o . =o*|
|         . E .+*.|
+-----------------+

Setting up Authentication

Here's how you use these files to authenticate with a remote server. On the remote server, you would append the contents of your public key to the ~.ssh/authorized_keys file, if such a file exists. If it doesn't exist, you can simply create a new authorized_keys file in the remote account's ~/.ssh directory that contains the contents of your local id_dsa.pub file.

Then, if you weren't going to use keychain, you'd perform the following steps. On your local client, you would start a program called ssh-agent, which runs in the background. Then you would use a program called ssh-add to tell ssh-agent about your secret private key. Then, if you've set up your environment properly, the next time you run ssh, it will find ssh-agent running, grab the private key that you added to ssh-agent using ssh-add, and use this key to authenticate with the remote server.

Again, the steps in the previous paragraph is what you'd do if keychain wasn't around to help. If you are using keychain, and I hope you are, you would simply add the following line to your ~/.bash_profile or if a regular user to~/.bashrc :

   ~/.bash_profile
eval `keychain --eval id_dsa`

The next time you log in or source your ~/.bash_profile or if you use ~/.bashrc, keychain will start, start ssh-agent for you if it has not yet been started, use ssh-add to add your id_dsa private key file to ssh-agent, and set up your shell environment so that ssh will be able to find ssh-agent. If ssh-agent is already running, keychain will ensure that your id_dsa private key has been added to ssh-agent and then set up your environment so that ssh can find the already-running ssh-agent. It will look something like this:

Note that when keychain runs for the first time after your local system has booted, you will be prompted for a passphrase for your private key file if it is encrypted. But here's the nice thing about using keychain -- even if you are using an encrypted private key file, you will only need to enter your passphrase when your system first boots (or in the case of a server, when you first log in.) After that, ssh-agent is already running and has your decrypted private key cached in memory. So if you open a new shell, you will see something like this:

This means that you can now ssh to your heart's content, without supplying a passphrase.

You can also execute batch cron jobs and scripts that need to use ssh or scp, and they can take advantage of passwordless RSA/DSA authentication as well. To do this, you would add the following line to the top of a bash script:

   example-script.sh
eval `keychain --noask --eval id_dsa`

The extra --noask option tells keychain that it should not prompt for a passphrase if one is needed. Since it is not running interactively, it is better for the script to fail if the decrypted private key isn't cached in memory via ssh-agent.

Keychain Options

Specifying Agents

In the images above, you will note that keychain starts ssh-agent, but also starts gpg-agent. Modern versions of keychain also support caching decrypted GPG keys via use of gpg-agent, and will start gpg-agent by default if it is available on your system. To avoid this behavior and only start ssh-agent, modify your ~/.bash_profile as follows:

   ~/.bash_profile
eval `keychain --agents ssh --eval id_dsa`

The additional --agents ssh option tells keychain just to manage ssh-agent, and ignore gpg-agent even if it is available.

Clearing Keys

Sometimes, it might be necessary to flush all cached keys in memory. To do this, type:

root # keychain --clear

Any agent(s) will continue to run.

Improving Security

To improve the security of keychain, some people add the --clear option to their ~/.bash_profile keychain invocation. The rationale behind this is that any user logging in should be assumed to be an intruder until proven otherwise. This means that you will need to re-enter any passphrases when you log in, but cron jobs will still be able to run when you log out.

Stopping Agents

If you want to stop all agents, which will also of course cause your keys/identities to be flushed from memory, you can do this as follows:

root # keychain -k all

If you have other agents running under your user account, you can also tell keychain to just stop only the agents that keychain started:

root # keychain -k mine

GPG

Keychain can ask you for your GPG passphrase if you provide it the GPG key ID. To find it out:

user $ gpg -k
pub   2048R/DEADBEEF 2012-08-16
uid                  Name (Comment) <email@host.tld>
sub   2048R/86D2FAC6 2012-08-16

Note the DEADBEEF above is the ID. Then, in your login script, do your usual

user $ keychain --dir ~/.ssh/.keychain ~/.ssh/id_rsa DEADBEEF
user $ source ~/.ssh/.keychain/$HOST-sh
user $ source ~/.ssh/.keychain/$HOST-sh-gpg

Learning More

The instructions above will work on any system that uses bash as its default shell, such as most Linux systems and Mac OS X.

To learn more about the many things that keychain can do, including alternate shell support, consult the keychain man page, or type keychain --help | less for a full list of command options.

I also recommend you read my original series of articles about OpenSSH that I wrote for IBM developerWorks, called OpenSSH Key Management. Please note that keychain 1.0 was released along with Part 2 of this article, which was written in 2001. keychain has changed quite a bit since then. In other words, read these articles for the conceptual and OpenSSH information, but consult the keychain man page for command-line options and usage instructions :)

As mentioned at the top of the page, keychain development sources can be found in the keychain git repository. Please use the funtoo-dev mailing list and #funtoo irc channel for keychain support questions as well as bug reports.


   Note

Browse all our available articles below. Use the search field to search for topics and keywords in real-time.

Article Subtitle
Article Subtitle
Awk by Example, Part 1 An intro to the great language with the strange name
Awk by Example, Part 2 Records, loops, and arrays
Awk by Example, Part 3 String functions and ... checkbooks?
Bash by Example, Part 1 Fundamental programming in the Bourne again shell (bash)
Bash by Example, Part 2 More bash programming fundamentals
Bash by Example, Part 3 Exploring the ebuild system
BTRFS Fun
Funtoo Filesystem Guide, Part 1 Journaling and ReiserFS
Funtoo Filesystem Guide, Part 2 Using ReiserFS and Linux
Funtoo Filesystem Guide, Part 3 Tmpfs and Bind Mounts
Funtoo Filesystem Guide, Part 4 Introducing Ext3
Funtoo Filesystem Guide, Part 5 Ext3 in Action
GUID Booting Guide
Learning Linux LVM, Part 1 Storage management magic with Logical Volume Management
Learning Linux LVM, Part 2 The cvs.gentoo.org upgrade
Libvirt
Linux Fundamentals, Part 1
Linux Fundamentals, Part 2
Linux Fundamentals, Part 3
Linux Fundamentals, Part 4
LVM Fun
Making the Distribution, Part 1
Making the Distribution, Part 2
Making the Distribution, Part 3
Maximum Swappage Getting the most out of swap
On screen annotation Write on top of apps on your screen
OpenSSH Key Management, Part 1 Understanding RSA/DSA Authentication
OpenSSH Key Management, Part 2 Introducing ssh-agent and keychain
OpenSSH Key Management, Part 3 Agent Forwarding
Partition Planning Tips Keeping things organized on disk
Partitioning in Action, Part 1 Moving /home
Partitioning in Action, Part 2 Consolidating data
POSIX Threads Explained, Part 1 A simple and nimble tool for memory sharing
POSIX Threads Explained, Part 2
POSIX Threads Explained, Part 3 Improve efficiency with condition variables
Sed by Example, Part 1
Sed by Example, Part 2
Sed by Example, Part 3
Successful booting with UUID Guide to use UUID for consistent booting.
The Gentoo.org Redesign, Part 1 A site reborn
The Gentoo.org Redesign, Part 2 The Documentation System
The Gentoo.org Redesign, Part 3 The New Main Pages
The Gentoo.org Redesign, Part 4 The Final Touch of XML
Traffic Control
Windows 10 Virtualization with KVM