Difference between pages "Package:Whenjobs" and "The Gentoo.org Redesign, Part 1"

(Difference between pages)
 
(Created page with "{{Article |Subtitle=A site reborn |Summary=Have you ever woken up one morning and suddenly realized that your cute little personal development Web site isn't really that great...")
 
Line 1: Line 1:
{{Ebuild
+
{{Article
|Summary=A cron daemon replacement that is powerful, yet simple.
+
|Subtitle=A site reborn
|CatPkg=sys-process/whenjobs
+
|Summary=Have you ever woken up one morning and suddenly realized that your cute little personal development Web site isn't really that great? If so, you're in good company. In this series, Daniel Robbins shares his experiences as he redesigns the www.gentoo.org Web site using technologies like XML, XSLT, and Python. Along the way, you may find some excellent approaches to use for your next Web site redesign. In this article, Daniel creates a user-centric action plan and introduces pytext, an embedded Python interpreter.
|Maintainer=Golodhrim,  
+
|Author=Drobbins
|Repository=Funtoo Overlay
+
 
}}
 
}}
{{fancywarning|This document is a work in progress, as we are still investigating whenjobs for funtoo usage. For information about it see Funtoo tickets about [http://bugs.funtoo.org/browse/FL-316 whenjobs], [http://bugs.funtoo.org/browse/FL-337 initscript], [http://bugs.funtoo.org/browse/FL-338 User Feedback] and [http://bugs.funtoo.org/browse/FL-351 this document].}}
+
== An unruly horde ==
== What are whenjobs? ==
+
  
Whenjobs was written by Richard Jones from [http://www.redhat.com RedHat Linux]. Whenjobs is designed to be a cron daemon replacement with some improvements over normal cron-jobs. Further, we have added some improvements to Whenjobs. Whenjobs gives users a more simple syntax for jobs to run and, with Funtoo improvements, an effective user-management option for whenjobs -- that way we fixed the default behaviour of whenjobs to not been able to run as root and let us execute the daemon on a per-user basis by default.
+
Fellow software developer, may I ask you a question? Why is it that although many of us are intimately familiar with Web technologies such as HTML, CGI, Perl, Python, Java technology, and XML, our very own Web sites -- the ones devoted to our precious development projects -- look like they were thrown together by an unruly horde of hyperactive 12-year-olds? Why, oh why, is this so?
  
=== Questions for help and testing ===
+
Could it be because most of the time, we've left our Web site out to rot while we squander our precious time hacking away on our free software projects? The answer, at least in my case, is a most definite "Yes."
  
We would like to get your feedback to [http://bugs.funtoo.org/browse/FL-338 FL-338]. So, please test and report your experience with whenjobs.
+
When I'm not writing articles for IBM developerWorks or being a new dad, I'm feverishly working on the next release of Gentoo Linux, along with my skilled team of volunteers. And, yes, Gentoo Linux has its own Web site (see Resources). As of right now (March 2001), our Web site isn't that special; that's because we don't spend much time working on it because we're generally engrossed in improving Gentoo Linux itself. Sure, our site does have several admittedly cute logos that I whipped up using Xara X (see Resources), but when you look past the eye candy, our site leaves a lot to be desired. Maybe yours does too. If so, I have one thing to say to you -- welcome to the club.
  
== How to install whenjobs ==
+
== www.gentoo.org ==
  
The installation of {{Package|sys-process/whenjobs}} is really easy, just merge it:
+
In our case, our Web site dilemma exists because our project has been growing, and our Web site hasn't. Now that Gentoo Linux is approaching the 1.0 release (when it'll be officially ready for non-developers) and is growing in popularity, we need to start seriously looking at how our Web site can better serve its users. Here's a snapshot of www.gentoo.org:
  
<console>
+
<div style="margin: 10px;">[[File:L-redesign-01.gif|frame|class=img-responsive|The current (March 2001) state of affairs at www.gentoo.org]]</div>
###i## emerge -avt whenjobs
+
</console>
+
  
== How to get started ==
+
As you can see, we have all the bare essentials -- a description of Gentoo Linux, a features list, a daily Changelog (automatically updated thanks to Python), and a bunch of important links (to the download sites, to our mailing list sign-up pages, and to cvsWeb). We also have links to three documentation resources -- the Gentoo Linux Install Guide and Development Guides, and Christian Zander's NVIDIA Troubleshooting Guide.
  
As mentioned above, we added a user-management feature. Also, whenjobs has a changed syntax compared to normal cronjobs -- we will discuss that now.
+
However, while the site seems O.K., we're missing a lot of things. The most obvious is documentation -- our installation and development guides need a lot of work. And then we need to add an FAQ, new links, new user information...the list is endless.
  
=== User management ===
+
== Content vs. display ==
  
The user management for whenjobs is done with a single file located at <tt>/etc/whenjobs.users.conf</tt>. Just add a user to that file by a comma seperated list like:
+
And now we come to our second problem. Right now, all of our work is done in raw HTML; I hack away at the index.html file until it looks O.K. Even worse, our Web documentation is written in raw HTML. This isn't a good thing from a development perspective because our raw content (consisting of paragraphs, sections, chapters) is garbled together with a bunch of display-related HTML tags. This, of course, makes it difficult to change both the content and the look of our site. While this approach has worked so far, it is bound to cause problems as our site continues to grow.
  
<pre>
+
Clearly, we need to be using better technologies behind the scenes. Instead of using HTML directly, we need to start using things like XML, XSLT, and Python. The goal is to automate as much as possible so that we can add and expand our site with ease. If we do our job well, even major future changes to our site should be relatively painless.
root,user1,user2,user3
+
</pre>
+
  
where user1-user3 must be system-usernames, as they are checked. Please do not add any other lines to that file, as there is no way for comments in the file. In addition, adding other lines will break whenjobs -- you will not be able to start the daemon later. Currently, there is no initscript, but we are working on one and that initscript will then use that file in the same way.
+
== A strategy! ==
  
=== whenjobs commands ===
+
It was clear that we had a lot of work ahead of us. In fact, there was so much to be done that I didn't know where to begin. Just as I was trying to sort out everything in my head, I came across Laura Wonnacott's "Site Savvy" InfoWorld column (see Resources). In it, she explained the concept of "user-centric" design -- how to improve a Web site while keeping the needs of your target audience (in this case, Gentoo Linux users and developers) in focus. Reading the article and taking a look at the "Handbook of User-Centered Design" link from the article helped me to formulate a strategy -- an action plan -- for the redesign:
  
There are some basic commands for whenjobs you should be aware of before we get to explain the script syntax for whenjobs:
+
# First, clearly define the official goal of the Web site -- in writing. What's it there for, and what's it supposed to do?
 +
#  Identify the different categories of users who will be using your site -- your target audience. Rank them in order of priority: Which ones are most important to you?
 +
# Set up a system for getting feedback from your target audience, so they can let you know what you're doing right and wrong.
 +
# Evaluate the feedback, and use it to determine what parts of the site need to be improved or redesigned. Tackle high-priority sections first.
 +
# Once you've selected the part of the site to improve, get to work! During your implementation, make sure that the content and design of the new section caters specifically to the needs of your target audience and fixes all known deficiencies.
 +
# When the section redesign is complete, add it to your live site, even if it has a look that's markedly different from your current site. This way, your users can begin benefitting from the newly redesigned section immediately. If there's a problem with the redesign, you'll get user feedback more quickly. Finally, making incremental improvements to your site (rather than revamping the whole site and then rolling it out all at once -- surprise!) will help prevent your users from feeling alienated by your (possibly dramatic) site changes.
 +
#  After completing step 6, jump to step 4 and repeat.
  
To edit/list a job script, use:
+
== The mission statement ==
  
<console>
+
I was happy to discover that we already had step 3 in place. We had received several e-mail suggestions from visitors to the site, and our developer mailing list also served as a way of exchanging suggestions and comments. However, I had never really completed steps 1 or 2. While the answers may seem obvious, I did find it helpful to actually sit down and write out our mission statement:
# ##i##whenjobs -e | --edit
+
# ##i##whenjobs -l | --list
+
</console>
+
  
in the above case we added both the short and long version in one line so please use either the '''-e''' or the '''--edit''' version as there is no piping done at that part. We will use the same syntax for further examples if there are multiple ways of calling the function we need.
+
www.gentoo.org exists to assist those who use and develop for Gentoo Linux by providing relevant, up-to-date information about Gentoo Linux and Linux in general, focusing on topics related to Gentoo Linux installation, use, administration, and development. As the central hub for all things Gentoo, the site should also feature important news relevant to Gentoo Linux users and developers. In addition to catering to Gentoo Linux users and developers, www.gentoo.org has the secondary purpose of meeting the needs of potential Gentoo Linux users, providing the information they need to decide whether Gentoo Linux is right for them.
  
Another import part is to set or get variables we want to set or set in whenjobs, that can be done with:
+
== The target audience ==
  
<console>
+
So far, so good. Now for step 2 -- defining our target audience:
to get a variable:
+
###i## whenjobs --get variable
+
to set a variable or multiple varibles:
+
###i## whenjobs --set variable=value [variable=value ...]
+
and to display all set variables:
+
###i## whenjobs --variables
+
</console>
+
  
Another important function in whenjobs is the way to start, stop and request the status of the per-user daemon:
+
www.gentoo.org has three target audiences -- Gentoo Linux developers, users, and potential users. While no one group is absolutely a higher priority than another, right now the needs of Gentoo Linux developers are our highest priority, followed by Gentoo Linux users, and then potential users. This is because Gentoo Linux is currently in a prerelease state. When Gentoo Linux reaches version 1.0, Gentoo Linux users and potential users will also become a priority.
<console>
+
# ##i##whenjobs --daemon-start
+
# ##i##whenjobs --daemon-stop
+
# ##i##whenjobs --daemon-status
+
# ##i##whenjobs --daemon-restart
+
</console>
+
  
Finally we have the ability to inspect running jobs:
+
== Comments and suggestions ==
<console>
+
# ##i##whenjobs --jobs
+
# ##i##whenjobs --cancel serial
+
# ##i##whenjobs --start "name"
+
# ##i##whenjobs --tail serial
+
</console>
+
  
=== How to get started with whenjobs now ===
+
O.K., now it's time to evaluate the suggestions and comments we've collected:
  
First edit the above mentioned <tt>whenjobs.users.conf</tt> file if not already done and start a daemon on a per-user basis with
+
Over the past few months, we've received a number of suggestions from Web site visitors. Overwhelmingly, people are requesting better documentation -- for both developers and users. Several developers have asked if we could create a mailing list that would be devoted exclusively to describing CVS commits.
  
<console>
+
Interestingly, we've also received a couple of e-mails asking whether Gentoo Linux is a commercial or free product. I'm guessing that because our main logo is inscribed with the name "Gentoo Technologies, Inc." (our legal corporation name), people assume that we have a commercial focus. Modifying our logo so that it reads "Gentoo Linux" and adding small opening paragraph to the main page explaining that we are a free software project should help.
# ##i##whenjobs --daemon-start
+
</console>
+
  
Thats all for starting whenjobs and now we have time to write some whenjobs-scripts. We will use here some basic examples nothing for real time usage but it should be able to give you the impression on how to write your own scripts and use the variables in the later process. First we need to edit our whenjobs-script. This can be done by two ways:
+
== The improvement list ==
  
<console>
+
O.K., now let's turn these suggestions into a list of possible improvements:
Version A) (manual way, not recommended)
+
# ##i##EDITOR ~/.whenjobs/jobs.ml
+
Edit the file with the scripts you want to use and save it, but after that you need to upload it so that whenjobs knows about it
+
# ##i##whenjobs --upload
+
Version B) (automatically by whenjobs, recommended)
+
# ##i##whenjobs -e  | --edit
+
just save your script now and whenjobs will upload it automatically for you.
+
</console>
+
  
So far we are now fine with getting to the scripts, next let us add some basics. We will now start with a periodic call, like if we would like to check out load everyage every 10 minutes we would do it like this:
+
* Revamp main page
 +
** Implementation: update logo and add free software blurb
 +
** Goal: to clearly state that we are a free software project
 +
** Target group: potential users
 +
**  Difficulty: medium
 +
* Improve basic user documentation
 +
**  Implementation: new XML/XSLT system, verbose documentation
 +
** Goal: to make it easier for users to install Gentoo Linux
 +
** Target group: new users
 +
** Difficulty: medium
 +
*Improve/create developer documentation
 +
** Implementation: new XML/XSLT system, CVS guide, dev guide, Portage guide
 +
**  Goal: to help our developers to do a great job
 +
** Target group: developers
 +
** Difficulty: hard
 +
*Add a CVS mailing list
 +
** Implementation: use our existing mailman mailing list manager
 +
** Goal: to better inform our developers
 +
** Target group: developers
 +
** Difficulty: easy
  
{{fancywarning|A note aside, the scripts are real shell scripts, so parts beginning with a '''#''' are comments and parts without are the shell script commands that are executed!}}
+
== A selection! ==
  
<source lang="ocaml">
+
Two things leap out from the list, for different reasons. The first is the CVS mailing list -- this one is a no-brainer because it's so easy to implement. Often, it makes sense to implement the easiest changes first so that users can benefit from them right away.
every 10 minutes :
+
<<
+
  # Get the current load average.
+
  load=`awk '{print $1}' /proc/loadavg`
+
  whenjobs --set --type float load=$load
+
>>
+
</source>
+
  
The power of whenjobs comes in game when you would like to base on a variable you set somewhere else:
+
The second big thing that leaps out from the list is the need for developer documentation. This is a longer-term project that will require much more work. From my conversations with the other developers, we all appear to be in agreement that some kind of XML/XSL approach is the right solution.
  
<source lang="ocaml">
+
== The XML/XSL prototype ==
when load >= 6 :
+
<<
+
  mail -s "ALERT: high load average: $load" MAILADDRESS < /dev/null
+
>>
+
</source>
+
  
That part will notify a user via email when his load average is greater or equal to 6, as when statements are "edge-triggered".
+
To help start the process, I developed a prototype XML syntax to be used for all our online documentation. By using this XML syntax (called "guide"), our documentation will be clearly organized into paragraphs, sections, and chapters (using XML tags like <section>, <chapter>, etc.) while remaining free of any display-related tags. To create the HTML for display on our site, I created a prototype set of XSL transforms. By using an XSLT processor such as Sablotron, our guide XML files can be converted into HTML as follows:
  
The '''--type''' switch above for setting a variable can be one of '''bool, int, float, string or unit'''
+
devguide.xml + guide.xsl ---XSLT processor---> devguide.html
  
==== Periodic expressions ====
+
The great thing about this XML/XSLT approach is that it separates our raw content (XML) from the display-related information contained in the guide.xsl (XSLT) file. If we ever need to update the look of our Web pages, we simply modify the guide.xsl file and run all our XML through the XSLT processor (Sablotron), creating updated HTML pages. Or, if we need to add a few chapters to the development guide, we can modify devguide.xml. Once we're done, we then run the XML through Sablotron, which then spits out a fully-formatted devguide.html file with several added chapters. Think of XML as the content and XSLT as the display-related formatting macros.
  
For periodic expressions you have to use the following syntax
+
While our entire team is convinced that XML/XSLT is the way to go, we haven't yet agreed upon an official XML syntax. Achim, our development lead, suggested that we use docbook instead of rolling our own XML syntax. However, the prototype guide XML format has helped to start the decision-making process. Because we developers are going to be the ones using the XML/XSL on a daily basis, it's important to choose a solution that we're comfortable with and meets all of our needs. By my next article, I should have a working XML/XSL doc system to show off to you.
  
<source lang="ocaml">
+
== Technology demo: pytext ==
every <period> :
+
<<
+
  # shell script
+
>>
+
</source>
+
  
where '''<period>''' is one of the following period expressions:
+
For the most part, our current Web site isn't using any new or super-cool technologies that are worth mentioning. However, there's one notable exception -- our tiny pytext embedded Python interpreter.
  
{| class="wikitable"
+
Like many of you, I'm a huge Python fan and much prefer it over other scripting languages, so when it came time to add some dynamic content to our Web site, I naturally wanted to use Python. And, as you probably know, when coding dynamic HTML content, it's usually much more convenient to embed the language commands inside the HTML, rather than the other way around. Thus, the need for an embedded Python interpreter that can take a document like this:
! <period>
+
! Description
+
!
+
! Special <period>
+
! Description
+
|-
+
| second
+
| runs every second
+
|
+
| X seconds
+
| runs every X seconds
+
|-
+
| minute
+
| runs every minute
+
|
+
| X minutes
+
| runs every X minutes
+
|-
+
| hour
+
| runs every hour
+
|
+
| X hours
+
| runs every X hours
+
|-
+
| day
+
| runs every day, at midnight UTC
+
|
+
| X days
+
| runs every X days, at midnight UTC
+
|-
+
| week
+
| runs every week, on a Thursday at midnight UTC
+
|
+
| X weeks
+
| runs every X weeks, on a Thursday at midnight UTC
+
|-
+
| month
+
| runs every month, on the 1st at midnight UTC
+
|
+
| X months
+
| runs every X month, on the 1st at midnight UTC
+
|-
+
| year
+
| runs every year, on the 1/1 at midnight UTC
+
|
+
| X years
+
| runs every X years, on the 1/1 at midnight UTC
+
|-
+
| decade
+
| runs every 10 years
+
|
+
| X decades
+
| runs every X decades
+
|-
+
| century
+
| runs every 100 years
+
|
+
| X centuries
+
| runs every X centuries
+
|-
+
| millenium
+
| runs every 1000 years
+
|
+
| X millenia
+
| runs every X mellenia
+
|}
+
  
==== When expressions ====
+
<pre>
 +
<p>
 +
Yeah, sure; I got some questions:<br>
 +
<!--code
 +
names=["bob","jimmy","ralph"]
 +
items=["socks","lunch","accordion"]
 +
for x in items:
 +
for y in names:
 +
print "Anyone seen",y+"'s",x+"?<br>"
 +
-->
 +
See, told you so.
 +
</pre>
  
For dependent jobs you need to use the when-statements with the following syntax:
+
....and transform it into this:
  
<source lang="ocaml">
+
<pre>
when <expr> :
+
<p>
<<
+
Yeah, sure; I got some questions:<br>
  # shell script
+
Anyone seen bob's socks?<br>
>>
+
Anyone seen jimmy's socks?<br>
</source>
+
Anyone seen ralph's socks?<br>
 +
Anyone seen bob's lunch?<br>
 +
Anyone seen jimmy's lunch?<br>
 +
Anyone seen ralph's lunch?<br>
 +
Anyone seen bob's accordion?<br>
 +
Anyone seen jimmy's accordion?<br>
 +
Anyone seen ralph's accordion?<br>
 +
See, told you so.
 +
</pre>
  
where '''<expr>''' is a when expression. But don't forget the colon between periods expression or when expression and the shell script.
+
Here's the source code for pytext:
  
All in all you can say, that a when-expression is a job which runs, when the described conditions become true.
+
Code Listing 2.4:
 +
{{file|name=pytext|lang=python|desc=The pytext embedded Python interpreter|body=
 +
#!/usr/bin/env python2
  
{| class="wikitable"
+
# pytext 2.1
! <expr>
+
# Copyright 1999-2001 Daniel Robbins
! meaning
+
# Distributed under the GPL
!
+
! <expr>
+
! meaning
+
|-
+
| expr && expr
+
| boolean "and" of the two sub-expressions
+
|
+
| ! expr
+
| boolean negative of expr
+
|-
+
| expr <nowiki>||</nowiki> expr
+
| boolean "or" of the two sub-expressions
+
|
+
| expr + expr
+
| for numeric sub-expression, this performs addition, for strings it performs string concatenation, else it returns an error.
+
|-
+
| expr < expr
+
| evaluates sub-expressions and compares them with the operator
+
|
+
| expr - expr
+
| evaluates sub-expressions and if both are numeric uses operator on them else returns error
+
|-
+
| expr <= expr
+
| evaluates sub-expressions and compares them with the operator
+
|
+
| expr * expr
+
| evaluates sub-expressions and if both are numeric uses operator on them else returns error
+
|-
+
| expr == expr
+
| evaluates sub-expressions and compares them with the operator
+
|
+
| expr / expr
+
| evaluates sub-expressions and if both are numeric uses operator on them else returns error
+
|-
+
| expr >= expr
+
| evaluates sub-expressions and compares them with the operator
+
|
+
| expr mod expr
+
| evaluates sub-expressions and if both are numeric uses operator on them else returns error (infix operator)
+
|-
+
| expr > expr
+
| evaluates sub-expressions and compares them with the operator
+
|
+
| len expr
+
| returns the length of the string in expr
+
|-
+
| variable
+
| returns the value of named variable
+
|
+
| prev variable
+
| returns previous value of named variable
+
|-
+
| changes variable
+
| same as !(prev variabel == variable)
+
|
+
| increases variable
+
| same as prev variable < variable
+
|-
+
| decreases variable
+
| prev variable > variable
+
|
+
| reloaded ()
+
| do not use it, it does not what you want (manpage warning)
+
|-
+
| false
+
| constant equals always false
+
|
+
| true
+
| constant equals always true
+
|-
+
| "any string"
+
| empty string in boolean = false, else equals true
+
|
+
| N
+
| any integer, boolean 0=false, non-zero=true
+
|-
+
| N. | .N | N.N | N.NeN
+
| and floating point number, boolean 0=false, non-zero=true
+
|
+
|
+
|
+
|}
+
  
==== shell scripts ====
+
import sys
  
The code between '''<< ... >>''' is a simple shell script and is executed using $SHELL. If $SHELL is not set, it is executed with '''/bin/sh'''
+
def runfile(myarg):
 +
  "interprets a text file with embedded elements"
 +
  mylocals={}
 +
  try:
 +
      a=open(myarg,'r')
 +
  except IOError:
 +
      sys.stderr.write("!!! Error opening "+myarg+"!\n")
 +
      return
 +
  mylines=a.readlines()
 +
  a.close()
 +
  pos=0
 +
  while pos<len(mylines):
 +
      if mylines[pos][0:8]=="<!--code":
 +
  mycode=""
 +
  pos=pos+1
 +
  while (pos<len(mylines)) and (mylines[pos][0:3]!="-->"):
 +
      mycode=mycode+mylines[pos]
 +
      pos=pos+1
 +
  exec(mycode,globals(),mylocals)
 +
      else:
 +
  sys.stdout.write(mylines[pos])
 +
      pos=pos+1
  
{| class="wikitable"
+
if len(sys.argv)>1:
! available variable
+
  for x in sys.argv[1:]:
! Description
+
      runfile(x)
|-
+
  sys.exit(0)
| $JOBNAME
+
else:
| The name of the job. If the job has been named explicitly, then that name is available through this variable, else it will be some implicit name like '''job$1'''.
+
  sys.stderr.write
|-
+
    ("pytext 2.1 -- Copyright 1999-2001 Daniel Robbins. ")
| $JOBSERIAL
+
  sys.stderr.write
| The serial number of the job. This is simply a variable that increments each time a job is run, and is unique to that run of the job.
+
    ("Distributed under the\nGNU Public License\n\n")
|-
+
  sys.stderr.write
| $HOME, $LOGNAME etc
+
    ("Usage: "+sys.argv[0]+" file0 [file1]...\n")
| these are available as normal
+
  sys.exit(1)
|}
+
}}
  
The shell scripts run with its current directory set to an temporary directory, that is cleaned up automacically after the job exists. So you don't have to worry about cleaning them up later. If you would like to store some values permanently, save the files to a well-known directory, eg. $HOME, '''/var''' etc.
+
== How pytext works ==
  
All shell scripts are executed as the ordinary user. They have no special privileges.
+
Here's how it works. It scans each input line, and most of the time, each input line is simply echoed to stdout. However, if pytext encounters a line beginning with <!--code, then the contents of every line up to the first line beginning with --> are appended to a string called mycode. Pytext then executes the mycode string using the built-in exec() function, effectively creating an embedded Python interpreter.
  
==== Job names ====
+
There's something really beautiful about this particular implementation -- we call exec() in such a way that all modifications to the global and local namespaces are saved. This makes it possible to import a module or define a variable in one embedded block, and then access this previously-created object in a later block, as this example clearly demonstrates:
  
If you like to give a job a unique name use the following syntax:
+
<pre>
 +
<!--code
 +
import os
 +
foo=23
 +
-->
  
<source lang="ocaml">
+
Hello
job "JOBNAME"
+
every <period> :
+
<<
+
  # shell script
+
>>
+
</source>
+
  
==== OCAML expressions ====
+
<!--code
 +
print foo
 +
if os.path.exists("/tmp/mytmpfile"):
 +
print "it exists"
 +
else:
 +
print "I don't see it"
 +
-->
 +
</pre>
  
You can also use OCAML expressions in the code. they are useful for factoring common code or strings, for example:
+
Handy, eh? pytext serves is an excellent demonstration of the power of Python, and is an extremely useful tool for Python fans. For our current site, we call pytext from a cron job, using it to periodically generate the HTML code for our main page Changelog:
 
+
<source lang="ocaml">
+
let prefix = "daily_"
+
 
+
job (prefix ^ "virus_scan")
+
every day :
+
<<
+
  # shell script
+
>>
+
 
+
job (prefix ^ "disk_check")
+
every day :
+
<<
+
  # shell script
+
>>
+
</source>
+
 
+
===== initial value of variables =====
+
 
+
Variables are empty until they first get set, you can set a default starting value for a variable if you like with the following code
+
 
+
<source lang="ocaml">
+
let () =
+
  Whentools.set_variable "variable" "value";
+
  Whentools.set_variable_int "counter" 0
+
</source>
+
 
+
===== Pre functions =====
+
 
+
You can let arrange to run a '''pre''' function before a job runs. This function may decide to not run the job. One possible usage for that is the that you only want to have one job at time from the same job to run:
+
 
+
<source lang="ocaml">
+
job "only one"
+
pre (Whentools.one ())
+
every <period> :
+
<<
+
  # shell script
+
>>
+
</source>
+
 
+
===== Post functions =====
+
 
+
The same is for stuff after a job has run. This is handled by the '''post''' function.
+
 
+
<source lang="ocaml">
+
job "talk to me after finished"
+
post (Whentools.mailto "you@example.com")
+
every <period> :
+
<<
+
  # shell script
+
>>
+
</source>
+
 
+
===== Basic available Whentools functions =====
+
 
+
{| class="wikitable"
+
! function
+
! Description
+
|-
+
| style="vertical-align:top;"| whentools.mailto [~only_on_failure:true] [~from:from_address] email_address result
+
| This built-in post function sends the result of the script by email to the given email address.
+
 
+
If the optional "~only_on_failure:true" flag is set, then it is only sent out if the script failed.
+
 
+
If the optional "~from" flag is set, then the from address is set accordingly.  This is sometimes needed when sending mail.
+
 
+
Note the "result" parameter is passed implicitly by the daemon. You do not need to add it.
+
 
+
Here are some examples of using the mailto function:
+
 
+
<source lang="ocaml">
+
job "ex.1"
+
post (Whentools.mailto "you@example.com")
+
every 10 seconds :
+
<<
+
  # shell script 1
+
>>
+
 
+
job "ex.2"
+
post (Whentools.mailto ~only_on_failure:true "you@example.com")
+
every 10 seconds :
+
<<
+
  # shell script 2
+
>>
+
 
+
let from = "me@example.com"
+
let to_addr = "you@example.com"
+
 
+
job "ex.3"
+
post (Whentools.mailto ~from to_addr)
+
every 10 seconds :
+
<<
+
  # shell script 3
+
>>
+
</source>
+
|-
+
| style="vertical-align:top;"| Whentools.max n
+
| This built-in pre function ensures that a maximum of n instances of the job are running.
+
 
+
It checks the list of running jobs, and if n or more instances are already running, then it returns "false", which ensures that the new job is not started.
+
|-
+
| style="vertical-align:top;"| Whentools.one ()
+
| This built-in pre function ensures that only one instance of the job is running.  It is the same as calling: Whentools.max 1
+
 
+
|-
+
| style="vertical-align:top;"| Whentools.set_variable name string
+
| Set variable name to the string
+
|-
+
| Whentools.set_variable_bool name b
+
| Set variable name to the boolean value b
+
|-
+
| style="vertical-align:top;"| Whentools.set_variable_int name i
+
| Set variable name to the integer value i
+
|-
+
| style="vertical-align:top;"| Whentools.set_variable_string name s
+
| Set variable name to the string value <s>.  This is the same as Whentools.set_variable
+
|-
+
| style="vertical-align:top;"| Whentools.set_variable_float name f
+
| Set variable name to the floating point value f
+
|}
+
 
+
For the preinfo passed to the pre functions and results for the post functions have a view in the manpage.
+
 
+
== Examples ==
+
 
+
Finally here are some examples to which questions came up, hope you find them helpful for your first own tries... :)
+
 
+
<source lang="ocaml">
+
every 1 minute :
+
<<
+
  testtime=`date +%H%M`
+
  whenjobs --set --type int test=${testtime}
+
  whenjobs --set --type int runtime=0016
+
>>
+
 
+
when test == 0017 :
+
<<
+
  echo `date` >\> ~/test.log
+
>>
+
 
+
when test == runtime
+
<<
+
  whenjobs --get runtime >\> ~/test.log
+
>>
+
</source>
+
 
+
The above whenjobs have the need to run each day at a specific time, so we show you here two ways of doing it.
+
 
+
First we define a variable test for whenjobs based on the date with the HHMM output what would for example result in 0017 for 12:17am and 1428 for 2:28pm. then in the first when expression we test if our variable equals 0017 and if yes it runs, the second version is to define a second variable called runtime and then do like the second test does a test for it based on comparing both variables.
+
 
+
But now enough with that long doc, happy whenjobing for all of you... :)
+
  
 +
<console>
 +
$ ##i##pytext index.ehtml > index.html
 +
</console>
  
[[Category:HOWTO]]
+
That's it for now; I'll see you next time when we'll take a look at the first stage of the www.gentoo.org redesign!
[[Category:Labs]]
+
{{ArticleFooter}}
[[Category:Ebuilds]]
+
{{EbuildFooter}}
+

Revision as of 08:21, December 31, 2014

A site reborn

Have you ever woken up one morning and suddenly realized that your cute little personal development Web site isn't really that great? If so, you're in good company. In this series, Daniel Robbins shares his experiences as he redesigns the www.gentoo.org Web site using technologies like XML, XSLT, and Python. Along the way, you may find some excellent approaches to use for your next Web site redesign. In this article, Daniel creates a user-centric action plan and introduces pytext, an embedded Python interpreter.

Support Funtoo and help us grow! Donate $15 per month and get a free SSD-based Funtoo Virtual Container.

An unruly horde

Fellow software developer, may I ask you a question? Why is it that although many of us are intimately familiar with Web technologies such as HTML, CGI, Perl, Python, Java technology, and XML, our very own Web sites -- the ones devoted to our precious development projects -- look like they were thrown together by an unruly horde of hyperactive 12-year-olds? Why, oh why, is this so?

Could it be because most of the time, we've left our Web site out to rot while we squander our precious time hacking away on our free software projects? The answer, at least in my case, is a most definite "Yes."

When I'm not writing articles for IBM developerWorks or being a new dad, I'm feverishly working on the next release of Gentoo Linux, along with my skilled team of volunteers. And, yes, Gentoo Linux has its own Web site (see Resources). As of right now (March 2001), our Web site isn't that special; that's because we don't spend much time working on it because we're generally engrossed in improving Gentoo Linux itself. Sure, our site does have several admittedly cute logos that I whipped up using Xara X (see Resources), but when you look past the eye candy, our site leaves a lot to be desired. Maybe yours does too. If so, I have one thing to say to you -- welcome to the club.

www.gentoo.org

In our case, our Web site dilemma exists because our project has been growing, and our Web site hasn't. Now that Gentoo Linux is approaching the 1.0 release (when it'll be officially ready for non-developers) and is growing in popularity, we need to start seriously looking at how our Web site can better serve its users. Here's a snapshot of www.gentoo.org:

The current (March 2001) state of affairs at www.gentoo.org

As you can see, we have all the bare essentials -- a description of Gentoo Linux, a features list, a daily Changelog (automatically updated thanks to Python), and a bunch of important links (to the download sites, to our mailing list sign-up pages, and to cvsWeb). We also have links to three documentation resources -- the Gentoo Linux Install Guide and Development Guides, and Christian Zander's NVIDIA Troubleshooting Guide.

However, while the site seems O.K., we're missing a lot of things. The most obvious is documentation -- our installation and development guides need a lot of work. And then we need to add an FAQ, new links, new user information...the list is endless.

Content vs. display

And now we come to our second problem. Right now, all of our work is done in raw HTML; I hack away at the index.html file until it looks O.K. Even worse, our Web documentation is written in raw HTML. This isn't a good thing from a development perspective because our raw content (consisting of paragraphs, sections, chapters) is garbled together with a bunch of display-related HTML tags. This, of course, makes it difficult to change both the content and the look of our site. While this approach has worked so far, it is bound to cause problems as our site continues to grow.

Clearly, we need to be using better technologies behind the scenes. Instead of using HTML directly, we need to start using things like XML, XSLT, and Python. The goal is to automate as much as possible so that we can add and expand our site with ease. If we do our job well, even major future changes to our site should be relatively painless.

A strategy!

It was clear that we had a lot of work ahead of us. In fact, there was so much to be done that I didn't know where to begin. Just as I was trying to sort out everything in my head, I came across Laura Wonnacott's "Site Savvy" InfoWorld column (see Resources). In it, she explained the concept of "user-centric" design -- how to improve a Web site while keeping the needs of your target audience (in this case, Gentoo Linux users and developers) in focus. Reading the article and taking a look at the "Handbook of User-Centered Design" link from the article helped me to formulate a strategy -- an action plan -- for the redesign:

  1. First, clearly define the official goal of the Web site -- in writing. What's it there for, and what's it supposed to do?
  2. Identify the different categories of users who will be using your site -- your target audience. Rank them in order of priority: Which ones are most important to you?
  3. Set up a system for getting feedback from your target audience, so they can let you know what you're doing right and wrong.
  4. Evaluate the feedback, and use it to determine what parts of the site need to be improved or redesigned. Tackle high-priority sections first.
  5. Once you've selected the part of the site to improve, get to work! During your implementation, make sure that the content and design of the new section caters specifically to the needs of your target audience and fixes all known deficiencies.
  6. When the section redesign is complete, add it to your live site, even if it has a look that's markedly different from your current site. This way, your users can begin benefitting from the newly redesigned section immediately. If there's a problem with the redesign, you'll get user feedback more quickly. Finally, making incremental improvements to your site (rather than revamping the whole site and then rolling it out all at once -- surprise!) will help prevent your users from feeling alienated by your (possibly dramatic) site changes.
  7. After completing step 6, jump to step 4 and repeat.

The mission statement

I was happy to discover that we already had step 3 in place. We had received several e-mail suggestions from visitors to the site, and our developer mailing list also served as a way of exchanging suggestions and comments. However, I had never really completed steps 1 or 2. While the answers may seem obvious, I did find it helpful to actually sit down and write out our mission statement:

www.gentoo.org exists to assist those who use and develop for Gentoo Linux by providing relevant, up-to-date information about Gentoo Linux and Linux in general, focusing on topics related to Gentoo Linux installation, use, administration, and development. As the central hub for all things Gentoo, the site should also feature important news relevant to Gentoo Linux users and developers. In addition to catering to Gentoo Linux users and developers, www.gentoo.org has the secondary purpose of meeting the needs of potential Gentoo Linux users, providing the information they need to decide whether Gentoo Linux is right for them.

The target audience

So far, so good. Now for step 2 -- defining our target audience:

www.gentoo.org has three target audiences -- Gentoo Linux developers, users, and potential users. While no one group is absolutely a higher priority than another, right now the needs of Gentoo Linux developers are our highest priority, followed by Gentoo Linux users, and then potential users. This is because Gentoo Linux is currently in a prerelease state. When Gentoo Linux reaches version 1.0, Gentoo Linux users and potential users will also become a priority.

Comments and suggestions

O.K., now it's time to evaluate the suggestions and comments we've collected:

Over the past few months, we've received a number of suggestions from Web site visitors. Overwhelmingly, people are requesting better documentation -- for both developers and users. Several developers have asked if we could create a mailing list that would be devoted exclusively to describing CVS commits.

Interestingly, we've also received a couple of e-mails asking whether Gentoo Linux is a commercial or free product. I'm guessing that because our main logo is inscribed with the name "Gentoo Technologies, Inc." (our legal corporation name), people assume that we have a commercial focus. Modifying our logo so that it reads "Gentoo Linux" and adding small opening paragraph to the main page explaining that we are a free software project should help.

The improvement list

O.K., now let's turn these suggestions into a list of possible improvements:

  • Revamp main page
    • Implementation: update logo and add free software blurb
    • Goal: to clearly state that we are a free software project
    • Target group: potential users
    • Difficulty: medium
  • Improve basic user documentation
    • Implementation: new XML/XSLT system, verbose documentation
    • Goal: to make it easier for users to install Gentoo Linux
    • Target group: new users
    • Difficulty: medium
  • Improve/create developer documentation
    • Implementation: new XML/XSLT system, CVS guide, dev guide, Portage guide
    • Goal: to help our developers to do a great job
    • Target group: developers
    • Difficulty: hard
  • Add a CVS mailing list
    • Implementation: use our existing mailman mailing list manager
    • Goal: to better inform our developers
    • Target group: developers
    • Difficulty: easy

A selection!

Two things leap out from the list, for different reasons. The first is the CVS mailing list -- this one is a no-brainer because it's so easy to implement. Often, it makes sense to implement the easiest changes first so that users can benefit from them right away.

The second big thing that leaps out from the list is the need for developer documentation. This is a longer-term project that will require much more work. From my conversations with the other developers, we all appear to be in agreement that some kind of XML/XSL approach is the right solution.

The XML/XSL prototype

To help start the process, I developed a prototype XML syntax to be used for all our online documentation. By using this XML syntax (called "guide"), our documentation will be clearly organized into paragraphs, sections, and chapters (using XML tags like <section>, <chapter>, etc.) while remaining free of any display-related tags. To create the HTML for display on our site, I created a prototype set of XSL transforms. By using an XSLT processor such as Sablotron, our guide XML files can be converted into HTML as follows:

devguide.xml + guide.xsl ---XSLT processor---> devguide.html

The great thing about this XML/XSLT approach is that it separates our raw content (XML) from the display-related information contained in the guide.xsl (XSLT) file. If we ever need to update the look of our Web pages, we simply modify the guide.xsl file and run all our XML through the XSLT processor (Sablotron), creating updated HTML pages. Or, if we need to add a few chapters to the development guide, we can modify devguide.xml. Once we're done, we then run the XML through Sablotron, which then spits out a fully-formatted devguide.html file with several added chapters. Think of XML as the content and XSLT as the display-related formatting macros.

While our entire team is convinced that XML/XSLT is the way to go, we haven't yet agreed upon an official XML syntax. Achim, our development lead, suggested that we use docbook instead of rolling our own XML syntax. However, the prototype guide XML format has helped to start the decision-making process. Because we developers are going to be the ones using the XML/XSL on a daily basis, it's important to choose a solution that we're comfortable with and meets all of our needs. By my next article, I should have a working XML/XSL doc system to show off to you.

Technology demo: pytext

For the most part, our current Web site isn't using any new or super-cool technologies that are worth mentioning. However, there's one notable exception -- our tiny pytext embedded Python interpreter.

Like many of you, I'm a huge Python fan and much prefer it over other scripting languages, so when it came time to add some dynamic content to our Web site, I naturally wanted to use Python. And, as you probably know, when coding dynamic HTML content, it's usually much more convenient to embed the language commands inside the HTML, rather than the other way around. Thus, the need for an embedded Python interpreter that can take a document like this:

<p>
Yeah, sure; I got some questions:<br>
<!--code
names=["bob","jimmy","ralph"]
items=["socks","lunch","accordion"]
for x in items:
for y in names:
print "Anyone seen",y+"'s",x+"?<br>"
-->
See, told you so.

....and transform it into this:

<p>
Yeah, sure; I got some questions:<br>
Anyone seen bob's socks?<br>
Anyone seen jimmy's socks?<br>
Anyone seen ralph's socks?<br>
Anyone seen bob's lunch?<br>
Anyone seen jimmy's lunch?<br>
Anyone seen ralph's lunch?<br>
Anyone seen bob's accordion?<br>
Anyone seen jimmy's accordion?<br>
Anyone seen ralph's accordion?<br>
See, told you so.

Here's the source code for pytext:

Code Listing 2.4:

pytext (python source code) - The pytext embedded Python interpreter
#!/usr/bin/env python2
 
# pytext 2.1
# Copyright 1999-2001 Daniel Robbins
# Distributed under the GPL
 
import sys
 
def runfile(myarg):
   "interprets a text file with embedded elements"
   mylocals={}
   try:
      a=open(myarg,'r')
   except IOError:
      sys.stderr.write("!!! Error opening "+myarg+"!\n")
      return
   mylines=a.readlines()
   a.close()
   pos=0
   while pos<len(mylines):
      if mylines[pos][0:8]==""):
       mycode=mycode+mylines[pos]
       pos=pos+1
  exec(mycode,globals(),mylocals)
       else:
  sys.stdout.write(mylines[pos])
       pos=pos+1
 
if len(sys.argv)>1:
   for x in sys.argv[1:]:
       runfile(x)
   sys.exit(0)
else:
   sys.stderr.write
     ("pytext 2.1 -- Copyright 1999-2001 Daniel Robbins. ")
   sys.stderr.write
     ("Distributed under the\nGNU Public License\n\n")
   sys.stderr.write
     ("Usage: "+sys.argv[0]+" file0 [file1]...\n")
   sys.exit(1)

How pytext works

Here's how it works. It scans each input line, and most of the time, each input line is simply echoed to stdout. However, if pytext encounters a line beginning with are appended to a string called mycode. Pytext then executes the mycode string using the built-in exec() function, effectively creating an embedded Python interpreter.

There's something really beautiful about this particular implementation -- we call exec() in such a way that all modifications to the global and local namespaces are saved. This makes it possible to import a module or define a variable in one embedded block, and then access this previously-created object in a later block, as this example clearly demonstrates:

<!--code
import os
foo=23
-->

Hello

<!--code
print foo
if os.path.exists("/tmp/mytmpfile"):
print "it exists"
else:
print "I don't see it"
-->

Handy, eh? pytext serves is an excellent demonstration of the power of Python, and is an extremely useful tool for Python fans. For our current site, we call pytext from a cron job, using it to periodically generate the HTML code for our main page Changelog:

$ pytext index.ehtml > index.html

That's it for now; I'll see you next time when we'll take a look at the first stage of the www.gentoo.org redesign!


Support Funtoo and help us grow! Donate $15 per month and get a free SSD-based Funtoo Virtual Container.

Have you ever woken up one morning and suddenly realized that your cute little personal development Web site isn't really that great? If so, you're in good company. In this series, Daniel Robbins shares his experiences as he redesigns the www.gentoo.org Web site using technologies like XML, XSLT, and Python. Along the way, you may find some excellent approaches to use for your next Web site redesign. In this article, Daniel creates a user-centric action plan and introduces pytext, an embedded Python interpreter.
About the Author

Daniel Robbins is best known as the creator of Gentoo Linux and author of many IBM developerWorks articles about Linux. Daniel currently serves as Benevolent Dictator for Life (BDFL) of Funtoo Linux. Funtoo Linux is a Gentoo-based distribution and continuation of Daniel's original Gentoo vision.

Got Funtoo?

Have you installed Funtoo Linux yet? Discover the power of a from-source meta-distribution optimized for your hardware! See our installation instructions and browse our CPU-optimized builds.

Funtoo News

Drobbins

IP Space Migration Continues

All Funtoo user containers in the 8.28 IP space will be moving into our new IP space (172.97) over the next few days. If you have DNS set up -- be sure to watch your container and update to the new IP! container.host.funtoo.org DNS will be updated after the move.
2015-08-27 by Drobbins
Drobbins

Funtoo Hosting IP Move

Funtoo user containers with IPs in the 72.18.x.x range will be gradually migrating to new IP addresses this week. If you have DNS entries for your containers, please be aware that your DNS will need to be updated.
2015-08-11 by Drobbins
Drobbins

New ARM Stages

New ARM Stages, built with a new toolchain, are now hitting mirrors. Existing ARM users should re-install using these stages (dated Aug 3, 2015 or later,) rather than upgrade using emerge.
2015-08-06 by Drobbins
More...

More Articles

Browse all our Linux-related articles, below:

A

B

F

G

K

L

M

O

P

S

T

W

X