Difference between pages "Dell PowerEdge 11G Servers" and "The Gentoo.org Redesign, Part 1"

(Difference between pages)
 
(Created page with "{{Article |Subtitle=A site reborn |Summary=Have you ever woken up one morning and suddenly realized that your cute little personal development Web site isn't really that great...")
 
Line 1: Line 1:
__NOTITLE__
+
{{Article
 +
|Subtitle=A site reborn
 +
|Summary=Have you ever woken up one morning and suddenly realized that your cute little personal development Web site isn't really that great? If so, you're in good company. In this series, Daniel Robbins shares his experiences as he redesigns the www.gentoo.org Web site using technologies like XML, XSLT, and Python. Along the way, you may find some excellent approaches to use for your next Web site redesign. In this article, Daniel creates a user-centric action plan and introduces pytext, an embedded Python interpreter.
 +
|Author=Drobbins
 +
}}
 +
==  An unruly horde ==
  
== Funtoo Linux on Dell PowerEdge 11G Servers ==
+
Fellow software developer, may I ask you a question? Why is it that although many of us are intimately familiar with Web technologies such as HTML, CGI, Perl, Python, Java technology, and XML, our very own Web sites -- the ones devoted to our precious development projects -- look like they were thrown together by an unruly horde of hyperactive 12-year-olds? Why, oh why, is this so?
  
This section provides valuable information regarding the use of Funtoo Linux on Dell PowerEdge 11G servers. This information has been validated on Dell PowerEdge R710 systems with Intel Xeon 5500 processors but should also apply to varying degrees to the entire Dell PowerEdge 11G family.
+
Could it be because most of the time, we've left our Web site out to rot while we squander our precious time hacking away on our free software projects? The answer, at least in my case, is a most definite "Yes."
  
'''Please be sure to read all important compatibility notes associated with your specific Dell PowerEdge model, which can be found in the ''Important Compatibility Notes'' column in the table below:'''
+
When I'm not writing articles for IBM developerWorks or being a new dad, I'm feverishly working on the next release of Gentoo Linux, along with my skilled team of volunteers. And, yes, Gentoo Linux has its own Web site (see Resources). As of right now (March 2001), our Web site isn't that special; that's because we don't spend much time working on it because we're generally engrossed in improving Gentoo Linux itself. Sure, our site does have several admittedly cute logos that I whipped up using Xara X (see Resources), but when you look past the eye candy, our site leaves a lot to be desired. Maybe yours does too. If so, I have one thing to say to you -- welcome to the club.
  
{| {{table}}
+
== www.gentoo.org ==
!System
+
!Form Factor
+
!Processor
+
!Integrated NIC
+
!Important Compatibility Notes
+
|-
+
|R210
+
|1U
+
|Intel Xeon 3400 series
+
|BCM 5716
+
|<ref group="hw" name="bcm">'''When using a non-RHEL-5 kernel, it's highly recommended to <tt>emerge broadcom-netxtreme2</tt> after installing your kernel, in order to use the latest drivers from Broadcom.com rather than the in-kernel <tt>bnx2</tt> (1GbE) or <tt>bnx2x</tt> (10GbE) drivers. This will help resolve firmware initialization and other hardware compatibility issues that may result in your network interfaces being unavailable at boot or performing sub-optimally under load. See [[#Kernel Compatibility|Kernel Compatibility]] for more information.'''</ref>
+
|-
+
|R310
+
|1U
+
|Intel Xeon 3400 series
+
|2x1GbE (BCM 5716)
+
|<ref group="hw" name="bcm"/>
+
|-
+
|R410
+
|1U
+
|Intel Xeon 5500 series
+
|2x1GbE (BCM 5716)
+
|<ref group="hw" name="bcm"/><ref group="hw" name="intel5500">'''The Intel Xeon 5500 and 5600 series processors used in Dell PowerEdge 11G servers have known errata (bugs) related to C-states (CPU power saving states) that can and will result in unexpected and unpleasant server behavior in real-world, day-to-day operation. Intel has released CPU microcode updates in 2010 and early 2011 to address these issues. Be sure to update your BIOS to the most recent available from Dell. Dell includes Intel CPU microcode updates as part of their BIOS updates. BIOS 3.0.0 or greater is recommended.'''
+
  
For more information on Intel Xeon 5500 errata, see [http://www.intel.com/assets/pdf/specupdate/321324.pdf Intel's Xeon Processor 5500 Series Specification Update, April 2011].</ref>
+
In our case, our Web site dilemma exists because our project has been growing, and our Web site hasn't. Now that Gentoo Linux is approaching the 1.0 release (when it'll be officially ready for non-developers) and is growing in popularity, we need to start seriously looking at how our Web site can better serve its users. Here's a snapshot of www.gentoo.org:
|-
+
|R415
+
|1U
+
|AMD Opteron 4100 series
+
|2x1GbE (BCM 5716)
+
|<ref group="hw" name="bcm"/>
+
|-
+
|R510
+
|2U
+
|Intel Xeon 5500 series
+
|2x1GbE (BCM 5716)
+
|<ref group="hw" name="bcm"/><ref group="hw" name="intel5500"/>
+
|-
+
|R515
+
|2U
+
|AMD Opteron 4100 series
+
|2x1GbE (BCM 5716)
+
|<ref group="hw" name="bcm"/>
+
|-
+
|R710
+
|2U
+
|Intel Xeon 5500/5600 series
+
|4x1GbE (BCM 5709c)
+
|<ref group="hw" name="bcm"/><ref group="hw" name="intel5500"/>
+
|-
+
|R715
+
|2U
+
|AMD Opteron 6100 series
+
|4x1GbE (BCM 5709c)
+
|<ref group="hw" name="bcm"/>
+
|-
+
|R810
+
|2U
+
|Intel Xeon 6500/7500 series
+
|4x1GbE (BCM 5709c)
+
|<ref group="hw" name="bcm"/>
+
|-
+
|R815
+
|2U
+
|AMD Opteron 6100 series
+
|4x1GbE (BCM 5709c)
+
|<ref group="hw" name="bcm"/>
+
|-
+
|R910
+
|4U
+
|Intel Xeon 7500/E7 series
+
|4x1GbE (BCM 5709c) or 2x10GbE + 2x1GbE (BCM 57771)
+
|<ref group="hw" name="bcm"/>
+
|-
+
|T310
+
|Tower
+
|Intel Xeon 3400 series
+
|optional BCM 5709c (PCI-E)
+
|<ref group="hw" name="bcm"/>
+
|-
+
|T610
+
|Tower
+
|Intel Xeon 5500/5600 series
+
|2x1GbE (BCM 5709c)
+
|<ref group="hw" name="bcm"/><ref group="hw" name="intel5500"/>
+
|-
+
|T710
+
|Tower
+
|Intel Xeon 5500/5600 series
+
|4x1GbE (BCM 5709c)
+
|<ref group="hw" name="bcm"/><ref group="hw" name="intel5500"/>
+
|-
+
|}
+
  
==== Important Hardware Notices ====
+
<div style="margin: 10px;">[[File:L-redesign-01.gif|frame|class=img-responsive|The current (March 2001) state of affairs at www.gentoo.org]]</div>
  
<references group="hw"/>
+
As you can see, we have all the bare essentials -- a description of Gentoo Linux, a features list, a daily Changelog (automatically updated thanks to Python), and a bunch of important links (to the download sites, to our mailing list sign-up pages, and to cvsWeb). We also have links to three documentation resources -- the Gentoo Linux Install Guide and Development Guides, and Christian Zander's NVIDIA Troubleshooting Guide.
  
=== Updating Firmware ===
+
However, while the site seems O.K., we're missing a lot of things. The most obvious is documentation -- our installation and development guides need a lot of work. And then we need to add an FAQ, new links, new user information...the list is endless.
  
The most reliable method to update firmware is to use Dell Repository Manager. This program requires Windows but has the ability to make a bootable Linux ISO image that can update your server firmware in an efficient manner. The PowerEdge R710 also has a built-in "Unified Server Configurator" that can be used to update firmware in a pinch, but it is can take an extremely long time to download and apply server updates.
+
== Content vs. display ==
  
'''Use of the Dell Repository Manager to build a Linux bootable ISO image is the recommended method of updating Dell firmware. A step-by-step guide for using Dell Repository Manager can be found below:'''
+
And now we come to our second problem. Right now, all of our work is done in raw HTML; I hack away at the index.html file until it looks O.K. Even worse, our Web documentation is written in raw HTML. This isn't a good thing from a development perspective because our raw content (consisting of paragraphs, sections, chapters) is garbled together with a bunch of display-related HTML tags. This, of course, makes it difficult to change both the content and the look of our site. While this approach has worked so far, it is bound to cause problems as our site continues to grow.
  
==== Dell Repository Manager ====
+
Clearly, we need to be using better technologies behind the scenes. Instead of using HTML directly, we need to start using things like XML, XSLT, and Python. The goal is to automate as much as possible so that we can add and expand our site with ease. If we do our job well, even major future changes to our site should be relatively painless.
'''Important: It is recommended that you download the latest Dell Repository Manager which can be downloaded following instructions in [http://en.community.dell.com/support-forums/servers/f/177/p/19433362/20058722.aspx#20058722 this thread post]. It resolves issues creating ISOs under Windows 7.'''
+
  
Here's a complete list of steps to update Dell firmware using the Dell Repository Manager:
+
== A strategy! ==
  
# Download Dell Repository Manager from http://ftp.dell.com/FOLDER00313115M/1/Dell_Repository_Manager_1.4.113.msi and install on a Windows system.
+
It was clear that we had a lot of work ahead of us. In fact, there was so much to be done that I didn't know where to begin. Just as I was trying to sort out everything in my head, I came across Laura Wonnacott's "Site Savvy" InfoWorld column (see Resources). In it, she explained the concept of "user-centric" design -- how to improve a Web site while keeping the needs of your target audience (in this case, Gentoo Linux users and developers) in focus. Reading the article and taking a look at the "Handbook of User-Centered Design" link from the article helped me to formulate a strategy -- an action plan -- for the redesign:
# Launch the program.
+
# In the main window, choose to automatically import ftp.dell.com, and click "Import Repository": <br/>[[image:dellrepoman.PNG]]
+
# Welcome: The Create/Load Repository dialog will open. Select "Create New Repository" and click "Next".
+
# Name and Description: Type in a name of your choice, such as "Dell R710", and click "Next".
+
# Select Repository: Select "(Server) ftp.dell.com" and click "Next".
+
# Select Form Factor: Select the type(s) of equipment you want to build a driver disk for and click "Next".
+
# Select OS: Select "Linux". Click "Next".
+
# Select Models: Choose the specific model of equipment ("PowerEdge R710", for example.) Click "Next".
+
# Select Bundle(s): Choose to "ONLY include most recent and custom bundle(s)". Click "Next".
+
# Additional Components: Select "Yes". Click "Next".
+
# Summary: click "Finish".
+
# "Please wait" will appear for a few minutes, and then the dialog will disappear.
+
# The "Bundles" tab will now be active with your bundle visible.
+
# Select your bundle by clicking the square check-box to the left of it.
+
# Click "Export" in the lower right corner of the main window, and click "Next" on the Welcome screen.
+
# Export Destination: Select "Deployment Media (Linux only) Export to ISO/Script format for deployment." Click "Next".
+
# At this point, you may be prompted to install a plugin. Install the plugin and click "Next".
+
# Select an output folder for the ISO, click "OK", and then "Next".
+
# Select Custom Script: Select "No" and click "Next".
+
# Click "Finish".
+
# Your ISO will take 5-10 minutes to build. The program will let you know when it's done.
+
# Burn ISO to CD-R/DVD-R.
+
# Insert burnt disc into server, reboot server, press F11 for boot menu and choose to boot from CD.
+
# The disc will boot. Select the first menu option to launch the firmware update process.
+
# The process will take 20-30 minutes to complete. Press Alt-F2 for a console if you get bored.
+
# When the firmware update process is complete, a message will appear on the screen. Hit Enter to reboot.
+
  
The disc can now be used to update other similar systems in your datacenter.
+
# First, clearly define the official goal of the Web site -- in writing. What's it there for, and what's it supposed to do?
 +
#  Identify the different categories of users who will be using your site -- your target audience. Rank them in order of priority: Which ones are most important to you?
 +
# Set up a system for getting feedback from your target audience, so they can let you know what you're doing right and wrong.
 +
# Evaluate the feedback, and use it to determine what parts of the site need to be improved or redesigned. Tackle high-priority sections first.
 +
# Once you've selected the part of the site to improve, get to work! During your implementation, make sure that the content and design of the new section caters specifically to the needs of your target audience and fixes all known deficiencies.
 +
# When the section redesign is complete, add it to your live site, even if it has a look that's markedly different from your current site. This way, your users can begin benefitting from the newly redesigned section immediately. If there's a problem with the redesign, you'll get user feedback more quickly. Finally, making incremental improvements to your site (rather than revamping the whole site and then rolling it out all at once -- surprise!) will help prevent your users from feeling alienated by your (possibly dramatic) site changes.
 +
#  After completing step 6, jump to step 4 and repeat.
  
==== iDRAC6 ====
+
== The mission statement ==
  
iDRAC is the Dell Integrated Remote Access Controller, which is typically accessed via the dedicated management interface using a Web browswer.
+
I was happy to discover that we already had step 3 in place. We had received several e-mail suggestions from visitors to the site, and our developer mailing list also served as a way of exchanging suggestions and comments. However, I had never really completed steps 1 or 2. While the answers may seem obvious, I did find it helpful to actually sit down and write out our mission statement:
  
Sometimes, the iDRAC firmware will not be successfully updated via the firmware CD-R created using the steps in the section above. This was experienced on our servers when attempting to update from iDRAC firmware 1.54 to 1.70. To work around this issue, you can log in directly to the iDRAC via a Web browser and update the iDRAC's firmware (just the iDRAC firmware itself, not the firmware of other system components) via its user interface.
+
www.gentoo.org exists to assist those who use and develop for Gentoo Linux by providing relevant, up-to-date information about Gentoo Linux and Linux in general, focusing on topics related to Gentoo Linux installation, use, administration, and development. As the central hub for all things Gentoo, the site should also feature important news relevant to Gentoo Linux users and developers. In addition to catering to Gentoo Linux users and developers, www.gentoo.org has the secondary purpose of meeting the needs of potential Gentoo Linux users, providing the information they need to decide whether Gentoo Linux is right for them.
  
To do this, follow these steps:
+
== The target audience ==
  
# Go to http://support.dell.com, and download the latest Dell iDRAC6 firmware. You will want to download the Windows executable format file, ie. <tt>iDRAC6_1.70_A02_FW_IMG.exe</tt>.
+
So far, so good. Now for step 2 -- defining our target audience:
# Execute this file on a Windows machine. It is a self-extracting archive and will prompt you for a location to store the firmware. Choose a location and a file named <tt>firmimg.d6</tt> will be extracted. This is the iDRAC firmware you will upload.
+
# Log in to the iDRAC using a Web browser. If updating from 1.54, use Google Chrome instead of Internet Explorer 9 to avoid issues.
+
# Select "Remote Access" (or in newer versions, "iDRAC Settings") from the menu on the left, and the "Update" tab at the top of the page.
+
# On this page, click the "Choose File" button and select the <tt>firmimg.d6</tt> file extracted earlier.
+
# Click "Upload". Uploading the firmware will take a minute or two.
+
# Once the firmware image is uploaded, you will be able to click a button to update the firmware. You will be presented with a status page similar to this:<br/>[[Image:Idrac6update.PNG]]
+
# Once the iDRAC firmware is 100% complete, you can reload the iDRAC Web page and log in to the new version of iDRAC.
+
# Note that because iDRAC is independent from the underlying hardware, your Linux system will typically not power cycle during this process, so it will still be available.
+
  
=== Kernel Compatibility ===
+
www.gentoo.org has three target audiences -- Gentoo Linux developers, users, and potential users. While no one group is absolutely a higher priority than another, right now the needs of Gentoo Linux developers are our highest priority, followed by Gentoo Linux users, and then potential users. This is because Gentoo Linux is currently in a prerelease state. When Gentoo Linux reaches version 1.0, Gentoo Linux users and potential users will also become a priority.
  
The following table documents kernel compatibility with the Dell PowerEdge R710 server, and is likely to apply to other PowerEdge 11G servers based on Intel Xeon 5500/5600 series processors.
+
== Comments and suggestions ==
  
{{fancynote|Funtoo Linux kernel testing is performed on Dell PowerEdge R710 servers equipped with Intel 5500 series processors.}}
+
O.K., now it's time to evaluate the suggestions and comments we've collected:
  
All kernels listed below were built by setting the <tt>binary</tt> USE variable and emerging, which causes full kernel sources as well as a binary kernel and initrd (built using <tt>genkernel</tt>) to be installed. Note that for the <tt>openvz-rhel5-stable</tt> kernel, udev must be downgraded to 146-r3 in order for the system to function properly after reboot. This can be accomplished by adding <tt>>=sys-fs/udev-147</tt> to <tt>/etc/portage/package.mask</tt> and running <tt>emerge udev</tt> prior to booting your new kernel.
+
Over the past few months, we've received a number of suggestions from Web site visitors. Overwhelmingly, people are requesting better documentation -- for both developers and users. Several developers have asked if we could create a mailing list that would be devoted exclusively to describing CVS commits.
  
{| {{table}}
+
Interestingly, we've also received a couple of e-mails asking whether Gentoo Linux is a commercial or free product. I'm guessing that because our main logo is inscribed with the name "Gentoo Technologies, Inc." (our legal corporation name), people assume that we have a commercial focus. Modifying our logo so that it reads "Gentoo Linux" and adding small opening paragraph to the main page explaining that we are a free software project should help.
!Kernel
+
!Version
+
!Minimum Dell BIOS
+
!Req'd USE flags
+
!Stability
+
!Req'd drivers
+
!Req'd udev
+
|-
+
|openvz-rhel5-stable
+
|2.6.18.028.089.1
+
|3.0.0<ref group="kernel" name="bios">Without an up-to-date BIOS, you may experience system instability or a system clock that jumps forward (and back) 5 minutes for no apparent reason. Upgrading to at least Dell BIOS 3.0.0 should update Intel CPU microcode sufficiently to correct these issues.</ref>
+
|<tt>binary</tt>
+
|'''Excellent'''
+
|Broadcom <tt>bnx2</tt> driver module bundled with kernel appears to be OK
+
|=sys-fs/udev-146*
+
|-
+
|openvz-rhel5-stable
+
|2.6.18.028.091.1
+
|3.0.0<ref group="kernel" name="bios"/>
+
|<tt>binary</tt>
+
|'''Excellent'''
+
|Broadcom <tt>bnx2</tt> driver module bundled with kernel appears to be OK
+
|=sys-fs/udev-146*
+
|-
+
|ubuntu-server
+
|2.6.32.32.62
+
|3.0.0<ref group="kernel" name="bios"/>
+
|<tt>binary</tt>
+
|'''Excellent'''
+
|<tt>emerge broadcom-netxtreme2</tt> for reliable BCM5709+ support (integrated NIC)
+
|N/A <ref group="kernel" name="udevany"/>
+
|-
+
|openvz-rhel6-stable
+
|2.6.32.014.1
+
|3.0.0<ref group="kernel" name="bios"/>
+
|<tt>binary</tt>
+
|''Buggy, do not use''
+
|<tt>emerge broadcom-netxtreme2</tt> for reliable BCM5709+ support (integrated NIC)
+
|N/A <ref group="kernel" name="udevany">Any standard Funtoo Linux udev version is fine.</ref>
+
|-
+
|openvz-rhel6-stable
+
|2.6.32.015.1
+
|3.0.0<ref group="kernel" name="bios"/>
+
|<tt>binary</tt>
+
|''Buggy, do not use''
+
|<tt>emerge broadcom-netxtreme2</tt> for reliable BCM5709+ support (integrated NIC)
+
|N/A <ref group="kernel"  name="udevany"/>
+
|-
+
|openvz-rhel6-stable
+
|2.6.32.016.1
+
|3.0.0<ref group="kernel" name="bios"/>
+
|<tt>binary</tt>
+
|''Buggy, do not use''
+
|<tt>emerge broadcom-netxtreme2</tt> for reliable BCM5709+ support (integrated NIC)
+
|N/A <ref group="kernel" name="udevany"/>
+
|}
+
  
<references group="kernel"/>
+
== The improvement list ==
  
== Server Best Practices ==
+
O.K., now let's turn these suggestions into a list of possible improvements:
  
This section contains a list of recommended programs, utilities and best practices for production servers.
+
* Revamp main page
 +
** Implementation: update logo and add free software blurb
 +
** Goal: to clearly state that we are a free software project
 +
** Target group: potential users
 +
**  Difficulty: medium
 +
* Improve basic user documentation
 +
**  Implementation: new XML/XSLT system, verbose documentation
 +
** Goal: to make it easier for users to install Gentoo Linux
 +
** Target group: new users
 +
** Difficulty: medium
 +
*Improve/create developer documentation
 +
** Implementation: new XML/XSLT system, CVS guide, dev guide, Portage guide
 +
**  Goal: to help our developers to do a great job
 +
** Target group: developers
 +
** Difficulty: hard
 +
*Add a CVS mailing list
 +
** Implementation: use our existing mailman mailing list manager
 +
** Goal: to better inform our developers
 +
** Target group: developers
 +
** Difficulty: easy
  
=== Accurate System Time (NTP) ===
+
== A selection! ==
  
My favorite and recommended NTP client/server is <tt>net-misc/chrony</tt>. It is recommended for production servers:
+
Two things leap out from the list, for different reasons. The first is the CVS mailing list -- this one is a no-brainer because it's so easy to implement. Often, it makes sense to implement the easiest changes first so that users can benefit from them right away.
  
<pre>
+
The second big thing that leaps out from the list is the need for developer documentation. This is a longer-term project that will require much more work. From my conversations with the other developers, we all appear to be in agreement that some kind of XML/XSL approach is the right solution.
# emerge chrony
+
# rc-update add chronyd default
+
</pre>
+
  
Use something like the following for your <tt>/etc/chrony/chrony.conf</tt>:
+
== The XML/XSL prototype ==
 +
 
 +
To help start the process, I developed a prototype XML syntax to be used for all our online documentation. By using this XML syntax (called "guide"), our documentation will be clearly organized into paragraphs, sections, and chapters (using XML tags like <section>, <chapter>, etc.) while remaining free of any display-related tags. To create the HTML for display on our site, I created a prototype set of XSL transforms. By using an XSLT processor such as Sablotron, our guide XML files can be converted into HTML as follows:
 +
 
 +
devguide.xml + guide.xsl ---XSLT processor---> devguide.html
 +
 
 +
The great thing about this XML/XSLT approach is that it separates our raw content (XML) from the display-related information contained in the guide.xsl (XSLT) file. If we ever need to update the look of our Web pages, we simply modify the guide.xsl file and run all our XML through the XSLT processor (Sablotron), creating updated HTML pages. Or, if we need to add a few chapters to the development guide, we can modify devguide.xml. Once we're done, we then run the XML through Sablotron, which then spits out a fully-formatted devguide.html file with several added chapters. Think of XML as the content and XSLT as the display-related formatting macros.
 +
 
 +
While our entire team is convinced that XML/XSLT is the way to go, we haven't yet agreed upon an official XML syntax. Achim, our development lead, suggested that we use docbook instead of rolling our own XML syntax. However, the prototype guide XML format has helped to start the decision-making process. Because we developers are going to be the ones using the XML/XSL on a daily basis, it's important to choose a solution that we're comfortable with and meets all of our needs. By my next article, I should have a working XML/XSL doc system to show off to you.
 +
 
 +
== Technology demo: pytext ==
 +
 
 +
For the most part, our current Web site isn't using any new or super-cool technologies that are worth mentioning. However, there's one notable exception -- our tiny pytext embedded Python interpreter.
 +
 
 +
Like many of you, I'm a huge Python fan and much prefer it over other scripting languages, so when it came time to add some dynamic content to our Web site, I naturally wanted to use Python. And, as you probably know, when coding dynamic HTML content, it's usually much more convenient to embed the language commands inside the HTML, rather than the other way around. Thus, the need for an embedded Python interpreter that can take a document like this:
  
 
<pre>
 
<pre>
server time.apple.com
+
<p>
maxupdateskew 100
+
Yeah, sure; I got some questions:<br>
driftfile /etc/chrony/chrony.drift
+
<!--code
keyfile /etc/chrony/chrony.keys
+
names=["bob","jimmy","ralph"]
commandkey 1
+
items=["socks","lunch","accordion"]
dumponexit
+
for x in items:
dumpdir /var/log/chrony
+
for y in names:
initstepslew 10 time.apple.com
+
print "Anyone seen",y+"'s",x+"?<br>"
logdir /var/log/chrony
+
-->
log measurements statistics tracking
+
See, told you so.
logchange 0.5
+
mailonchange me@emailprovider.com 0.5
+
rtcfile /etc/chrony/chrony.rtc
+
rtconutc
+
sched_priority 1
+
lock_all
+
 
</pre>
 
</pre>
  
Chronyd can then be started immediately by running <tt>rc</tt> to start all new services:
+
....and transform it into this:
  
 
<pre>
 
<pre>
# rc
+
<p>
 +
Yeah, sure; I got some questions:<br>
 +
Anyone seen bob's socks?<br>
 +
Anyone seen jimmy's socks?<br>
 +
Anyone seen ralph's socks?<br>
 +
Anyone seen bob's lunch?<br>
 +
Anyone seen jimmy's lunch?<br>
 +
Anyone seen ralph's lunch?<br>
 +
Anyone seen bob's accordion?<br>
 +
Anyone seen jimmy's accordion?<br>
 +
Anyone seen ralph's accordion?<br>
 +
See, told you so.
 
</pre>
 
</pre>
  
You should notice a marked improvement in your system clock's accuracy. If your system time was off by a significant amount, <tt>chronyd</tt> will gradually correct your clock while the system runs.
+
Here's the source code for pytext:
  
=== SMART Disk Monitoring ===
+
Code Listing 2.4:
 +
{{file|name=pytext|lang=python|desc=The pytext embedded Python interpreter|body=
 +
#!/usr/bin/env python2
  
Emerge <tt>smartmontools</tt> and use an <tt>/etc/smartd.conf</tt> with these settings for a PERC 6/i with 5 physical disks installed:
+
# pytext 2.1
 +
# Copyright 1999-2001 Daniel Robbins
 +
# Distributed under the GPL
  
<pre>
+
import sys
/dev/sda -m myemail@foo.com -d megaraid,0
+
/dev/sda -m myemail@foo.com -d megaraid,1
+
/dev/sda -m myemail@foo.com -d megaraid,2
+
/dev/sda -m myemail@foo.com -d megaraid,3
+
/dev/sda -m myemail@foo.com -d megaraid,4
+
</pre>
+
  
To figure out what configuration to use for your system, use the following command:
+
def runfile(myarg):
 +
  "interprets a text file with embedded elements"
 +
  mylocals={}
 +
  try:
 +
      a=open(myarg,'r')
 +
  except IOError:
 +
      sys.stderr.write("!!! Error opening "+myarg+"!\n")
 +
      return
 +
  mylines=a.readlines()
 +
  a.close()
 +
  pos=0
 +
  while pos<len(mylines):
 +
      if mylines[pos][0:8]=="<!--code":
 +
  mycode=""
 +
  pos=pos+1
 +
  while (pos<len(mylines)) and (mylines[pos][0:3]!="-->"):
 +
      mycode=mycode+mylines[pos]
 +
      pos=pos+1
 +
  exec(mycode,globals(),mylocals)
 +
      else:
 +
  sys.stdout.write(mylines[pos])
 +
      pos=pos+1
 +
 
 +
if len(sys.argv)>1:
 +
  for x in sys.argv[1:]:
 +
      runfile(x)
 +
  sys.exit(0)
 +
else:
 +
  sys.stderr.write
 +
    ("pytext 2.1 -- Copyright 1999-2001 Daniel Robbins. ")
 +
  sys.stderr.write
 +
    ("Distributed under the\nGNU Public License\n\n")
 +
  sys.stderr.write
 +
    ("Usage: "+sys.argv[0]+" file0 [file1]...\n")
 +
  sys.exit(1)
 +
}}
 +
 
 +
== How pytext works ==
 +
 
 +
Here's how it works. It scans each input line, and most of the time, each input line is simply echoed to stdout. However, if pytext encounters a line beginning with <!--code, then the contents of every line up to the first line beginning with --> are appended to a string called mycode. Pytext then executes the mycode string using the built-in exec() function, effectively creating an embedded Python interpreter.
 +
 
 +
There's something really beautiful about this particular implementation -- we call exec() in such a way that all modifications to the global and local namespaces are saved. This makes it possible to import a module or define a variable in one embedded block, and then access this previously-created object in a later block, as this example clearly demonstrates:
  
 
<pre>
 
<pre>
# smartctl -d megaraid,X --all /dev/sdY
+
<!--code
 +
import os
 +
foo=23
 +
-->
 +
 
 +
Hello
 +
 
 +
<!--code
 +
print foo
 +
if os.path.exists("/tmp/mytmpfile"):
 +
print "it exists"
 +
else:
 +
print "I don't see it"
 +
-->
 
</pre>
 
</pre>
  
Start using X=0 and Y=a, and keep incrementing X until you discover all physical disks backing sda. Then repeat with sdb, etc. until you discover all physical disks in your system. In typical configurations, you will have <tt>megaraid,0</tt> thru <tt>megaraid,(num_physical_disks-1)</tt>.
+
Handy, eh? pytext serves is an excellent demonstration of the power of Python, and is an extremely useful tool for Python fans. For our current site, we call pytext from a cron job, using it to periodically generate the HTML code for our main page Changelog:
  
{{fancynote|On Dell R910s with PERC H700, all physical disks are addressable through all /dev/sdY simultaneously, so there is no need to iterate through Y. ie: all of the same physical disks will be shown simultaneously for /dev/sda, /dev/sdb, /dev/sdc etc... so there is no need to repeat the steps for sdY+1 and onwards.}}
+
<console>
 +
$ ##i##pytext index.ehtml > index.html
 +
</console>
  
[[Category:Hardware Compatibility]]
+
That's it for now; I'll see you next time when we'll take a look at the first stage of the www.gentoo.org redesign!
 +
{{ArticleFooter}}

Revision as of 08:21, December 31, 2014

A site reborn

Have you ever woken up one morning and suddenly realized that your cute little personal development Web site isn't really that great? If so, you're in good company. In this series, Daniel Robbins shares his experiences as he redesigns the www.gentoo.org Web site using technologies like XML, XSLT, and Python. Along the way, you may find some excellent approaches to use for your next Web site redesign. In this article, Daniel creates a user-centric action plan and introduces pytext, an embedded Python interpreter.

Support Funtoo and help us grow! Donate $15 per month and get a free SSD-based Funtoo Virtual Container.

An unruly horde

Fellow software developer, may I ask you a question? Why is it that although many of us are intimately familiar with Web technologies such as HTML, CGI, Perl, Python, Java technology, and XML, our very own Web sites -- the ones devoted to our precious development projects -- look like they were thrown together by an unruly horde of hyperactive 12-year-olds? Why, oh why, is this so?

Could it be because most of the time, we've left our Web site out to rot while we squander our precious time hacking away on our free software projects? The answer, at least in my case, is a most definite "Yes."

When I'm not writing articles for IBM developerWorks or being a new dad, I'm feverishly working on the next release of Gentoo Linux, along with my skilled team of volunteers. And, yes, Gentoo Linux has its own Web site (see Resources). As of right now (March 2001), our Web site isn't that special; that's because we don't spend much time working on it because we're generally engrossed in improving Gentoo Linux itself. Sure, our site does have several admittedly cute logos that I whipped up using Xara X (see Resources), but when you look past the eye candy, our site leaves a lot to be desired. Maybe yours does too. If so, I have one thing to say to you -- welcome to the club.

www.gentoo.org

In our case, our Web site dilemma exists because our project has been growing, and our Web site hasn't. Now that Gentoo Linux is approaching the 1.0 release (when it'll be officially ready for non-developers) and is growing in popularity, we need to start seriously looking at how our Web site can better serve its users. Here's a snapshot of www.gentoo.org:

The current (March 2001) state of affairs at www.gentoo.org

As you can see, we have all the bare essentials -- a description of Gentoo Linux, a features list, a daily Changelog (automatically updated thanks to Python), and a bunch of important links (to the download sites, to our mailing list sign-up pages, and to cvsWeb). We also have links to three documentation resources -- the Gentoo Linux Install Guide and Development Guides, and Christian Zander's NVIDIA Troubleshooting Guide.

However, while the site seems O.K., we're missing a lot of things. The most obvious is documentation -- our installation and development guides need a lot of work. And then we need to add an FAQ, new links, new user information...the list is endless.

Content vs. display

And now we come to our second problem. Right now, all of our work is done in raw HTML; I hack away at the index.html file until it looks O.K. Even worse, our Web documentation is written in raw HTML. This isn't a good thing from a development perspective because our raw content (consisting of paragraphs, sections, chapters) is garbled together with a bunch of display-related HTML tags. This, of course, makes it difficult to change both the content and the look of our site. While this approach has worked so far, it is bound to cause problems as our site continues to grow.

Clearly, we need to be using better technologies behind the scenes. Instead of using HTML directly, we need to start using things like XML, XSLT, and Python. The goal is to automate as much as possible so that we can add and expand our site with ease. If we do our job well, even major future changes to our site should be relatively painless.

A strategy!

It was clear that we had a lot of work ahead of us. In fact, there was so much to be done that I didn't know where to begin. Just as I was trying to sort out everything in my head, I came across Laura Wonnacott's "Site Savvy" InfoWorld column (see Resources). In it, she explained the concept of "user-centric" design -- how to improve a Web site while keeping the needs of your target audience (in this case, Gentoo Linux users and developers) in focus. Reading the article and taking a look at the "Handbook of User-Centered Design" link from the article helped me to formulate a strategy -- an action plan -- for the redesign:

  1. First, clearly define the official goal of the Web site -- in writing. What's it there for, and what's it supposed to do?
  2. Identify the different categories of users who will be using your site -- your target audience. Rank them in order of priority: Which ones are most important to you?
  3. Set up a system for getting feedback from your target audience, so they can let you know what you're doing right and wrong.
  4. Evaluate the feedback, and use it to determine what parts of the site need to be improved or redesigned. Tackle high-priority sections first.
  5. Once you've selected the part of the site to improve, get to work! During your implementation, make sure that the content and design of the new section caters specifically to the needs of your target audience and fixes all known deficiencies.
  6. When the section redesign is complete, add it to your live site, even if it has a look that's markedly different from your current site. This way, your users can begin benefitting from the newly redesigned section immediately. If there's a problem with the redesign, you'll get user feedback more quickly. Finally, making incremental improvements to your site (rather than revamping the whole site and then rolling it out all at once -- surprise!) will help prevent your users from feeling alienated by your (possibly dramatic) site changes.
  7. After completing step 6, jump to step 4 and repeat.

The mission statement

I was happy to discover that we already had step 3 in place. We had received several e-mail suggestions from visitors to the site, and our developer mailing list also served as a way of exchanging suggestions and comments. However, I had never really completed steps 1 or 2. While the answers may seem obvious, I did find it helpful to actually sit down and write out our mission statement:

www.gentoo.org exists to assist those who use and develop for Gentoo Linux by providing relevant, up-to-date information about Gentoo Linux and Linux in general, focusing on topics related to Gentoo Linux installation, use, administration, and development. As the central hub for all things Gentoo, the site should also feature important news relevant to Gentoo Linux users and developers. In addition to catering to Gentoo Linux users and developers, www.gentoo.org has the secondary purpose of meeting the needs of potential Gentoo Linux users, providing the information they need to decide whether Gentoo Linux is right for them.

The target audience

So far, so good. Now for step 2 -- defining our target audience:

www.gentoo.org has three target audiences -- Gentoo Linux developers, users, and potential users. While no one group is absolutely a higher priority than another, right now the needs of Gentoo Linux developers are our highest priority, followed by Gentoo Linux users, and then potential users. This is because Gentoo Linux is currently in a prerelease state. When Gentoo Linux reaches version 1.0, Gentoo Linux users and potential users will also become a priority.

Comments and suggestions

O.K., now it's time to evaluate the suggestions and comments we've collected:

Over the past few months, we've received a number of suggestions from Web site visitors. Overwhelmingly, people are requesting better documentation -- for both developers and users. Several developers have asked if we could create a mailing list that would be devoted exclusively to describing CVS commits.

Interestingly, we've also received a couple of e-mails asking whether Gentoo Linux is a commercial or free product. I'm guessing that because our main logo is inscribed with the name "Gentoo Technologies, Inc." (our legal corporation name), people assume that we have a commercial focus. Modifying our logo so that it reads "Gentoo Linux" and adding small opening paragraph to the main page explaining that we are a free software project should help.

The improvement list

O.K., now let's turn these suggestions into a list of possible improvements:

  • Revamp main page
    • Implementation: update logo and add free software blurb
    • Goal: to clearly state that we are a free software project
    • Target group: potential users
    • Difficulty: medium
  • Improve basic user documentation
    • Implementation: new XML/XSLT system, verbose documentation
    • Goal: to make it easier for users to install Gentoo Linux
    • Target group: new users
    • Difficulty: medium
  • Improve/create developer documentation
    • Implementation: new XML/XSLT system, CVS guide, dev guide, Portage guide
    • Goal: to help our developers to do a great job
    • Target group: developers
    • Difficulty: hard
  • Add a CVS mailing list
    • Implementation: use our existing mailman mailing list manager
    • Goal: to better inform our developers
    • Target group: developers
    • Difficulty: easy

A selection!

Two things leap out from the list, for different reasons. The first is the CVS mailing list -- this one is a no-brainer because it's so easy to implement. Often, it makes sense to implement the easiest changes first so that users can benefit from them right away.

The second big thing that leaps out from the list is the need for developer documentation. This is a longer-term project that will require much more work. From my conversations with the other developers, we all appear to be in agreement that some kind of XML/XSL approach is the right solution.

The XML/XSL prototype

To help start the process, I developed a prototype XML syntax to be used for all our online documentation. By using this XML syntax (called "guide"), our documentation will be clearly organized into paragraphs, sections, and chapters (using XML tags like <section>, <chapter>, etc.) while remaining free of any display-related tags. To create the HTML for display on our site, I created a prototype set of XSL transforms. By using an XSLT processor such as Sablotron, our guide XML files can be converted into HTML as follows:

devguide.xml + guide.xsl ---XSLT processor---> devguide.html

The great thing about this XML/XSLT approach is that it separates our raw content (XML) from the display-related information contained in the guide.xsl (XSLT) file. If we ever need to update the look of our Web pages, we simply modify the guide.xsl file and run all our XML through the XSLT processor (Sablotron), creating updated HTML pages. Or, if we need to add a few chapters to the development guide, we can modify devguide.xml. Once we're done, we then run the XML through Sablotron, which then spits out a fully-formatted devguide.html file with several added chapters. Think of XML as the content and XSLT as the display-related formatting macros.

While our entire team is convinced that XML/XSLT is the way to go, we haven't yet agreed upon an official XML syntax. Achim, our development lead, suggested that we use docbook instead of rolling our own XML syntax. However, the prototype guide XML format has helped to start the decision-making process. Because we developers are going to be the ones using the XML/XSL on a daily basis, it's important to choose a solution that we're comfortable with and meets all of our needs. By my next article, I should have a working XML/XSL doc system to show off to you.

Technology demo: pytext

For the most part, our current Web site isn't using any new or super-cool technologies that are worth mentioning. However, there's one notable exception -- our tiny pytext embedded Python interpreter.

Like many of you, I'm a huge Python fan and much prefer it over other scripting languages, so when it came time to add some dynamic content to our Web site, I naturally wanted to use Python. And, as you probably know, when coding dynamic HTML content, it's usually much more convenient to embed the language commands inside the HTML, rather than the other way around. Thus, the need for an embedded Python interpreter that can take a document like this:

<p>
Yeah, sure; I got some questions:<br>
<!--code
names=["bob","jimmy","ralph"]
items=["socks","lunch","accordion"]
for x in items:
for y in names:
print "Anyone seen",y+"'s",x+"?<br>"
-->
See, told you so.

....and transform it into this:

<p>
Yeah, sure; I got some questions:<br>
Anyone seen bob's socks?<br>
Anyone seen jimmy's socks?<br>
Anyone seen ralph's socks?<br>
Anyone seen bob's lunch?<br>
Anyone seen jimmy's lunch?<br>
Anyone seen ralph's lunch?<br>
Anyone seen bob's accordion?<br>
Anyone seen jimmy's accordion?<br>
Anyone seen ralph's accordion?<br>
See, told you so.

Here's the source code for pytext:

Code Listing 2.4:

pytext (python source code) - The pytext embedded Python interpreter
#!/usr/bin/env python2
 
# pytext 2.1
# Copyright 1999-2001 Daniel Robbins
# Distributed under the GPL
 
import sys
 
def runfile(myarg):
   "interprets a text file with embedded elements"
   mylocals={}
   try:
      a=open(myarg,'r')
   except IOError:
      sys.stderr.write("!!! Error opening "+myarg+"!\n")
      return
   mylines=a.readlines()
   a.close()
   pos=0
   while pos<len(mylines):
      if mylines[pos][0:8]==""):
       mycode=mycode+mylines[pos]
       pos=pos+1
  exec(mycode,globals(),mylocals)
       else:
  sys.stdout.write(mylines[pos])
       pos=pos+1
 
if len(sys.argv)>1:
   for x in sys.argv[1:]:
       runfile(x)
   sys.exit(0)
else:
   sys.stderr.write
     ("pytext 2.1 -- Copyright 1999-2001 Daniel Robbins. ")
   sys.stderr.write
     ("Distributed under the\nGNU Public License\n\n")
   sys.stderr.write
     ("Usage: "+sys.argv[0]+" file0 [file1]...\n")
   sys.exit(1)

How pytext works

Here's how it works. It scans each input line, and most of the time, each input line is simply echoed to stdout. However, if pytext encounters a line beginning with are appended to a string called mycode. Pytext then executes the mycode string using the built-in exec() function, effectively creating an embedded Python interpreter.

There's something really beautiful about this particular implementation -- we call exec() in such a way that all modifications to the global and local namespaces are saved. This makes it possible to import a module or define a variable in one embedded block, and then access this previously-created object in a later block, as this example clearly demonstrates:

<!--code
import os
foo=23
-->

Hello

<!--code
print foo
if os.path.exists("/tmp/mytmpfile"):
print "it exists"
else:
print "I don't see it"
-->

Handy, eh? pytext serves is an excellent demonstration of the power of Python, and is an extremely useful tool for Python fans. For our current site, we call pytext from a cron job, using it to periodically generate the HTML code for our main page Changelog:

$ pytext index.ehtml > index.html

That's it for now; I'll see you next time when we'll take a look at the first stage of the www.gentoo.org redesign!


Support Funtoo and help us grow! Donate $15 per month and get a free SSD-based Funtoo Virtual Container.

Have you ever woken up one morning and suddenly realized that your cute little personal development Web site isn't really that great? If so, you're in good company. In this series, Daniel Robbins shares his experiences as he redesigns the www.gentoo.org Web site using technologies like XML, XSLT, and Python. Along the way, you may find some excellent approaches to use for your next Web site redesign. In this article, Daniel creates a user-centric action plan and introduces pytext, an embedded Python interpreter.
About the Author

Daniel Robbins is best known as the creator of Gentoo Linux and author of many IBM developerWorks articles about Linux. Daniel currently serves as Benevolent Dictator for Life (BDFL) of Funtoo Linux. Funtoo Linux is a Gentoo-based distribution and continuation of Daniel's original Gentoo vision.

Got Funtoo?

Have you installed Funtoo Linux yet? Discover the power of a from-source meta-distribution optimized for your hardware! See our installation instructions and browse our CPU-optimized builds.

Funtoo News

Drobbins

Perl Updates

Gentoo has bumped perl from 5.20 to 5.22. Be sure to run perl-cleaner --all after the upgrade.
2015-07-25 by Drobbins
Drobbins

ARM Rebuild

ARM systems will use new stage3's that are not compatible with earlier versions.
2015-06-27 by Drobbins
Drobbins

ABI X86 64 and 32

Funtoo Linux has new 32-bit compatibility libraries inherited from Gentoo. Learn about them here.
2015-06-18 by Drobbins
More...

More Articles

Browse all our Linux-related articles, below:

A

B

F

G

K

L

M

O

P

S

T

W

X