Difference between pages "Making the Distribution, Part 2" and "ConsoleOutput MediaWiki Extension"

From Funtoo
(Difference between pages)
Jump to navigation Jump to search
("and then some" is correct (slang))
 
 
Line 1: Line 1:
{{Article
The ConsoleOutput MediaWiki extension was created by Daniel Robbins to provide highlighting of user input for interactive terminal session blocks. To use it, surround user input with  <tt>&lt;console&gt;</tt> opening and closing tags, and put a <tt>##i##</tt> input code immediately before user input on each line. This will cause all text from the <tt>##i##</tt> point to the end of the line to be in a darker color. The rest of the text will be in a lighter grey, allowing it to be distinguished more easily from user input. Here are other sequences that can be used:
|Summary=In his previous article, Daniel Robbins told the story of how he became a Stampede Linux developer and why he eventually left Stampede to start the Enoch Linux distribution. In this go-round he lets you in on the strange events that happened after the Enoch development team discovered a little-known, blazingly fast compiler.
|Article Category=General
|Author=Drobbins
|Previous in Series=Making the Distribution, Part 1
|Next in Series=Making the Distribution, Part 3
}}
==  From Enoch to Gentoo, via minor setbacks and corporate run-ins ==


=== First steps to Enoch ===
* <tt>##b##</tt> - highlight the rest of the line in bold.
* <tt>##b## text here ##!b##</tt> - highlight the text between both markers in bold.
* <tt>##i## text here ##!i##</tt> - highlight the text between both markers in a darker color.


In my previous article, I gave you the low-down on my days with the Stampede development team and why I left (to get away from lower-level politically-minded, project-controlling "freaks"). Because of the interference from these meddlesome by-standers, I figured it would be easier to put together my own Linux distribution than to continue improving Stampede under such dirty conditions! Fortunately I took with me a considerable amount of experience based on my (may I say substantial?) work for Stampede, including maintaining several of their packages, designing the initialization scripts, and leading the slpv6 (next-generation package management project).
This extension can be further extended to add color support, as desired, and some of the CSS for color support can already be found in the code below.


The distribution I began working on, code-named Enoch, was going to be blazingly fast because it would completely automate the package creation and upgrading process. I have to admit that this was in large part because I was a one-member team and couldn't afford to spend my time on repetitive work that my development box could be automated to do for me. And since I was designing a complete distribution from scratch (rather than "spinning off" from someone like RedHat), I had my work cut out for me and needed all the free time I could scrounge up.
Here are a few examples of the ConsoleOutput extension. First this is how you might typically display {{c|ls}} output, with a particular directory highlighted:


After getting my basic Enoch system up and running, I headed back to irc.openprojects.net and started my own channel called #enoch. From there I gradually assembled a team of about ten developers. In those early days we all hung out on IRC and worked on the distribution in our spare time. As we communally and cooperatively hacked away at it, finding and fixing new bugs, Enoch became more functional and professional every day.
<console>
www@www-smw ~/public_html $ ##i##ls
COPYING  LocalSettings.php    api.php  ##b##extensions##!b##   index.php  maintenance          redirect.php    skins              thumb_handler.php5
CREDITS  README                api.php5  images        index.php5  mw-config            redirect.php5  tests              wiki.phtml
FAQ      RELEASE-NOTES-1.19    bin      img_auth.php  languages  opensearch_desc.php  redirect.phtml  thumb.php
HISTORY  StartProfiler.sample  cache    img_auth.php5  load.php    opensearch_desc.php5  resources      thumb.php5
INSTALL  UPGRADE              docs      includes      load.php5  profileinfo.php      serialized      thumb_handler.php
www@www-smw ~/public_html $ ##i##cd extensions/
</console>


=== The first roadblock ===
And here is how you might display a more detailed example of console output, using colors:


One inevitable day, Enoch hit its first roadblock. After adding Xfree86, glib, and gtk+, I decided to get xmms (an X11/gtk+-based MP3/CD player app) working. I figured it was time to celebrate with some music! But after installing xmms, I tried to start it... and X locked up! At first I thought xmms locked up because I used insane compiler optimizations ("-O6 -mpentiumpro", in case you were wondering). My first thought, to compile xmms with standard optimizations, didn't solve the problem. So I started looking elsewhere. After spending a full week of development time trying to track down the problem, I got an e-mail from an Enoch user, Omegadan, who was also experiencing xmms lockups.
{{console|body=
 
# ##i##bluetoothctl
We corresponded for a while, and after many hours of testing we determined that the problem was a POSIX threads-related issue. For some reason, a pthread_mutex_trylock() call did not return the way it should. As the creator of a distribution, these were the types of bugs I really didn't want to encounter. I counted on the developers to release perfect sources so I could focus on enhancing the Linux experience rather than getting buggy sources to work. Of course I soon learned that this was an unrealistic expectation, and that problems like this will always pop up from time to time.
[##g##NEW##!g##] Controller 00:02:72:C9:62:65 antec [default]
 
##bl##[bluetooth]##!bl### ##i##power on
As it turned out, the problem wasn't with xmms, gtk+, or glib. And it wasn't an issue with Xfree86 3.3.5 not being thread-safe and locking up. Surprisingly, we found the bug in the Linux POSIX threads implementation itself, part of the GNU C library (glibc) version 2.1.2. I was shocked at the time to find that such a critical part of Linux had such a major bug. (And we used a release version of glibc in Enoch, not a prerelease or CVS version!).
Changing power on succeeded
 
##bl##[bluetooth]##!bl### ##i##agent on
So how did we track down the problem? Actually, we never were able to come up with a bug fix, but at one point I stumbled across a couple of e-mails on the glibc developer mailing list from another person who had the same problem. The glibc developer who replied posted a patch that solved the thread problem for us. But I was curious why RedHat 6 (which also used glibc 2.1.2) didn't suffer from this problem since the patch was just posted and RedHat 6 had been available for some time. To find out, I downloaded RedHat's glibc SRPM (source RPM) and took a look at their patches.
Agent registered
 
##bl##[bluetooth]##!bl### ##i##scan on
RedHat had their own homegrown glibc patch that solved the pthread_mutex_trylock() issue. Apparently they experienced the same problem and created their own custom fix. Too bad they didn't send this patch "upstream" to the glibc developers so it could be shared with the rest of the world. But who knows, maybe RedHat sent the patch upstream and for some reason the glibc developers didn't accept it. Or maybe the thread bug was triggered by a specific combination of compiler and binutils versions, and RedHat never ran into it (although they did have a thread patch in their SRPM). I suppose we'll never know exactly what happened. But I did learn that RedHat SRPMs contain a lot of private bug fixes and tweaks that never seem to make it upstream to the original developers. I'm going to rant about this for a little while.
Discovery started
 
##bl##[bluetooth]##!bl### ##i##devices
=== Rant ===
Device 00:1F:20:3D:1E:75 Logitech K760
 
##bl##[bluetooth]##!bl### ##i##pair 00:1F:20:3D:1E:75
When you put together a Linux distribution it's really important that any bug fixes you create are sent upstream to the original developers. As I see it, this is one of the many ways that distribution creators contribute to Linux. We're the guys who actually get all these different programs working as a unified whole. We should send our fixes upstream as we unify so that other users and distributions can benefit from our discoveries. If you decide to keep bug fixes to yourself, you're not helping anyone; you're just ensuring that a lot of people will waste time fixing the same problem over and over again. This kind of policy goes against the whole open source ethic and stunts the growth of Linux development. Maybe I should say that it "bugs" us all.
Attempting to pair with 00:1F:20:3D:1E:75
 
[##y##CHG##!y##] Device 00:1F:20:3D:1E:75 Connected: yes
It's unfortunate that some distributions (ahem) aren't as good (RedHat) as others (Debian) about sharing their work with the community.
##r##[agent]##!r## Passkey: 454358
 
##r##[agent]##!r## Passkey: ##i##4##!i##54358
=== Compiler drama ===
##r##[agent]##!r## Passkey: ##i##45##!i##4358
 
##r##[agent]##!r## Passkey: ##i##454##!i##358
During the time we were trying to fix the glibc threads problem, I e-mailed Ulrich Drepper (one of the guys at Cygnus who is heavily involved with glibc development). I mentioned the POSIX thread problem we were having, and that Enoch was using pgcc for optimum performance. And he responded with something like this (I'm paraphrasing here): "Our own compiler included with the CodeFusion product has an excellent x86 backend that produces executables far faster than those generated with pgcc." Obviously, I was very interested in testing out this mystery "turbo" compiler the Cygnus guys had created.
##r##[agent]##!r## Passkey: ##i##4543##!i##58
 
##r##[agent]##!r## Passkey: ##i##45435##!i##8
I thereupon requested a demo copy of Cygnus Codefusion 1.0 so that I could test it out, and Omegadan and I were amazed to find that this compiler was everything that Ulrich claimed and then some. The x86 backend increased the performance of some of the CPU-intensive executables (like bzip2) by close to 90%! All applications seemed to benefit from at least a 10% real-world performance increase, and all we did was swap out compilers. Enoch even booted 30 - 40% faster. The performance gains were far, far greater than what we gained by switching from gcc to pgcc. Obviously, after experiencing it for ourselves, we wanted to use this compiler for Enoch. Fortunately, the sources were included on the CodeFusion CD and were released under the GPL, so we were fully permitted to use this compiler... or so we thought.
##r##[agent]##!r## Passkey: ##i##454358##!i##
 
[##y##CHG##!y##] Device 00:1F:20:3D:1E:75 Paired: yes
=== Let the freakiness begin ===
Pairing successful
 
[##y##CHG##!y##] Device 00:1F:20:3D:1E:75 Connected: no
I sent an e-mail to the marketing manager at Cygnus to let them know our intentions, expecting a "yeah, go for it, thanks for using our compiler" response. Instead the reply was that although we were (technically) allowed to use the Cygnus compiler, we were strongly urged not to use or include the compiler sources with Enoch. I responded by asking why they had released the source under the GPL, if that was the case. It's my guess that if they had a choice, they wouldn't have used the GPL, but because they derived their compiler from egcs (released under the GPL), they had no choice.
##bl##[bluetooth]##!bl### ##i##connect 00:1F:20:3D:1E:75
 
Attempting to connect to 00:1F:20:3D:1E:75
This is a good example of a situation where the GPL prevented a company from creating a proprietary product based on open sources. My educated guess is that Cygnus was afraid that if we used their compiler we would undermine their boxed product sales, which would be especially strange because none of their marketing materials (nor the InfoWorld review) mentioned the new compiler included with CodeFusion. CodeFusion was marketed solely as a "development IDE" product, not as a compiler.
[##y##CHG##!y##] Device 00:1F:20:3D:1E:75 Connected: yes
Connection successful
##bl##[bluetooth]##!bl### ##i##quit
[##r##DEL##!r##] Controller 00:02:72:C9:62:65 antec [default]
#
}}


In an attempt to put some of their paranoia to rest, I offered to endorse CodeFusion and place the endorsement on our Web site with a link to help spur CodeFusion sales. Personally I didn't think that a "turbo" Enoch would negatively affect their sales, since CodeFusion was marketed as an IDE. But I tried nevertheless to make them happy. The IDE component of CodeFusion was a commercial product, and we had no desire or intention (or right) to distribute it with Enoch.
To install, make the following modifications to your skin:


I e-mailed my (generous?) offer to Cygnus and received another strange response. They wanted authority over all of our "marketing materials" (apparently, this also included the content of our Web site!) Another shocker. The Cygnus marketing team seemed to have no grasp of how the Linux community or the GPL worked, so I decided to cut off communication with Cygnus for the indefinite future. In the mean time, we created a private "turbo" and public "non-turbo" version of Enoch, leaving the final decision for later.
<syntaxhighlight lang="css">
--- mediawiki-1.19.1/skins/vector/screen.css    2012-06-13 18:22:39.000000000 +0000
+++ public_html/skins/vector/screen.css 2012-08-27 04:34:47.507912892 +0000
@@ -683,10 +683,47 @@
        list-style-image: url(images/bullet-icon.png);
}
-pre {
-      line-height: 1.3em;
+/* ConsoleOutput.php start */
+
+.shell, pre, code, tt, div.mw-geshi {
+        font-size: 12px;
+        font-family: Consolas, 'andale mono','lucida console', monospace;
+}
+
+.shell, pre, div.mw-geshi {
+        background-color: #F8F8FF;
+        line-height: 15px;
+        padding: 10px;
+        border: none;
+        border-top: 2px solid #C6C9E0;
+        border-bottom: 2px solid #C6C9E0;
+        margin: 0;
+        overflow-x: auto;
+        overflow-y: hidden;
+}
+
+.code {
+        color: #666;
+}
+
+.code_input {
+        color: #000;
}
+.code_red {
+        color: #f00;
+}
+
+.code_blue {
+        color: #00f;
+}
+
+.shell_green {
+        color: #080;
+}
+
+/* ConsoleOutput.php end */
+
/* Site Notice (includes notices from CentralNotice extension) */
#siteNotice {
        font-size: 0.8em;
</syntaxhighlight>


But after several months they integrated the CodeFusion x86 backend into gcc 2.95.2. Now everyone could benefit from the nice new backend, not just the people who knew about the "secret GPL compiler" included on the CodeFusion CD. But we decided to go ahead and use gcc rather than the CodeFusion compiler. In addition to being more stable, gcc 2.95.2 also allowed us avoid Cygnus, which by this time had been purchased by RedHat for a ridiculous sum of money. (Note: the new x86 backend in gcc 2.95.2 is what gave newer Linux distributions the significant speed boost that we all got to experience. It also gave FreeBSD 4.0 a nice speed boost over 3.3.6. Notice the difference?)
Then install the following code in your <tt>extensions</tt> directory and include it with a <tt>require_once( "$IP/extensions/ConsoleOutput.php" );</tt> in <tt>LocalSettings.php</tt>:


=== On the soapbox ===
<syntaxhighlight lang="php">
<?php
$wgExtensionCredits['validextensionclass'][] = array(
    'name' => 'ConsoleOutput',
    'author' => 'Daniel Robbins',
    'url' => 'https://github.com/danielrobbins/mediawiki-consoleoutput',
    'description' => 'This extension allows you to display colorized console output in mediawiki'
);


Thanks to this and other experiences, I've learned a lot about for-profit open source companies. There's absolutely nothing bad about being a for-profit open source company. Nor is there anything morally wrong with producing proprietary closed-source software, if that's what you'd like to do. But it doesn't make any sense for open source companies to subvert or refuse to cooperate with the rest of the open source world, either by not supporting the GPL or by any other means. This is a practical point that clearly makes business sense.
if ( defined( 'MW_SUPPORTS_PARSERFIRSTCALLINIT' ) ) {
        $wgHooks['ParserFirstCallInit'][] = 'consoleOutputSetup';
} else {
        $wgExtensionFunctions[] = 'consoleOutputSetup';
}


Open source companies should realize that the free exchange of ideas and code is what they profit from. By opposing things like the standard GPL practices, they undermine the environment they rely upon to prosper and grow. If open source is the soil from which your business has sprouted, it makes sense to keep the soil healthy.
function consoleOutputSetup( $data )
{
    global $wgParser;
    $wgParser->setHook('console', 'consoleRender');
    return true;
}


I understand that there's a temptation to keep at least some information secret for short-term financial gain. Advanced code or special techniques provide a coveted competitive advantage, which could potentially result in increased sales and profit. But if the goal is to be the sole provider of a product, the product should be commercial rather than open source. Open source does not allow for exclusive access to the inner workings of anything. That's what it means.
function consoleRender($input, $args, $parser)
{
    if (count($args))
    {
        return "<strong class='error'>" .
              "ConsoleOutput: arguments not supported" .
              "</strong>";
    }


=== Back to Enoch ===
    # Display < and > as literals, so escape them:


Now, I'll step down from my soapbox and continue my story.
    $input = preg_replace('/>/','&gt;', $input);
    $input = preg_replace('/</','&lt;', $input);


As Enoch became more and more refined, we decided that a name change was in order, and "Gentoo Linux" was born. By this time we had released a couple of versions of Enoch (now Gentoo), and were racing to get to Gentoo Linux version 1.0. Around this time I also decided to upgrade my old Celeron 300 box (overclocked and rock-solid at 450Mhz) to a brand-new Abit BP6 (a dual Celeron board that had just hit the market). I sold my old box and put my dual Celeron 366 system together. After overclocking the processors to something on the order of 500Mhz, I was cruising. But I noticed that my new machine wasn't very stable.
    # http://www.perlmonks.org/?node_id=518444
    # See "Matching a pattern that doesn't include another pattern:


Obviously my first reaction was to go back down to 2x366Mhz. But now I experienced an even stranger problem. As long as my machine kept the CPUs chugging away, the machine didn't lock up. But if I left the machine idle overnight, there was a good probability that the system would lock up completely. Yes, an idle bug -- argh! After some research, I found several other Linux users with the same problem on this particular motherboard. A chip on the BP6 (was it the PCI controller?) seemed to be flaky or out of spec, which caused Linux to lock up at idle.
    $input = preg_replace('/##i##((?:(?!##!i##).)*)##!i##/','<span class="code_input">$1</span>', $input);
    $input = preg_replace('/##i##(.*)/','<span class="code_input">$1</span>', $input);
    $input = preg_replace('/##b##((?:(?!##!b##).)*)##!b##/','<b>$1</b>', $input);
    $input = preg_replace('/##b##(.*)/','<b>$1</b>', $input);
    return "<pre class=\"code\">" . $input . "&lt;/pre>";
}
?>
</syntaxhighlight>


I was more than a wee bit upset, and because I couldn't afford to order more PC parts, Gentoo development effectively halted. I became more and more pessimistic about Linux and decided to switch over to FreeBSD. Yes, FreeBSD. And that's where I'll end this installment -- see you in Part 3. :)
[[Category:MediaWiki Hacks]]
{{ArticleFooter}}

Revision as of 01:05, January 12, 2015

The ConsoleOutput MediaWiki extension was created by Daniel Robbins to provide highlighting of user input for interactive terminal session blocks. To use it, surround user input with <console> opening and closing tags, and put a ##i## input code immediately before user input on each line. This will cause all text from the ##i## point to the end of the line to be in a darker color. The rest of the text will be in a lighter grey, allowing it to be distinguished more easily from user input. Here are other sequences that can be used:

  • ##b## - highlight the rest of the line in bold.
  • ##b## text here ##!b## - highlight the text between both markers in bold.
  • ##i## text here ##!i## - highlight the text between both markers in a darker color.

This extension can be further extended to add color support, as desired, and some of the CSS for color support can already be found in the code below.

Here are a few examples of the ConsoleOutput extension. First this is how you might typically display ls output, with a particular directory highlighted:

www@www-smw ~/public_html $ ls
COPYING  LocalSettings.php     api.php   extensions    index.php   maintenance           redirect.php    skins              thumb_handler.php5
CREDITS  README                api.php5  images         index.php5  mw-config             redirect.php5   tests              wiki.phtml
FAQ      RELEASE-NOTES-1.19    bin       img_auth.php   languages   opensearch_desc.php   redirect.phtml  thumb.php
HISTORY  StartProfiler.sample  cache     img_auth.php5  load.php    opensearch_desc.php5  resources       thumb.php5
INSTALL  UPGRADE               docs      includes       load.php5   profileinfo.php       serialized      thumb_handler.php
www@www-smw ~/public_html $ cd extensions/

And here is how you might display a more detailed example of console output, using colors:

root # bluetoothctl 
[NEW] Controller 00:02:72:C9:62:65 antec [default]
root ##bl##[bluetooth]##!bl### power on
Changing power on succeeded
root ##bl##[bluetooth]##!bl### agent on
Agent registered
root ##bl##[bluetooth]##!bl### scan on
Discovery started
root ##bl##[bluetooth]##!bl### devices
Device 00:1F:20:3D:1E:75 Logitech K760
root ##bl##[bluetooth]##!bl### pair 00:1F:20:3D:1E:75
Attempting to pair with 00:1F:20:3D:1E:75
[CHG] Device 00:1F:20:3D:1E:75 Connected: yes
root ##r##[agent]##!r## Passkey: 454358
root ##r##[agent]##!r## Passkey: 454358
root ##r##[agent]##!r## Passkey: 454358
root ##r##[agent]##!r## Passkey: 454358
root ##r##[agent]##!r## Passkey: 454358
root ##r##[agent]##!r## Passkey: 454358
root ##r##[agent]##!r## Passkey: 454358
[CHG] Device 00:1F:20:3D:1E:75 Paired: yes
Pairing successful
[CHG] Device 00:1F:20:3D:1E:75 Connected: no
root ##bl##[bluetooth]##!bl### connect 00:1F:20:3D:1E:75
Attempting to connect to 00:1F:20:3D:1E:75
[CHG] Device 00:1F:20:3D:1E:75 Connected: yes
Connection successful
root ##bl##[bluetooth]##!bl### quit
[DEL] Controller 00:02:72:C9:62:65 antec [default]
root #

To install, make the following modifications to your skin:

--- mediawiki-1.19.1/skins/vector/screen.css    2012-06-13 18:22:39.000000000 +0000
+++ public_html/skins/vector/screen.css 2012-08-27 04:34:47.507912892 +0000
@@ -683,10 +683,47 @@
        list-style-image: url(images/bullet-icon.png);
 }
 
-pre {
-       line-height: 1.3em;
+/* ConsoleOutput.php start */
+
+.shell, pre, code, tt, div.mw-geshi {
+        font-size: 12px;
+        font-family: Consolas, 'andale mono','lucida console', monospace;
+}
+
+.shell, pre, div.mw-geshi {
+        background-color: #F8F8FF;
+        line-height: 15px;
+        padding: 10px;
+        border: none;
+        border-top: 2px solid #C6C9E0;
+        border-bottom: 2px solid #C6C9E0;
+        margin: 0;
+        overflow-x: auto;
+        overflow-y: hidden;
+}
+
+.code {
+        color: #666;
+}
+
+.code_input {
+        color: #000;
 }
 
+.code_red {
+        color: #f00;
+}
+
+.code_blue {
+        color: #00f;
+}
+
+.shell_green {
+        color: #080;
+}
+
+/* ConsoleOutput.php end */
+
 /* Site Notice (includes notices from CentralNotice extension) */
 #siteNotice {
        font-size: 0.8em;

Then install the following code in your extensions directory and include it with a require_once( "$IP/extensions/ConsoleOutput.php" ); in LocalSettings.php:

<?php
$wgExtensionCredits['validextensionclass'][] = array(
    'name' => 'ConsoleOutput',
    'author' => 'Daniel Robbins',
    'url' => 'https://github.com/danielrobbins/mediawiki-consoleoutput',
    'description' => 'This extension allows you to display colorized console output in mediawiki'
);

if ( defined( 'MW_SUPPORTS_PARSERFIRSTCALLINIT' ) ) {
        $wgHooks['ParserFirstCallInit'][] = 'consoleOutputSetup';
} else {
        $wgExtensionFunctions[] = 'consoleOutputSetup';
}

function consoleOutputSetup( $data )
{
    global $wgParser;
    $wgParser->setHook('console', 'consoleRender');
    return true;
}

function consoleRender($input, $args, $parser)
{
    if (count($args))
    {
        return "<strong class='error'>" .
               "ConsoleOutput: arguments not supported" .
               "</strong>";
    }

    # Display < and > as literals, so escape them:

    $input = preg_replace('/>/','&gt;', $input);
    $input = preg_replace('/</','&lt;', $input);

    # http://www.perlmonks.org/?node_id=518444
    # See "Matching a pattern that doesn't include another pattern:

    $input = preg_replace('/##i##((?:(?!##!i##).)*)##!i##/','<span class="code_input">$1</span>', $input);
    $input = preg_replace('/##i##(.*)/','<span class="code_input">$1</span>', $input);
    $input = preg_replace('/##b##((?:(?!##!b##).)*)##!b##/','<b>$1</b>', $input);
    $input = preg_replace('/##b##(.*)/','<b>$1</b>', $input);
    return "<pre class=\"code\">" . $input . "&lt;/pre>";
}
?>