• View
• Edit
• History
• Attach
• Print
• Backlinks

WikiSh Tutorial

Obsolete Wikish is no longer supported nor maintained and does not work with modern versions of PHP

Introduction

WikiSh is a scripting language which emulates linux shell scripts, most particularly bash. It's pronounced (according to HansB) like the language that someone from the land of Wik would speak -- like Swedes speak Swedish the people from Wik speak Wikish. Welcome to the world of WikiSh! :-)

It is not a language designed to be easy to learn. It is designed to be a language which accomplishes a lot with a small amount of programmer effort once the language is learned. As such it is powerful but it has a fairly steep learning curve on the front end.

In shell scripting commands are typed onto a command line. But since WikiSh is designed in and around PmWiki, there is no command line available (although the control panel gets close - we'll talk about that later). Therefore we must start by discussing how commands will be executed.

All WikiSh commands (except wikish_controlpanel) are implemented as Markup Expressions (hereafter referred to as MXes, using HansB's nomenclature). This means the commands themselves will be placed within pages as any other markup and the output of the command will normally replace the markup. (That may sound like a strange way to say it, but it's what you are used to when you use markup -- if you put a (:pagelist ...:) markup in your page you expect the (:pagelist ...:) to disappear and be replaced by the list of pages that is the output of the command -- that's all I'm saying.) With MXes the appropriate syntax is to surround them with curly-braces and parentheses.

{(echo "Hello, World")}

can be placed anywhere on a page (assuming WikiSh is appropriately installed) and that markup will be replaced by the output of the command, in this case the text "Hello, World".

Later on (tutorial part 8?) we will show you the control panel -- it makes it a lot easier to format complicated scripts and also makes it quicker and easier to run commands. It's the closest thing WikiSh has to a command line, but all that comes later...

Before you start entering MX commands, please put the markup (:linebreaks:) at the top of the page. It will save you a lot of headaches of figuring out why your examples don't end up like mine...

Probably the single best way to go through this tutorial is to have a working wikish site in which you try out the various commands. Ideally you would go ahead and install wikish on your own site, but if you'd rather not there is a live test site available at http://pmwiki.qdk.org/pmwiki.php. You can try out most WikiSh commands on any pages there, but you have write privileges on pages only in the Test group (Test.*). There is the usual control panel available at http://pmwiki.qdk.org/pmwiki.php/WikiSh/ControlPanel if you want to try things out there instead of on pages directly. (I know I said we wouldn't get to it until part 8, but it makes life so much easier I thought I'd give you a break...)

Control Panel BRIEF summary: Anything you would have typed inside {(...)} can be typed directly into the control panel command field (without the {( and )}) and it will show you the results below after you submit the command.

Your first MX command: echo

Since we've already seen the echo command at work, let's take a quick look at it and do some examples with it. First of all you need to recognize that this echo command is NOT the PHP echo command. They do similar things and have a similar syntax, but don't think that you are somehow using the PHP echo command when you use the WikiSh echo MX.

Echo simply outputs its arguments. If output is not explicitly sent somewhere else (redirected - we'll talk about that later) then the output of the command will simply replace the MX on the page.

If you are doing this tutorial interactively (i.e., you are actually typing commands in as you go) then create a new page called WikiSh.Sandbox, edit it, and place this MX at the top of the page:

When you save the page you should see this:

Newlines are represented in WikiSh (as they are in so many languages, following Mr. Ritchie's excellent example) with a backslash followed by a lowercase n: \n. Let's try using that in your command. Go back and edit the page and make your MX read like this:

When you save the page you should see this:

OOPS! That's not what we were expecting, was it?! Well, it's not WikiSh's fault. If you could have seen the output from the echo MX before PmWiki got ahold of it you would have seen this:

which is what you expected...

However, PmWiki has as its default behavior that when 2 lines are placed one right after the other they are assumed to be part of the same paragraph and are therefore joined onto the same line and wordwrapped (it's actually a very nice feature and very useful, but it's a bit of a hassle in the WikiSh context most of the time). Since we're interested in seeing the effects of our newline we need to tell PmWiki to respect our line breaks. Go back into your WikiSh.Sandbox and edit it to look like this:

(:linebreaks:)
{(echo "Hello, World\nGoodbye, cruel world")}

Now when you save it you see this, as we originally expected:

Get used to putting that at the top of any page where the output of WikiSh will be placed -- about 80% of the time you are going to want to have it there...

Variables

No computer language is much good without variables, so let's talk about variables next. PmWiki already has a fairly extensive system of variables and WikiSh is proud to be able to "stand on the shoulders" of such a well-built system. Thus PTVs, PVs, i18n, etc should all work within WikiSh. But it also has its own system of variables, normally set by the use of the (appropriately named) set MX and then used by surrounding them with ${...} characters.

Note that when SETTING variables you do NOT surround them with ${...} but when you use them (i.e., when you are looking for what value they contain) then you DO surround them with ${...}.

Let's take a shot at the set MX in your growing WikiSh.Sandbox -- edit it to look like this:

(:linebreaks:)
{(set MyAge = 29)}
{(echo "Old people who lie say their age is ${MyAge}")}

When you save it you should see this output:

You will notice that the set command sets a variable but does NOT output anything -- therefore the command just disappears. However, the ${MyAge} inside the argument to the echo MX got replaced with that number, 29.

Strings, quotes, and variables

Computers like numbers and do letters because we humans can't get over our love affair with words. Letters and words to a computer are called "strings" and strings often take a little bit of special handling. First of all you will notice that I've put quotes around words each time I've used them. That's a good habit to get into. The quotes aren't part of the string, but the computer uses them to tie together all those letters and words into a single entity. (Note that the echo command can most often accept arguments either with or without quotes, but it's a good habit to use them consistently.)

The set MX, in particular, needs to know when you are working with strings and when you are working with numbers. It will assume you are working with numbers unless you explicitly say "Hey, this is a string!" You do this with the -s flag for the set command like this:

(:linebreaks:)
{(set -s HisName = "John")}
{(set HisAge = 29)}
{(echo "${HisName} says he is ${MyAge}.  He's a liar.")}

That will result in this when you save it:

Flags and Options

You use the -s what?! What's a flag?! I thought a flag was ...

A "flag" is a way of telling a WikiSh command to use a certain option. Usually a flag consists of a hyphen (a minus sign) followed by a single letter. Sometimes you can also use a long form which is 2 hyphens followed by a word (for instance, --ignorecase). The set command doesn't have a lot of options, but they are all documented on the WikiSh page. If I put in this command:

then you will get error messages from PHP whenever you load this page. Remember, you are working with a computer language and so you've got to get it exactly right or else the computer will be very happy to tell you all the ways you got it wrong! In this case you just go back in to edit the page and add the -s flag like so:

Another flag for the set command is the -v flag -- it says to "be verbose". Therefore whatever value it assigned to the variable will also be outputted. Using our earlier example with this modification:

(:linebreaks:)
{(set -v MyAge = 29)}
{(echo "Old people who lie say their age is ${MyAge}")}

When you save it you should see this output:

29
Old people who lie say their age is 29

Do you see the 29 above the sentence that is generated by the echo MX? That is set, outputting the value that was assigned to the variable MyAge.

But wait, what if you want to use more than one flag to set more than one option? Any way you like -- just about any reasonable combination will work. All of the following are equivalent:

Just make sure that ALL flags occur BEFORE any arguments start. This does NOT work as you expect:

{(set -s HisName = -v "John")}
{(echo "We set HisName to ${HisName})}

You might have expected the -v to output "John" and then see the output of the echo MX on the next line like this:

John
We set HisName to John

HOWEVER, that's not the case. Instead the -v flag occurring after the = sign will be interpreted as part of the string and will NOT tell set to "be verbose" and in fact will mess up the value of the variable. You will actually end up with output like this:

So when you are setting options using flags (either short or long forms) always put the flags AFTER the command name but BEFORE any other part of the command line.

Setting options in a way that is kinder to our friends from the land of PmWiki

If you are an experienced pmwiki administrator or author but not familiar with WikiSh or other linux shell script languages then you will probably find these flags a little hard to get used to. You are probably more used to an opt=value type of syntax when setting an option. Well, we in the land of Wik are kind and generous and we do what we can to help you out in this. There is a (somewhat unpreferred) alternate syntax for setting options which uses this opt=value syntax. You could use it with the set MX so that the following lines are all functionally identical:

Note that if you use this alternate syntax then the restriction on where the option setting occurs is no longer valid. Note, however, that not all WikiSh MXes can accept this alternate syntax. Note also that it takes a lot more typing to say "v=1 s=1" than it does to say "-vs". Come on, get lazy! Give your fingers a break!

Variables that speak their mind

Most variables are set by the script-writer and that's the value that they have until the script-writer changes their value. However, some variables ("content variables") have value without anything being assigned. For instance,

Will output the number of seconds since some arbitrary time in the past. You could use this, for instance, to see how long your script has been running:

{(set timestarted = ${NOW})}
...
{(set elapsedtime = ${NOW} - ${timestarted})}
{(if test ${elapsedtime} -gt 20; then; echo "uh, oh, PHP timeout is coming in 10 seconds if you haven't reset it!"; fi)}

But that's a lot of work on your part. There's another "content variable" named ${SECONDS} which always contains the number of seconds since WikiSh was initialized. So you could do the same thing as above much more easily like this:

Ah, that's nicer. But wait! What if someone has changed the default timeout?! How will I know? Fortunately for you there's yet another "content variable" named ${SECONDSLEFT} which is constantly updated with the number of seconds left until timeout occurs (and it knows what PHP has set the timeout). So we can do the same thing like this:

There are a bunch of other content variables -- check out Content Variables in the main WikiSh page to see a complete list.

Variables can have style and flair, too!

Variables can be manipulated in many ways through your script. But sometimes you want to do something quick and easy and don't want to have to write a special command to do it -- you just want the variable to display in a different way. For instance, maybe you want to know how many characters are in ${MyVar} (how long it is). You can find this out by using ${#MyVar} (note the hash symbol between the opening curly brace and the variable name. The value of the variable will NOT be returned, but instead the length of that value will be returned.

Or maybe you want only a part of your variable (substring) -- let's say you want characters 3 to the end (counting with the first character as 0, not as 1). You could do that with the {(substr ...)} MX available in core Markup Expressions, but it's easier to just do this: ${MyVar:3}. If you wanted characters 3-5 (again, counting from 0 - that's important!) then you can do it like this: ${MyVar:3:5}.

Search/replace can also be done. Let's search for "Jane" and replace it with "Sally": ${MyVar/Jane/Sally}.

There are more ways to modify variables that you can read about in Variable Modifiers in the main WikiSh page, but here's a quick run-down using examples:

set -s MyVar = "Me Tarzan, you Jane"
echo "${MyVar}"                     # outputs "Me Tarzan, you Jane"
echo "${#MyVar}"                    # outputs 19
echo "${MyVar:11}"                  # outputs "you Jane"
echo "${MyVar:3:8}"                 # outputs "Tarzan"
echo "${MyVar/Jane/Sally}"          # outputs "Me Tarzan, you Sally"
# Note that you can use wildcards (NOT REGEXes!) in the search side
# Wildcards are ? for any single character and * for any multiple characters
echo "${MyVar/you*/you are Tammy}"  #outputs "Me Tarzan, you are Tammy"
# Patterns can be separated by a comma (this is a PmWiki thing)
echo "${MyVar/Tarzan,Jane/dunce}"   #outputs "Me dunce, you dunce"
# Output an alternate text if a variable has a value
echo "${MyVar:+WOW}"                # outputs "WOW"
# Output a default value if the variable has no value
echo "${MyVar:-hello}"              # outputs "Me Tarzan, you Jane"
echo "${YourVar:-hello}"            # outputs "hello" as long as ${YourVar} is not set
# Output a default value if the variable has no value and set the variable to that value
echo "${YourVar}"                   # outputs nothing - the variable is not set (or is set to blank)
echo "${YourVar:=hello}"            # outputs "hello" and also sets the variable to "hello"
echo "${YourVar}"                   # outputs "hello" - the value was set on the previous line
# Delete stuff off the beginning of the value
set -s filename = "my/long/path/to/this/filename"
echo "${filename#*/}"               # outputs "long/path/to/this/filename" (deleted SHORTEST matching pattern)
echo "${filename##*/}"              # outputs "filename" (deleted LONGEST matching pattern)
# Delete stuff off the end of the value
echo "${filename%/*}"               # outputs "my/long/path/to/this" (deleted SHORTEST matching pattern)
echo "${filename%%/*}"              # outputs "my" (deleted LONGEST matching pattern)

And that's probably enough about variables. What do you think? Why don't you take a break and come back in a few minutes to pick up with the interesting stuff -- sending output somewhere besides into the current page...

Sending stuff somewhere else

Commands are pretty useless things unless they actually produce something. So just about every WikiSh command has some kind of output (even the {(set ...)} command has the -v option). And normally that output replaces the command in the page. When you edited your page you put the command {(echo "Hello, World")} and when your page was displayed you saw "Hello, World" in that spot. That's fine and dandy in a lot of cases, but what if we could take that output and send it to another location...

Introducing redirection...

The 2 angle brackets (shift-comma for left-angle-bracket and shift-period for right-angle-bracket) look kind of like arrows, don't they? Well, even if they don't, pretend they do because that will help you understand what I'm about to say. If you want the output from your command to go somewhere else then at the end of the command (the very last argument) put that right-angle-bracket as an arrow pointing AWAY from your command and TO a pagename (or other location). If you wanted to get input from somewhere else and send it into your command then you would use the left-angle-bracket pointing TO your command FROM some other location. In both cases the arrow shows the direction the data is moving -- either from your command to somewhere else or from somewhere else to your command.

That's kind of hard to visualize - let's take a look at this:

Do you see the right-angle-bracket "pointing" from the command to a page named Newpage? Well, in the page where the echo command was you will NOT see the "Hello, World" text anymore -- that text has been sent somewhere else! If you now go and browse Newpage (in the same group as the page with the command in it) you will see that it now has the text "Hello, World". In fact, if you had anything else on that page before you will see that your old text has been completely over-written and you have ONLY that "Hello, World" text now. The page looks like this:

So let's try putting some other text there. Go back to the page with the echo command and edit it to read like this:

{(echo "Hello, World" >Newpage)}
{(echo "Goodbye, Cruel World" >Newpage)}

Now we go back to Newpage and we expect to see this:

But in fact we see this:

What happened to the happy text?!? Well, when you originally wrote "Hello, World" to Newpage it OVERWROTE any existing text. When you wrote "hello, world" again it again overwrote any existing text, leaving just "Hello, World" a single time. ''Then when the next echo statement came along and wrote "Goodbye, Cruel World" it overwrote everything again and you end up with just the last text written to that page.

You may be thinking that this redirection is not so useful if it keeps overwriting itself! Stay tuned - we'll fix this problem. But for now make sure that you fix that issue in your mind. If you redirect text simply into a page it always overwrites the page and you can lose important stuff that way. Be careful!

Fortunately simple redirection is not the only way to send output somewhere else.

Let's take that right-angle-bracket and double it and see what happens:

{(echo "Hello, World" >>Newpage)}
{(echo "Goodbye, Cruel World" >>Newpage)}

The doubled right-angle-bracket is kind of like saying "don't just send the output to that page but send it even further -- all the way to the end of that page". It is the "append" operator. After we run the 2 MXes above this is what our page looks like (remember we already had some text there):

Hmmm... That wasn't what I was looking for -- I had forgotten that we already had our depressing text there. Let's edit the MX again so it looks like this and then rerun it:

{(echo "Hello, World" >Newpage)}
{(echo "Goodbye, Cruel World" >>Newpage)}

Notice that the FIRST redirection was with a single right-angle-bracket while the second redirection is with a double right-angle bracket. Thus the first one overwrote anything that was there and the second one appended to what the first one left there. We end up with this:

Ah, finally!

When you use redirection, think very carefully whether you want to overwrite or append to pages -- the difference can be catastrophic if you get it mixed around...

In normal shell programming that's pretty much what you can do with output redirection. But WikiSh has an advantage of being built on a platform that is somewhat resource "hoggish" but has great flexibility as a result. Thus the following additional possibilities exist for redirecting your output within WikiSh (keep the visual in your mind of angle-brackets being arrows and it will help you understand the logic of the syntax):

Either of these will redirect your output to the TOP of the specified page (a prepend operation)

{(echo abc >Newpage<_TOP)}
{(echo abc >Newpage<_BEGIN)}

All 3 of these will redirect your output to the BOTTOM of the specified page (an append operation) (notice the last as our friend from just above)

{(echo abc >Newpage>_END)}
{(echo abc >Newpage>_BOTTOM)}
{(echo abc >>Newpage)}

This will look on the target page (Newpage) and find the first occurrence of a pattern ("Hello" in this case) and place the output from the command immediate after (note the direction of the "arrow") that line:

So if we had our Newpage that looked like this:

and we ran that echo MX we would end up with this as the contents of Newpage:

It took the output of the command "abc". Then it looked in the target page (Newpage) and found the first occurrence of a line that matched the pattern "Hello" -- in this case it was the first line reading "Hello, World". Since the second angle-bracket pointed to the RIGHT it put the output of the command after that line and we ended up with the page shown above.

If we took that same page and ran this command:

It will find the matching line on line 2 ("abc") and put the output of the MX ("xyz") on the line immediately before that line (since the "arrow" pointed left). We end up with a page that looks like this:

Now up at the beginning of the section on redirection I mentioned that you could also use a left-angle-bracket to point from a page to a command. In linux shell this is something you use all the time, but for various reasons there is very limited call for it in WikiSh. In fact, the only MX that knows about the arrow going in this direction is the read command. You can do things like this:

   echo "WOW! I got ${line} from Newpage!"

(You'll notice we don't have the {(...)} surrounding those MXes -- that means it's something that would be typed into the control panel and we haven't gotten there yet. But I think you get the idea...)

Note in the first line you have "while read line" and then the next word starts with the left-angle-bracket. That shows you that the data is going to move in the direction of the "arrow" -- from Newpage into the read command. We aren't looking at the read command right now, but (briefly) it takes each line and puts them one-by-one into a variable which can then be used in the while-loop. So the output of that little "script" above is this:

Knowing how read works and how the while-loop is formed is not so important. The important thing to see is that we took the data from Newpage and passed it in the direction of the arrow (we "redirected" it) into the INPUT of the command {(read ...)}.

Enough talking about it -- LET'S DO SOMETHING!

WikiSh is not just a single command or markup. It is a collection of many commands which work together very flexibly and powerfully. You've already seen one of the most basic commands in the form of the {(echo ...)} command. Let's take a look at some more of the more commonly used commands now...

Listing files

PmWiki has a very powerful markup called (:pagelist ...:) with some great capabilities. Those capabilities have been replicated in another recipe called PowerTools in the Markup Expression format so you can do all those same things with {(pagelist ...)}. Since PowerTools is built on the same MX foundation that WikiSh is built on, the 2 recipes are compatible one with the other -- just make sure you use the wrap=inline option.

So one of the great ways to list files is pagelist, but if you're looking for something that is JUST text and has a few other options you might be interested in the {(ls ...)} command. In its simplest form {(ls)} it will list all the pages, each separated by a newline. You can also specify which pages you want listed using the normal wildcards: {(ls Main.*)} would list all the pages in the group "Main". {(ls Main.* WikiSh.*)} would list all the pages in both groups "Main" and "WikiSh". You can list as many pages or wildcard patterns as you like -- although it may take a while...

The ls command also supports the same list=X option that pagelist offers. So you could specify either {(ls list=normal)} or {(ls --list:normal)} (remember the different ways of specifying options?) and it will list all files except the PmWiki group, the RecentChanges files, etc. Ls also honors the default list option if it is set up on your system and uses list=normal if you don't have a default set up. You can override this either by specifying the -a flag or by specifying --list:all.

You can see different information about the files by specifying the -l (long listing) flag. When you do this it will list the last author, the size of the file, the date/time of the last modification, and the filename. You can order by size (-S), by time (-t), by name (default), or not sorting (--nosort). You can reverse any sort by including the -r flag.

All of the pieces of information in the last paragraph are page-based. For instance, the size is the number of characters in the most recent edition of the text of the page. Sometimes, however, you might be interested in what these pages look like on disc (so size would give you the full size of the file, for instance). When you want to see things from an OS level simply include the --os flag. All page attributes will then be file attributes instead.

Let's take a look at a few examples here (based on the pages I have in my system):

results in this output (all files in the Main group and all pages beginning with the letter C in the WikiSh group):

If I want to see more information I can ask for the same pages but in the long listing:

results in this:

   Peter Bowers      61 Mar 26 23:32 Main.Abc
   Peter Bowers     485 Feb 07 22:45 Main.Dirname
   Peter Bowers       1 Mar 15 23:56 Main.GroupAttributes
   Peter Bowers    5444 Nov 24 00:20 Main.HomePage
   Peter Bowers     196 Feb 05 08:54 Main.MkFiles
   Peter Bowers    1505 Mar 26 23:31 Main.PeterTesting
   Peter Bowers       2 Mar 06 16:05 Main.PeterTesting2
                   3096 Apr 08 08:31 Main.RecentChanges
   Peter Bowers       1 Mar 15 23:58 Main.SandBox
   Peter Bowers    2643 Apr 08 08:31 Main.WikiSandbox
   Peter Bowers     494 Jan 30 21:11 Main.WikiSh
   Peter Bowers     194 Apr 01 06:15 WikiSh.Cat
   Peter Bowers     239 Mar 02 20:47 WikiSh.CatData
   Peter Bowers      24 Mar 22 23:29 WikiSh.ControlPanel
   Peter Bowers     952 Apr 01 06:15 WikiSh.Cp
   Peter Bowers     706 Apr 01 06:19 WikiSh.CpTarget

Oh, I didn't mean to leave the Main.RecentChanges in there -- let me get rid of that:

results in:

   Peter Bowers      61 Mar 26 23:32 Main.Abc
   Peter Bowers     485 Feb 07 22:45 Main.Dirname
   Peter Bowers    5444 Nov 24 00:20 Main.HomePage
   Peter Bowers     196 Feb 05 08:54 Main.MkFiles
   Peter Bowers    1505 Mar 26 23:31 Main.PeterTesting
   Peter Bowers       2 Mar 06 16:05 Main.PeterTesting2
   Peter Bowers       1 Mar 15 23:58 Main.SandBox
   Peter Bowers    2643 Apr 08 08:31 Main.WikiSandbox
   Peter Bowers     494 Jan 30 21:11 Main.WikiSh
   Peter Bowers     194 Apr 01 06:15 WikiSh.Cat
   Peter Bowers     239 Mar 02 20:47 WikiSh.CatData
   Peter Bowers     952 Apr 01 06:15 WikiSh.Cp
   Peter Bowers     706 Apr 01 06:19 WikiSh.CpTarget

Which of those was changed most recently - let's sort it by modification time (-t):

(remember when specifying 2 short options I could say "-l -t" or I can just say "-lt") -- this results in:

   Peter Bowers    2643 Apr 08 08:31 Main.WikiSandbox
   Peter Bowers     706 Apr 01 06:19 WikiSh.CpTarget
   Peter Bowers     194 Apr 01 06:15 WikiSh.Cat
   Peter Bowers     952 Apr 01 06:15 WikiSh.Cp
   Peter Bowers      61 Mar 26 23:32 Main.Abc
   Peter Bowers    1505 Mar 26 23:31 Main.PeterTesting
   Peter Bowers       1 Mar 15 23:58 Main.SandBox
   Peter Bowers       2 Mar 06 16:05 Main.PeterTesting2
   Peter Bowers     239 Mar 02 20:47 WikiSh.CatData
   Peter Bowers     485 Feb 07 22:45 Main.Dirname
   Peter Bowers     196 Feb 05 08:54 Main.MkFiles
   Peter Bowers     494 Jan 30 21:11 Main.WikiSh
   Peter Bowers    5444 Nov 24 00:20 Main.HomePage

Now I know that the page size of HomePage looks larger than WikiSandbox, but I make a lot of changes to the sandbox. That means a lot of historical information has to be stored in that file -- let's see which one is larger when we look at the file size from an operating system level (--os) -- we'll sort by size (-S) just to make it clear:

this results in:

              0   91203 Dec 21 22:23 Main.HomePage
              0   40844 Apr 08 08:31 Main.WikiSandbox
              0   40122 Mar 26 23:31 Main.PeterTesting
              0    1620 Feb 07 22:45 Main.Dirname
              0    1617 Jan 30 21:11 Main.WikiSh
              0    1382 Apr 01 06:15 WikiSh.Cp
              0    1243 Feb 05 08:54 Main.MkFiles
              0    1222 Apr 01 06:19 WikiSh.CpTarget
              0    1105 Mar 02 20:47 WikiSh.CatData
              0     607 Apr 01 06:15 WikiSh.Cat
              0     434 Mar 26 23:32 Main.Abc
              0     432 Mar 15 23:58 Main.SandBox
              0     421 Mar 06 16:05 Main.PeterTesting2

(As you can see all files in my wiki.d directory are owned by root who has a uid of 0.) Well, it turns out that HomePage is still larger than WikiSandbox. But it's interesting that PeterTesting is almost as large as WikiSandbox -- I guess I must do a lot of testing... :-)

Hopefully by now you've got a pretty good idea of how to LIST pages and files. If you want to check out ALL the options you can always flip over the ls section of the main WikiSh page to read more about it.

Searching for text - GREP

What? The name "grep" isn't intuitively obvious to you?! Generalized Regular Expression Processor... That doesn't help, huh?

If the name "regular expression" doesn't ring a bell for you then you're probably going to benefit a great deal if you find a tutorial somewhere on the internet just to learn about them. In short they are an incredibly powerful way to describe what kind of text you are looking for. You're familiar with wildcards in filenames? Multiply that by about 100 or so...

So grep is used to search through text and return the matching lines (usually). For instance, if I search for "abc" in a file then it will return all the lines containing the text "abc". I do this like this: {(grep abc Main.GrepTest)}.

If you're not familiar with regular expressions they are going to seem a little magical (and maybe pretty intimidating), but let me give you the briefest of introductions to them...

BRIEF INTRO TO REGEX

• There are certain special characters:
  • . (period) - refers to any character (like ? in wildcards)
  • * (asterisk) - whatever comes before repeated 0 or more times (NOT like * in wildcards -- .* is like * in wildcards)
  • ? (question) - whatever comes before repeated 0 or 1 times
  • + (plus) - whatever comes before repeated 1 or more times
  • ^ (caret) - matches the beginning of the text/line
  • $ (dollar) - matches the end of the text/line
  • [xyz] - matches any character within the brackets
  • (There are lots of others, particularly parentheses, curly braces, etc.)
• Other characters match themselves (a matches a, b matches b)
  • If you want to match exactly one of the special characters (like a period or a dollar sign) then you can't just use that character -- it already has a special meaning. So you have to put a backslash in front of it to make it non-special
• Examples (each regular expression is surrounded by slashes in these examples):
  • /a/ matches the line "aaa" and "abc" and "xya" but not "xyz"
  • /./ matches the line "aaa", "abc", "xyz", etc - wherever there is at least one occurrence of any single character
  • /a*/ matches "a", "aa", "aaaaaaa" and "" (the last one is a repeated 0 times - nothing)
  • /^a/ matches "a" and "abc" but not "cba" (a is not at the beginning of this last one and our regular expression is looking for the beginning of the line followed by an a)
  • /^a$/ matches "a" but not "aa" or "ab" -- it is a line anchored on both ends with nothing but an a on it.
  • /^a.*z$/ matches "az" and "abz" and "abcdez" but not "abbbza". It's got to have an a at the beginning of the line and then any character zero or more times and then a z at the end of the line.
  • /ab?c/ matches "ac" and "abc" but not "abbc"
  • /ab*c/ matches "ac" and "abc" and "abbbbc"
  • /ab+c/ matches "abc" and "abbbbc" but not "ac".
  • /a[mn]z/ matches "amz" or "anz"
  • /a[mn]*z/ matches "az" or "amz" or "anz" or "ammmnnnz" or "amnmnmnz" but not "amnmnomnmnz" (note the "o" in the middle that is not one of the characters which are being repeated)

There are lots of great tutorials on regular expressions out on the internet. If you think you are going to be doing much searching it is very much in your interest to go through one of these tutorials.

Note that since WikiSh is built in PHP all regular expressions are using the PHP "flavor" of regular expressions. This won't have an impact on you in basic regular expressions, but if you get doing something advanced it will be important for you to be aware of this.

If you are pretty intimidated by regular expressions (or if you just have a piece of text you want to search for that has a bunch of "special characters" and you don't want to have to escape each of them with a backslash) then you can use the -F flag which says to search for the pattern as a fixed string instead of as a regular expression.

Hmmm... This part of the tutorial is getting pretty wordy and missing examples. Let's get on with it, shall we?

Let's say we have a page named WikiSh.GrepTest with this text:

abc
abcd
ab
a
aaa
abbba
the quick brown fox
jumped over the fence

To show some examples we will run a command that looks like this:

This says to search for a certain PATTERN (we will change this pattern each command) in the page WikiSh.GrepTest and output each line from that page that matches the pattern.

So for instance we might do this command:

In this case our pattern is simply the letter "a". So every line with the letter "a" in it will match the pattern. We end up with these results:

abc
abcd
ab
a
aaa
abbba

You'll notice that there are no lines in there about the quick brown fox or the fence he jumped over -- there weren't any letter "a"s in those lines so they were not printed out.

If we used a pattern of "ab" it our command would look like this:

And our results would look like this:

abc
abcd
ab
abbba

So far our patterns have been pretty boring -- let's try a more interesting pattern like this:

That pattern is searching for any line that starts with a and ends with another a with anything at all (or nothing) in between (I put the pattern in quotes this time -- it's a good idea to use quotes whenever you're going to start using dollar signs and other special characters -- the quotes "protect" the special characters from being mistaken for some other markup). Here's the results:

aaa
abbba

Grep also has an option to say "give me the lines that DON'T match" that pattern -- it is -v (reVerse the usual meaning of grep). So we could take the same command but use the -v flag and get a completely different result:

results:

abc
abcd
ab
a
the quick brown fox
jumped over the fence

You see every line in the file that did NOT match that pattern...

Grep is not limited to a single file -- you can specify as many files as you want:

That MX will search for the pattern "abc" in all the pages within the group Main, all the pages within the group WikiSh, and Page1 and Page2 within the group MyGroup. Note that it takes a lot of time to search through all the text on a page and so if you ask grep to search too many pages you may start stepping over PHP's default 30 second timeout (that depends on the speed of your web-host as well).

Putting it all together

The real power of WikiSh is not in each individual command, but in the way they can be put together... Your hammer and saw and ruler and level aren't so impressive on their own, but when you put them together you can build a house!

Let's say you are interested in generating a list of all pages in GroupA, GroupB, and GroupC that have a name of Page1 or Page3 or Page5 or Page7. You aren't sure whether you will have all 4 of those pages in each group, but you want to get a list of all that exist. You could do something like this:

But that's just painful! You could use wildcards like this:

But that would also return GroupX.PageA and GroupC.Page8 which are not what you are looking for. (Actually there is a wildcard solution that would solve this problem, but I'll leave that as an exercise for the student) Meanwhile we'll solve this by combining the 2 commands we've used so far:

{(ls Group?.Page? >Tmp)}
{(grep 'Group[ABC]' Tmp >Tmp1)}
{(grep 'Page[1357]' Tmp1)}

This searches for all groups which are named GroupX where X is any letter or number. Within those groups it searches for any pages which begin with "Page" and then end with any single letter or number. That list is redirected (remember? the > is like an arrow for where the output is going) into the page Tmp. The next command searches for the pattern Group[ABC] which gets rid of GroupX and Group3. It is not searching through all files anymore -- now it is searching through the lines of text on the page Tmp -- the output of this commmand are then redirected into another page called Tmp1. The final command searches through the lines of text on Tmp1 and finds only those pages with the appropriate pagename. The output of that command is NOT redirected anywhere, so it appears on your page.

See how the commands work together? Unfortunately it's a bit of a pain having to make all those temporary files... Wouldn't it be nice if...

Yes, as a matter of fact WikiSh has another way to redirect input and output. It's called a pipe because the idea is you take the output of one command and run it through a pipe and into the input of the next command. The way it is represented in the command is pipe-like, too -- it's a simple vertical bar.

Now in order to combine multiple commands together using pipes you have to be in a special MX within WikiSh called, appropriately enough, {(wikish ...)}. This MX is kind of the glue that holds all the other MXes together and we'll be using it a LOT as we move forward. Here's what our MX above would look like using pipes:

{(wikish ls Group?.Page? | grep 'Group[ABC]' | grep 'Page[1357]')}

You would read that like this: List these files and pipe that into grep for Group[ABC] and pipe that into another grep for Page[1357].

See how much easier that is with piping?

Now that we've worked through the whole example I'll confess that wildcards also support the character classes between square brackets. So you could have accomplished that with this simple command:

{(ls Group[ABC].Page[1357])}

But then you would never have learned about pipes -- think how sad you would be...

Consider another example. Let's say that you created several temporary pages in the WikiSh group while you were working on a WikiSh script. Each one was named Temp1, Temp2, TempA, etc. Now you want to list all the files in WikiSh, but you aren't interested in seeing those temporary pages. You CANNOT accomplish that with wildcards, but you can with a simple piping exercise:

{(wikish ls WikiSh.* | grep -v WikiSh.Temp)}

Isn't there an easier way?!?

I'm sick of typing MX commands into pages and having to save them and then edit them again. There's got to be an easier way...

Yes, there is. Create a page (it is recommended to call this page WikiSh.ControlPanel, but you can call it whatever you want) and put the single markup below in the file:

Now when you go to that page you will be able to type commands in and press a single button to run them. Also you'll be able to create commands that are many lines long without difficulty. When you press the "Execute Command" button you will always see the {(wikish ...)} command below (control panel commmands are always converted into their wikish equivalents before they are executed) and then you will see the output below that. Each command you type in will be saved into history and you have the option of copying certain commands into a favorite list as well. There are other options on this form, but they should be pretty self-explanatory. If something is unclear put a question on the WikiSh-Talk page and I'll clarify it.

From now on in this tutorial we will put commands in {( ... )} only if it makes better sense to do it on a page. For most of the commands we will just type them in and you will need to know that these commands are to be executed in the control panel.

(Any command you can type into the control panel can also be put in a {(wikish ...)} command -- just replace every newline with a semi-colon. It's a little hard to read sometimes, but it works.)

Decisions, decisions, decisions

   COMMAND
   ...

   COMMAND
   ...

Every WikiSh command sets a ${STATUS} variable upon completion. If the command was successful then ${STATUS} will be 0. If the command was unsuccessful then the ${STATUS} will be non-zero. In WikiSh (in contrast with languages such as PHP or Perl) a return status of 0 means true and a return status of non-zero means false. The "if ... then ... else ... fi" structure takes advantage of this return status. The command that comes right after the "if" will be evaluated and based on its ${STATUS} that is returned it will either execute the "then" block of commands or the "else" block of commands.

Let's see how this works using a command we're already familiar with: grep. Grep sets ${STATUS} to 0 (true) if it found something to return (i.e., at least one line matched the pattern if no -v was specified, at least one line didn't match the pattern if -v was specified, etc.). You should also know about another flag for grep, the -q flag. It tells grep to be quiet and don't actually output any lines -- just set the ${STATUS} variable appropriately. Knowing this we can write the following script:

   echo "They like me!  They said hello!"

   echo "I'm so depressed. Nobody even said hello."

If commands are connected to one another then it's always the LAST command whose ${STATUS} we are interested in. So if we used a pipe we could do this:

   echo "Ah, my favorite page is still there"

   echo "WHAT?! Someone deleted my favorite page?!?"

You can also combine commands using && (boolean and) or || (boolean or). If you're not familiar with boolean concepts this may feel a little strange, but here's how it would look:

   echo "Ah, my favorite page is still there and somebody greeted me too"

   echo "WHAT?! Someone deleted my favorite page?!?  Or maybe they just never said hello"

Or with the OR

   echo "Either my favorite page is still there or else someone greeted me - as long as one or the other happened I'm happy"

   echo "WHAT?! Someone deleted my favorite page AND they never said hello?  How bad can it get?!"

Two important facts:

• If commands are connected with || then as soon as one command returns a true none of the other commands in the line are executed - there's no need.
• Similarly if commands are connected with && then as soon as one command returns a false then none of the other commands in the line will be executed.

Knowing this you can actually do quick if-then's without even using if-then structure. Think about it - you'll see what I mean...

test

While grep is useful in certain boolean contexts (is this pattern in this file?) there are a lot of other boolean tests that we would like to do that grep simply cannot help us in. That's because you don't use your screwdriver to pound a nail -- we need to find the right tool. There is another WikiSh MX designed to test all kinds of data and it is named, appropriately enough, {(test ...)}.

It has way too many options to look at all of them in this tutorial, but you can always look over at the test command on the main WikiSh page for all the details. For now we'll look at the 3 main types of testing you can do and look at a couple different possibilities in each:

tests regarding pages and files

Does WikiSh.Page1 exist?

   echo "Yes, it exists"

   echo "Sorry, no such file exists"

Is WikiSh.Page1 newer than (has been modified since) WikiSh.Page2?

   echo "Yes, it is newer - please read Page1 for the most up-to-date stuff"

   echo "Sorry, it's not newer -- maybe you should look at Page2 for the most recent information"

Numeric tests

String tests (text)

Round and round she goes - where she stops nobody knows - LOOPING!

LOOPING! It's certainly one of the most important basics of any programming or scripting language and WikiSh has a few different ways you can do it.

while ... do ... done

   COMMAND
   ...

Here’s the overall way a “for loop” will look:

for VAR in ITEM1 ITEM2 … ITEMn
do
   COMMAND1
   COMMAND2
   …
   COMMANDn
done

The "VAR" is a variable name without surrounding ${…} – just the bare variable
name.
The items can be any series of words or characters or a list of pages (including wildcards
which will be expanded). They will be separated by any whitespace (space, tab, etc) and
so if you want to include whitespace in an item be sure that you surround that item with
quotes.
You can have any number of commands, including other loops, other if statements, or
any other command that could have been put in a Markup Expression.

Here’s a very simple “for” loop that counts to 5:

for i in 1 2 3 4 5
do
   echo “Counting ${i} out of 5”
done

Here’s a very simple “for” loop that processes through a list of pages – all it does is
output a simple text with the pagename, but you could very easily do a much more
functional processing in place of that {(echo …)} statement

for page in Main.*
do
   echo “I would have processed page ${page}”
   echo “Another line showing you can have multiple commands in the body of a loop”
done

Here’s something a little more interesting in terms of processing a list of files. In this
example we look for the word (pattern) “Harry” in each of a list of files and print a text
for each file in which we find that pattern.

for page in Main.*
do
   if grep –q Harry ${page}
   then
      echo “Page ${page} talks about Harry.”
   fi
done

Here’s how you would have done the same thing with a “while” loop (note the use of the
“-l” flag in grep to list just the pagename where the pattern is found and not the actual
text within the page that matches the pattern):

grep –l Harry Main.* | while read page
do
   echo “Page ${page} talks about Harry.”
done

Here’s something a little more complicated – we’ll find pages that contain BOTH the
pattern “Jack” AND the pattern “beanstalk” and output a text with the pagename. Note

{(grep –i 
beanstalk)}

for page in Main.*
do
   if grep –q Jack ${page} && grep –iq beanstalk ${page}
   then
      echo “Page ${page} talks about Jack and the Beanstalk – I just love that story.”
   fi
done

Here’s a simple loop that counts to 5 using while instead of for. Note the use of the
{(test …)} and the {(set …)} MXes to do this.

set i = 1
while test ${i} –le 5
do
   echo “Counting ${i} out of 5”
   set i = ${i} + 1
done

Here’s a way to use loops to create a nice-looking table of every line in 3 different pages
with the line number of that line beside it:

echo “|| width=90% border=1
echo “||!Pagename ||!Line# ||!Line ||”
for page in PageA PageB PageC
do
   set i = 1
   while read line <${page}
   do
      echo “||${page} ||${i} ||${line} ||”
      set i = ${i} + 1
   done
done

Taking it a step further, let’s create this table in such a way that the pagename is listed
only on the first line of that page and leave it blank on all the rest of the lines on that
page.

echo “|| width=90% border=1
echo “||!Pagename ||! Line#||!Line ||”
for page in PageA PageB PageC
do
   set i = 1
   while read line <${page}
   do
      if test ${i} == 1
      then
         echo “||${page} || ${i}||${line} ||”
      else
         echo “|| ||${i} ||${line} ||”
      fi
      set i = ${i} + 1
   done
done

OK we’ve done a bunch of things that have been interesting but of little practical value.
Let’s do something that is actually helpful to an administrator or author within the
PmWiki environment. We’ll take a set of pages and make them into a wiki trail.

If you don’t know what a wiki trail is, you really ought to learn about it – it’s a great
feature PmWiki provides. Basically it allows a set of pages to be connected and allows
someone to click on a link to move throughout the “trail” one by one in a certain order.

Quick explanation of Wiki Trails

To implement this you have to put a line like this on each page in the trail:

Where “TrailPage” is the name of another page which contains a list of links to each of
the pages in the order you want the trail to follow. Something like this:

* PageA
* PageX
* Page3

If you want to know more about wiki trails or if you haven’t understood this explanation,
take a look at WHAT LINK DO I USE?!?. But for now our job is to put the
“<< | … | >>” text at the end of each page and to create the list in the TrailPage page.
Here’s the script:

ls NewGroup.* | grep –v Temp | while read page
do
   echo “<<| NewGroup.TrailDef |>>” >>${page}
   echo “* ${page}” >>NewGroup.TrailDef
done

That works great from a functional standpoint, but it ignores a very important factor
when working within a PHP environment in web design. PHP has a default timeout of 30
seconds within which any PHP script can run. Since WikiSh is written in PHP any
script you write in this scripting language is also subject to this PHP limitation. This
means if you process a bunch of files in one loop (especially if you have a slow host)
there’s a great chance you will hit this timeout and your script will quit in the middle.

To solve this problem we have to make our script a little more complicated. Basically we
have to tell the {(read …)} command to save the status in your web session and
then restore that status when the command is run again. Then you will need to run the
command several times until the whole job is done. Each time the script is run it will
process through a few more pages in the loop.

If you include the --restoreset option to the {(read …)} MX then it will
automatically look for any saved status in your session (it will ignore any input coming
from a pipe or redirected standard input). If you include the --saveset option then
{(read …)} will save the status of the command in such a way that it can pick up
where it left off.

ls NewGroup.* | grep –v Temp | while read --restoreset page
do
   echo “<<| NewGroup.TrailDef |>>” >>${page}
   echo “* ${page}” >>NewGroup.TrailDef
   if test ${SECONDSLEFT} –lt 7
   then
      read --saveset --nonext
      echo “NOTE: This script is NOT complete – you MUST run it again!”
      echo “PROGRESS: “ `read --status` “ processed out of “ `read --lines` “total”
      exit
   fi
done

Mini-scripts inside other scripts

Backquotes

Often it is very convenient to use the output of one MX as part of the command-line of another MX. For instance, let's say I want to do a search-and-replace of "foo" to "bar", but I don't want to process all files with sed -- I want to search using grep and only run sed on the pages that actually contain an occurrence of "foo". I could do that like this:

sed -i 's/foo/bar/g' `grep -l foo Test.*`

Because of the backquotes around the grep -l foo Test.* that portion of the line will be executed FIRST and the OUTPUT of that MX will be placed on the command line IN PLACE OF the backquoted command. So if Test.Abc and Test.Xyz are the only 2 pages in the Test group containing an occurrence of "foo" then the output of grep -l foo Test.* is

Test.Abc
Test.Xyz

The newline looks like it would be a problem, but WikiSh takes care of that for you. So by the time sed sees the command line it looks like this:

sed -i 's/foo/bar/g' Test.Abc Test.Xyz

Thus sed will only process the 2 pages instead of all the pages.

Note that this example is a bit pointless. Sed is just as efficient as Grep in searching for "foo" and so you might as well let sed do it in a single pass rather than having grep look through once and then sed going through a second time. But for illustration purposes I hope the example shows how backquotes can be used. Using the MX pagelist from powertools.php WOULD be a very good example of how to speed things up, but I'll leave that as an exercise for the student. (Sorry, couldn't resist - that was my bash prof's favorite line.)

Other MXes in parentheses

This particular feature is not unique to WikiSh but rather to MXes in general. You can put any MX in inner parentheses and it will be executed before anything in the outer parentheses and the output of the inner MX becomes part of the "command" of the outer MX. Using the example from above you could write it like this (I'm putting it as it would be placed on a non-control-panel page):

{(sed -i 's/foo/bar/g' (grep -l foo Test.*))}

In fact, it sounds identical to backquotes. However, there is a HUGE difference:

• Backquotes get executed immediately prior to the command they are a part of
• Inner-parenthesized MXes get executed prior to ANY part of the outer
• (There's another, less important, differece as well: parenthesized MXes must NOT be within a quoted string whereas backquoted commands can be inside of a quoted string)

This becomes significant as we compare these 2 scripts:

set -s name = "John Smith"
echo "My name is `echo ${name} | sed 's/John/Sam/'`

In this case we get a value in the variable ${name} and then immediately prior to echoing "My name is John Smith" we change the John to a Sam. The result of this script is "My name is Sam Smith".

set -s name = "John Smith"
echo "My name is " (wikish echo ${name} | sed 's/John/Sam/')

In this latter script we execute the parenthesized MX before we do anything else (note the need to include "wikish" because this inner MX is an independent MX on it's own). So we echo what is in ${name} and change "John" to "Sam". But, wait! We don't have anything in ${name} yet! The inner parenthesized MX executed before ANY other part of the outer MX and that means it executed before the variable ${name} was initialized. So the inner MX echoes a blank value through the sed statement and results in a blank value. THEN and only then the outer MX is processed -- if there were some way to look at it after the inner MX was processed and before the outer one was processed you would see something like this:

set -s name = "John Smith"
echo "My name is "

The result, as might be expected now that we understand the issue, is "My name is" with nothing following it.

There are times when nested parentheses can be used to great effect in markup expressions, but in most cases with WikiSh the backquotes will result in a much more expected result.

Using non-WikiSh MXes

Any MX can be used within a WikiSh script, but not all MXes are created equal. If it is not a WikiSh-compatible MX then (1) it cannot receive input from a pipe, (2) it cannot redirect standard output, (3) it cannot use WikiSh variables, and (4) it possibly will not recognize wildcards to specify pagenames. Although a foreign MX cannot receive from a pipe, any MX which produces output can have that output sent through a pipe.

Here are some suggestions of how to get around these limitations:

1. You can get around this by redirecting the output into temporary page and then using that page, perhaps with backquotes. Just check carefully what the MX expects on its command line.
2. Any MX which produces output can "feed" a pipe. It may not be able to receive input from a pipe, but it can send output to a pipe. So if you want to redirect somewhere, simply pipe the output of your "foreign MX" into cat and then redirect to your heart's content.
3. If it occurs within backquotes then you will be able to use WikiSh variables, but otherwise you're kind of stuck. If you really need this feature contact me and I'll see if I can't figure some meta-way to allow foreign MXes to use WikiSh variables
4. Just specify the pages by typing them out or perhaps use a command like ls within backquotes

Probably the most useful non-WikiSh MX you will find occasion to use is the pagelist MX found within the PowerTools recipe. Make sure to specify the option sep=\n to change the output from comma-separated to newline-separated if you are using the default fmt. If you are at all familiar with PmWiki you are aware of the extreme power of pagelists and also the tremendous optimization that has been achieved in the production of these -- WikiSh does not make use of the cache at all, and so you will do yourself a BIG favor by using pagelist whenever you are going to be searching through a large number of pages.

Specifying files "in-line"

Most WikiSh MXes expect pagenames on their command-line. If, however, you prefix an argument with a hyphen and a space then that argument will be assumed to be the in-line content of a page rather than the name of the page.

cat "this is so much fun"

This will attempt to find a page in the current group named ThisIsSoMuchFun and show you the contents of it.

cat - "this is so much fun"

This will see the argument "this is so much fun" as the contents of an in-line file and it will show you the contents of that page. You will see "this is so much fun" as the result of this MX.

Obviously these "in-line" files can be mixed and matched with regular pagenames and etc. You can specify as many of them as you want. Quotes are going to be very significant. Note that a parenthesized MX normally results in a "quoted" argument. That's significant.