\documentstyle[12pt]{article}

\newcommand{\RR}{R\!\!\!\!I\,}
\newtheorem{algorithm}{Algorithm}[section]

\title{Numerical Linear Algebra}
\author
{James W. Demmel\thanks{The author was supported by NSF grant
ASC-9005933, DARPA contract DAAL03-91-C-0047 via a subcontract from
the  University of Tennessee administered by ARO, and DARPA grant
DM28E04120 via a subcontract from Argonne National Laboratory. This
work was partially performed during a visit to the Institute for
Mathematics and its Applications at the University of Minnesota.}\\
Computer Science Division and Mathematics Department\\
University of California at Berkeley\\
Berkeley, CA 94720 USA\\E-mail: demmel@cs.berkeley.edu
\and Michael T. Heath\thanks{The author was supported by DARPA
contract DAAL03-91-C-0047 via a subcontract from the University of
Tennessee administered by ARO.}\\
Department of Computer Science\\and\\National Center for Supercomputing
Applications\\
University of Illinois\\
Urbana, IL 61801 USA\\E-mail: heath@ncsa.uiuc.edu
\and Henk A. van der Vorst\thanks{This work was supported in part by
NCF/Cray Research University Grant CRG 92.03.}\\
Mathematical Institute\\Utrecht University\\
Utrecht, The Netherlands\\E-mail: vorst@math.ruu.nl}

\begin{document}
\maketitle

\section{Introduction}

We begin with an example to motivate our study of parallel and vector algorithms
for linear algebra problems. Consider solving a system of linear equations 
$Ax=b$, where $A$ is a given dense $n$-by-$n$ matrix, $b$ is a given
$n$-by-1 vector, and $x$ is an $n$-by-1 vector of unknowns we want to compute. 
The algorithm we will use is {\em Cholesky}, which is a variation of Gaussian 
elimination suitable for matrices $A$ with the special (but common) properties 
of {\em symmetry} and {\em positive definiteness} (these are discussed in 
more detail below).  Cholesky, as well as Gaussian elimination, consists almost 
entirely of simple, regular operations which are in principle easy to parallelize 
or vectorize: adding multiples of one column of the matrix to other columns. 
Since vector processors like the Cray Y-MP are designed to perform this operation 
particularly well, we expect to get good performance running Cholesky on a Cray Y-MP.

\begin{table}[h]
\caption{Performance in Megaflops of Cholesky on a Cray Y-MP}\label{table_CholeskyCrayYMP}
\begin{center}
\begin{tabular}{lrr}
\hline
  &  1 Proc. & 8 Procs. \\ \hline
Maximum speed     &          330 & 2640 \\
Matrix-matrix multiply     & 312 & 2425 \\
Matrix-vector multiply     & 311 & 2285 \\
% Solve $TX=B$               & 309 & 2398 \\
Solve $Tx=b$               & 272 &  584 \\
LINPACK (Cholesky, $n=500$) & 72 &   72 \\
LAPACK (Cholesky, $n=500$) & 290 & 1414 \\
LAPACK (Cholesky, $n=1000$)& 301 & 2115 \\ \hline
\end{tabular}
\end{center}
\end{table}

The second column of Table~\ref{table_CholeskyCrayYMP} gives the speed on a single
processor of a Cray Y-MP; here vectorization is feature to be exploited.
The third column gives the speed using all 8 processors; here parallelization as well
as vectorization is possible. The first row of the table gives the maximum possible
speed in megaflops of the machine; no algorithm for any problem can go faster.
The next three rows give the speed of various Basic Linear Algebra Subroutines
(or BLAS), such as multiplying two large, square (dimension $n=500$) matrices, multiplying
a large square matrix by a vector, and solving a triangular system of equations
$Tx=b$ by substitution. These BLAS have been written in Cray assembly language and
fully exploit the Cray Y-MP architecture; we see that they generally come quite
close to the maximum performance given in the first row. 

The fifth row of the table gives the performance of the Cholesky subroutine 
{\tt spofa} from the
LINPACK subroutine library \cite{linpack}. LINPACK is a well-known and widely used
library of Fortran 77 linear algebra routines, but was written before the advent of vector
and parallel computers. Its performance is very poor, over 4.5 times slower than
the maximum machine speed on a single processor and over 36 times slower than maximum on
8 processors. This is very disappointing, since at first glance it is hard to
imagine an algorithm better suited to the Cray than Cholesky.

The last two rows of Table~\ref{table_CholeskyCrayYMP} give the performance of a
different version of Cholesky, {\tt sposv}, taken from the newer LAPACK library of 
linear algebra routines \cite{lapackmanual}. LAPACK was written to exploit architectures 
like the Cray. It is also written in Fortran 77, but includes calls to the BLAS, which
are written in Cray assembler. It performs the same floating point operations as
LINPACK, yet goes 4 times faster on a single processor and almost 20 times faster
on 8 processors. Increasing the matrix dimension $n$ from 500 to 1000 increases
LAPACK's speed even more, to within 80\% of the peak machine speed.

The purpose of this example is to show that an ``obviously'' parallel or vector algorithm
may work much more poorly than expected, but by applying certain (systematic) transformations
to the algorithm, it can be made to work quite well. We understand these algorithmic
transformations most completely in the case of simple algorithms like Cholesky, on
simple (to the user) architectures like the Cray Y-MP. We understand them rather less well
for more complicated algorithms on more complicated architectures. Our goal in this chapter is
to discuss these transformations so the reader can learn to use them, and give pointers to
existing software (like LAPACK) where these transformations have already been performed.

In order to describe and solve the basic linear algebra problems of this chapter,
we need some notation. 
We will refer frequently to {\em matrices}, {\em vectors} and
{\em scalars}. A matrix will be denoted by an upper case letter like $A$,
and its $(i,j)$-th element by $a_{ij}$. Occasionally in detailed algorithmic
descriptions we will instead write $A(i,j)$. The submatrix of $A$ occupying
rows $i$ though $j$ and columns $k$ through $l$ will be denoted $A(i:j,k:l)$.
A lower case letter like $x$ will denote a vector, and its $i$-th element will
be written $x_i$.
Vectors will almost always be column vectors, which are the same as
matrices with one column. Lower case Greek letters (and occasionally lower
case letters) will denote scalars. $\RR$ will denote the set of real numbers,
$\RR^n$ the set of $n$-dimensional real vectors,
and $\RR^{m \times n}$ the set of $m$-by-$n$ real matrices.
$A^T$ will denote the {\em transpose} of the
matrix $A$: $(A^T)_{ij} = a_{ji}$. A matrix is called {\em symmetric} if $A=A^T$.
A matrix $A$ is {\em positive definite} if $x^T A x > 0$ for all nonzero
vectors $x$.  Other notation will be introduced as needed.

Computational science problems can often be reduced to solving one or a sequence of the
following {\bf standard linear algebra problems}:
\begin{itemize}
\item {\bf Solving linear systems of equations}: $A x=b$. Here $A$ is a given $n
$-by-$n$
nonsingular real or complex matrix, $b$ is a given column vector with $n$ entries, 
and $x$ is a column vector with $n$ entries we wish to compute.
\item {\bf Least squares problems}: compute
the $x$ which minimizes $\|Ax-b\|_{2}$ where $A$ is $m$-by-$n$, $b$ is $m$-by-$1
$, and
$x$ is $n$-by-$1$.  $\|y\|_{2} \equiv \sqrt{\sum|y_{i}|^{2}}$ is called the
{\em 2-norm} of the vector $y$.
If $m > n$, so we have more equations than unknowns, the system is called {\em
overdetermined}. In this case we cannot generally solve $Ax=b$ exactly.
If $m < n$, the system is called  {\em underdetermined}, and we will have 
infinitely many solutions.
\item {\bf Eigenvalue problems}: Given an $n$-by-$n$ matrix $A$,
find an $n$-by-$1$ nonzero vector $x$ and a scalar $\lambda$ so that $Ax=\lambda x$.
\end{itemize}

For example, linear equations often arise when solving ordinary or partial
differential equations numerically; the linear system must be solved to advance
the solution by a time step. Least squares problems arise in fitting curves or
surfaces to experimental data. Eigenvalue problems arise when analyzing vibrations.
There are also many important variations on these basic problems,\footnote{For
example, there is the
as the {\em generalized eigenvalue problem} $Ax = \lambda Bx$, where $A$ and $B$ are
given square matrices and we want to find the eigenvalue $\lambda$ and
eigenvector $x$, and the {\em singular value decomposition}, which can be described
as finding the eigenvalues and eigenvectors of $A^TA$ and $AA^T$.}
but to keep this chapter to a reasonable length, we will concentrate on
algorithms for solving systems of linear equations, and refer elsewhere
for least squares and eigenvalue problems 
\cite{demmelheathvorst,golubvanloan,watkinstext,MA221}.

A great many algorithms are available for all these problem. The choice of algorithm
depends on three things:
\begin{itemize}
\item The {\bf structure of $A$} has a large influence on the algorithm. For example,
solving $Ax=b$ using Gaussian elimination takes $2/3 \cdot n^3 + O(n^2)$ flops (floating point
operations) if $A$ is a dense matrix. If $A$ is {\em symmetric} and {\em positive definite}, 
we can use the Cholesky algorithm which costs half as much: $1/3 \cdot n^3 + O(n^2)$ flops.
If $A$ is {\em triangular} (i.e. either zero 
above the diagonal or zero below the diagonal), then we can solve $Ax=b$ in
only $n^2 + O(n)$ flops by simple substitution, which is much faster. 
Recognizing triangularity is easy, but other structures are more subtle. For example, 
if $A$ arises from solving certain elliptic partial differential equations 
(like Poisson's equation), then $Ax=b$ can actually be solved in $O(n)$ flops using 
{\em multigrid}, which is essentially as little work as possible.
\item The {\bf desired accuracy} limits which algorithms are suitable. For example,
if one needs to solve $Ax=b$ to 16 decimals of accuracy (more on what this means below),
one might use a careful and expensive implementation of Gaussian elimination, but if
one only needs 3 decimal digits, a few iterations of a much cheaper iterative
method like {\em conjugate gradients} might suffice.
\item The {\bf computer system} on which one wants to solve the problem influences
the algorithm choice greatly, and is the main topic of this chapter. Differences in
the computer architecture, available programming languages, and compiler quality
can make large differences in the way one implements a given algorithm; 
we saw this in Table~\ref{table_CholeskyCrayYMP} above, and will see it later as well.
The computer system can influence algorithm performance 
so much that one would choose very different algorithms on different machines.
New parallel architectures have also lead to the invention of new and interesting 
parallel algorithms.
\end{itemize}

Since the three standard problems listed above have been studied for so long, one 
might hope that good numerical software libraries would be available to solve them. 
This is partly true, as we illustrated by LAPACK \cite{lapackmanual,netlib}.
However, there are several reasons one cannot depend on libraries entirely.
First, it is impossible to anticipate all possible structures of $A$ that can arise in
practice and write corresponding libraries (although extensive libraries for various 
specialized $A$ do exist \cite{netlib}).
Second, high performance computer architectures, programming languages and compilers
have been changing so much lately that library writers have not been able to keep up.
There is no clear cut rule on when to use a library routine. In some cases an unoptimized
or serial implementation of an algorithm which fully exploits a matrix structure may be
much faster than a highly tuned, parallel implementation of a more general algorithm,
and in other cases the opposite may be true.
If the user's priority is ease of programming and reliability, 
a library routine is a good choice. If, instead, the user's priority is maximum performance
on one specialized problem, and software development time is secondary, custom software
is the likely choice. We will try to address the concerns of both classes of users in 
this chapter.

The rest of this chapter is organized as follows. 
Section~\ref{sec_MemoryHierarchies} discusses the cost models we will use
to understand algorithm performance.
Section~\ref{sec_BLAS} discusses the BLAS, which we use to construct many
high performance algorithms.
Section~\ref{sec_MatrixMultiplication} discusses how to implement matrix 
multiplication quickly on several different parallel architectures; these techniques 
serve as models for implementing more complicated algorithms.
Section~\ref{sec_DistLayout} discusses how to layout matrices on distributed
memory machines.
Section~\ref{sec_DenseGaussElim} discusses parallelizing Gaussian elimination on
dense matrices, and gives pointers to other related algorithms.
Section~\ref{sec_BandGaussElim} discusses parallelizing Gaussian elimination on
dense matrices, and gives pointers to other related algorithms.
Section~\ref{sec_SparseGaussElim} discusses parallelizing Gaussian elimination on
sparse matrices; this is a much harder problem, and much less software is
available. Section~\ref{sec_IterativeMethods} discusses iterative methods for
solving $Ax=b$, which in contrast to Gaussian elimination take an approximate
solution and iteratively improves it, rather than providing the final answer after
a fixed number of steps. 
Section~\ref{sec_ErrorBoundsForAx=b} shows how to bound the error in a computed
solution to $Ax=b$, for any of the methods discussed in this chapter.
Section~\ref{sec_Netlib} discusses how to use netlib to retrieve existing library
software electronically. Finally, section~\ref{sec_BLASRefGuide} gives a quick references
guide to the BLAS.

This is a necessarily partial survey of a large and active field. For further
reading in numerical linear algebra, see the textbooks by Golub and Van Loan 
\cite{golubvanloan}, Watkins \cite{watkinstext} or Demmel \cite{MA221}.
For more details on parallel numerical linear algebra in particular, see
\cite{demmelheathvorst,DDSV,gallivanetal1}, the last of which includes a bibliography
of over 2000 references.

\section{Memory Hierarchies}\label{sec_MemoryHierarchies}

To understand the speedups of LAPACK over LINPACK in Table~\ref{table_CholeskyCrayYMP},
and in general to be able to understand which algorithms are fast and which are slow,
we need to understand memory hierarchies.
Memory hierarchies were discussed in some detail in chapter~[@Computer Architecture], 
and we just review them briefly here, emphasizing their importance to algorithm design.

Computer memories are built as {\em hierarchies}, with a series
of different kinds of memories ranging from very fast, expensive, and therefore small
memory at the top of the hierarchy, down to slow, cheap and very large memory at
the bottom. For example, registers typically form the fastest memory, then cache,
main memory, disks, and finally tape as the slowest, largest and cheapest. Useful
floating point operations can only be done on data at the top of the hierarchy, in
the registers. Since an entire large matrix cannot fit in the registers, it must
be moved up and down through the hierarchy, transferred up to the registers when work
needs to be done, and transferred back down to the main memory (or disk or tape)
when it is no longer needed. It takes time to move between levels in the memory
hierarchy, and moving is slower the farther down in the hierarchy one goes.
Indeed, one such data movement takes far longer than performing a floating point
operation.
So the limiting factor for many algorithms is {\em not} the time needed
for floating point operations, but the time to move data in the memory hierarchy.
Therefore, a clever algorithm will attempt to minimize the number of these
memory moves (even if it might mean doing a few more floating point operations).

LINPACK's Cholesky performed so poorly in Table~\ref{table_CholeskyCrayYMP},
because it was {\em not} designed to
minimize memory movement on machines such as the Cray YMP (it was designed to
minimize another kind of memory movement, {\em page faults} between main memory
and disk). In contrast, the matrix-matrix multiplication
and other BLAS on the Cray YMP in Table~\ref{table_CholeskyCrayYMP}
were written (in assembly language) just for the Cray YMP to minimize data movement.

Since it is expensive and time-consuming to write every routine like Cholesky in 
assembly language for every new computer, we would like a better approach. 
Here is the most successful approach we have discovered so far.
Since operations like matrix-matrix multiplication are so common, computer manufacturers 
have standardized them as the {\em Basic Linear Algebra Subroutines} or {\em BLAS}
\cite{blas1,blas2,blas3}, 
\index{BLAS (Basic Linear Algebra Subroutines)}
and optimized them for their machines. In other words, a library of subroutines
for matrix-matrix multiplication, matrix-vector multiplication, and other
routines is available as a standard Fortran (or C) callable library on most
high performance machines, and underneath they have been optimized for each machine. 
If we can reorganize our algorithms to use these optimized BLAS to perform all or most
of the work, then our algorithms will go as fast as the manufacturer's optimized BLAS.
This was the approach taken in LAPACK: the algorithms in LINPACK (and the corresponding
library for eigenvalue problems, EISPACK \cite{eispack,eispack2}) were reorganized to
call the BLAS in their innermost loops, where most of the work is done. This led to the
speedups shown in Table~\ref{table_CholeskyCrayYMP}.

This approach was very successful on machines like the Cray, i.e. parallel vector processors
using fast shared memory with relatively few processors. On newer architectures, especially
distributed memory machines like the Intel Paragon and CM-5, this approach cannot be used
as straightforwardly, although many of the same ideas still apply. The difficulty with
distributed memory machines is twofold. First, the memory hierarchy is deeper,
including both ``local memory'' and ``remote memory'' layers at the bottom (remote memory
means memory physically on another processor). Individual nodes may also be more complicated;
for example, the CM-5 has 5 discernible levels of memory hierarchy on a single processor node; 
other machines are also complicated. Second, languages and compilers are still evolving,
so that there are many more possible ways to store a matrix on a machine, and ``obviously''
parallelizable or vectorizable loops may or may not be compiled well.

In the remaining sections, we will discuss the successful BLAS-based approach to numerical
software, and outline current activities on distributed memory machines.

\section{The BLAS}\label{sec_BLAS}

Let us examine the BLAS more carefully. For a more complete list of the BLAS,
see section~\ref{sec_BLASRefGuide}.
Table~\ref{table_BLAS} counts the number of
memory references and floating points operations performed by three related BLAS.
The last column gives the ratio $q$ of flops to memory references. The significance
of $q$ is that it tells us roughly how many flops we can perform per memory reference,
or how much useful work we can do compared to the time moving data; therefore,
the algorithms with the larger $q$ values are better building blocks for other
algorithms.

Table~\ref{table_BLAS} reflects a hierarchy of operations: Operations like {\tt saxpy}
operate on vectors and offer the worst $q$ values; these are called Level 1
BLAS \cite{blas1}, and include inner products and other simple operations.
Operations like matrix--vector multiplication operate on matrices and vectors,
and offer slightly better $q$ values; these are called Level 2 BLAS
\cite{blas2}, and include solving triangular systems of equations and
rank-1 updates of matrices ($A+xy^T$, $x$ and $y$ column vectors). Operations
like matrix--matrix multiplication operate on pairs of matrices, and
offer the best $q$ values; these are called Level 3 BLAS \cite{blas3}, and
include solving triangular systems of equations with many right hand sides.

\begin{table}[h]
\caption{Basic Linear Algebra Subroutines (BLAS)}\label{table_BLAS}
\begin{center}
\begin{tabular}{|l|l|c|c|c|}
\cline{1-5}
{\small Operation} $\!$\rule[-6pt]{0pt}{18pt} & {\small Definition} &
{\small Floating}  &
{\small Memory} & $q$ \\
          &            & {\small point}     & {\small references} & \\
          &            &{\small operations} &             & \\
\cline{1-5}
{\tt saxpy} $\!$\rule[-6pt]{0pt}{18pt} & $y_i\! =\!  \alpha x_i\! +\!
y_i$, $i\!=\!1,...,n$ & $2n$ & $3n\!+\!1$ & $2/3$ \\
\cline{1-5} {\small Matrix--vector mult} $\!$\rule[-6pt]{0pt}{18pt} & $y_i
= \sum_{j=1}^n A_{ij} x_j + y_i$ & $2n^2$ &
$n^2+3n$ & $2$ \\ \cline{1-5}
{\small Matrix--matrix mult} $\!$\rule[-6pt]{0pt}{18pt} & $C_{ij}\! =
\!\sum_{k=1}^n A_{ik}  B_{kj}\! +\! C_{ij}$ &
$2n^3$ & $4n^2$ & $n/2$ \\ \cline{1-5}
\end{tabular}
\end{center}
\end{table}

Since the Level 3 BLAS have the highest $q$ values, we endeavor to reorganize
our algorithms in terms of operations like matrix-matrix multiplication, rather
than saxpy (the LINPACK Cholesky is already constructed in terms of calls to
saxpy).

\section{Matrix Multiplication}\label{sec_MatrixMultiplication}
\index{block algorithms!matrix multiplication}

\subsection{Matrix Multiplication on a Shared Memory Machine}

Let us examine in detail how to implement matrix multiplication to minimize the
number of memory moves. 
Suppose we have two levels of memory hierarchy, fast and slow,
where the slow memory is large enough to contain the $n \times n$ matrices
$A$, $B$ and $C$, but the fast memory contains only $M$ words where
$n < M \ll n^2$. Further assume the data are reused optimally (which may
be optimistic if the decisions are made automatically by hardware).

The simplest algorithm one might try consists of three nested loops:


\begin{algorithm}\label{alg_UnblockedMatMult}
Unblocked Matrix Multiplication
\begin{tabbing}
jnk \= jnk \= jnk \= jnk \= jnk \kill
\> for $i=1:n$ \\
\> \> for $j=1:n$ \\
\> \> \> for $k=1:n$ \\
\> \> \> \> $C_{ij} = C_{ij} + A_{ik} \cdot B_{kj}$
\end{tabbing}
\end{algorithm}

We count the number of references to the slow memory as follows:
$n^3$ for reading $B$ $n$ times, $n^2$ for reading $A$ one row at
a time and keeping it in fast memory until it is no longer needed, and
$2n^2$ for reading one entry of $C$ at a time, keeping it in fast
memory until it is completely computed. This comes to $n^3 + 3n^2$ for a
$q$ of about 2, which is no better than the Level 2 BLAS and far from
the maximum
possible $n/2$. If $M \ll n$, so that we cannot keep a full row of $A$ in
fast memory, $q$ further decreases to 1, since the algorithm reduces
to a sequence of inner products, which are Level 1 BLAS. For every
permutation of the three loops on $i$, $j$ and $k$, one gets another
algorithm with $q$ about the same.

The next possibility is dividing $B$ and $C$ into {\em column blocks},
and computing $C$ block by block. Recall that we use the notation 
$B(i:j,k:l)$ to mean the submatrix in rows $i$ through $j$ and columns 
$k$ through $l$.  We partition
$B=[B^{(1)} , B^{(2)} , ... , B^{(N)}]$ where each $B^{(i)}$
is $n \times n/N$,
and similarly for $C$. Our column block algorithm is then

\vspace*{.4in}

\begin{algorithm}\label{alg_ColumnBlockedMatMult}
Column-blocked Matrix Multiplication
\begin{tabbing}
jnk \= jnk \= jnk \= jnk \= jnk \kill
\> for $j=1:N$ \\
\> \> for $k=1:n$ \\
\> \> \> $C^{(j)} = C^{(j)} + A(1:n,k) \cdot B^{(j)}(k,1:n/N)$
\end{tabbing}
\end{algorithm}

Assuming $M \ge  2n^2/N + n$, so that fast
memory can accommodate $B^{(j)}$, $C^{(j)}$ and one column of $A$
simultaneously, our memory reference count is as follows:
$2n^2$ for reading and writing each block
of $C$ once, $n^2$ for reading each block of $B$ once, and $Nn^2$ for
reading $A$ $N$ times. This yields $q \approx M/n$, so that $M$ needs to
grow with $n$ to keep $q$ large.

Finally, we consider {\em rectangular blocking}, where $A$ is broken into
an $N \times N$ block matrix with $n/N \, \times \, n/N$ blocks $A^{(ij)}$,
and $B$ and $C$ are similarly partitioned. The algorithm becomes


\begin{algorithm}\label{alg_RectangularBlockedMatMult}
Rectangular-blocked Matrix Multiplication
\begin{tabbing}
jnk \= jnk \= jnk \= jnk \= jnk \kill
\> for $i=1:N$ \\
\> \> for $j=1:N$ \\
\> \> \> for $k=1,N$ \\
\> \> \> \> $C^{(ij)} = C^{(ij)} + A^{(ik)} \cdot B^{(kj)}$
\end{tabbing}
\end{algorithm}

Assuming $M \ge 3(n/N)^2$ so that one block each from $A$, $B$ and $C$ fit
in memory simultaneously, our memory reference count is as follows:
$2n^2$ for reading and writing each block of $C$ once, $Nn^2$ for reading
$A$ $N$ times, and $Nn^2$ for reading $B$ $N$ times. This yields
$q \approx \sqrt{M/3}$, which is much better than the previous algorithms.
In \cite{hongkung} an analysis of this problem leads to an upper bound
for $q$ near $\sqrt{M}$, so we cannot expect to improve much on this
algorithm for square matrices. 

{\bf Exercise:} Confirm the values of $q$ computed for the above three
algorithms. $\Box$

On the other hand,
this brief analysis ignores a number of practical issues:
\begin{enumerate}
\item high level language constructs do not yet support block layout of
matrices as described here,
\item if the fast memory consists of vector registers and has vector
operations supporting {\tt saxpy} but not inner products, a column blocked
code
may be superior;
\item a real code will have to deal with nonsquare matrices, for which
the optimal block sizes may not be square.
\end{enumerate}
This is why computer manufacturers are often in the best position to write
the matrix-multiplication (and other BLAS) for their machines, because they
often have the best understanding of these machine specific 
details.\footnote{The matrix-multiplication subroutine in the CM-2
Scientific Subroutine Library took approximately 10 person-years of
effort to write \cite{johnsson-privatecomm}.}

Another quite different algorithm is Strassen's method \cite{horowitzsahni}, 
which multiplies
matrices recursively by dividing them into $2 \times 2$ block matrices, and
multiplying the subblocks using 7 matrix multiplications (recursively)
and 15 matrix
additions of half the size. This leads to an asymptotic complexity of
$n^{\log_2 7} \approx n^{2.81}$ instead of $n^3$. The value of this algorithm
is not just this asymptotic complexity but its reduction of the problem to
smaller subproblems which eventually fit in fast memory; once the
subproblems fit
in fast memory standard matrix multiplication may be used. This approach
has led to speedups on relatively large matrices on some machines
\cite{baileyleesimon}.
A drawback is the need for significant
workspace,
and somewhat lower numerical stability, although it is adequate for many
purposes \cite{demmelhigham91}.

{\bf Exercise:} Consider the basic linear algebra subroutine which
solves $TX=B$ for $X$, where $T$ is a given $n$-by-$n$ triangular matrix, 
$B$ is a given $n$-by-$m$ matrix, and $X$ is an $n$-by-$m$ matrix of 
unknowns. Give blocked implementations for this subroutine analogous to
the ones for matrix-multiplication above, and compare their 
ratios $q$ of flops to memory references. How do your answers depend on
$m$? $\Box$

\subsection{Matrix Multiplication on a Distributed Memory Machine}

In this section it will be convenient to number matrix entries (or subblocks)
and processors from $0$ to $n-1$ instead of $1$ to $n$.

On distributed memory machines the cost model is more complicated than on
shared memory machines, because we will need to worry about the
{\em data layout}, or how the matrices are
partitioned across the machine. This will determine both the amount
of parallelism and the cost of communication. Recall from chapter
[@Computer Architecture] that the cost of sending a message of $k$ words 
from one processor to another is $\alpha + k \beta$, where $\alpha$
is the {\em start-up cost} or {\em latency}, and $\beta$ is the per-word cost,
or reciprocal of {\em bandwidth}. Therefore to assess the cost of an algorithm
we need to count the number of floating point operations, the number of
messages sent (at a cost of $\alpha$ per message), and the total length
of messages sent (at a cost of $\beta$ per word).
We begin by showing how best to implement matrix multiplication without
regard to the layout's suitability for other matrix operations, and
return to the question of layouts in the next section.

The algorithm is due to Cannon \cite{cannon} and is well suited for computers
laid out in a square $N \times N$ mesh, i.e.\ where each processor communicates
most efficiently with the four other processors immediately north,
east, south and west of itself. We also assume the processors
at the edges of the grid are directly connected to the processors on
the opposite edge; this makes the topology that of a two-dimensional
torus. Let $A$ be partitioned into square subblocks
$A^{(ij)}$ as before, with $A^{(ij)}$ stored on processor $(i,j)$. Let
$B$ and $C$ be partitioned similarly. The algorithm is given below. It
is easily seen that whenever $A^{(ik)}$ and $B^{(kj)}$ `meet' in
processor $i,j$, they are multiplied and accumulated in $C^{(ij)}$;
the products for the different $C^{(ij)}$ are accumulated in different
orders.

%new  I made some spacing changes in this algorithm
\begin{algorithm}\label{alg_Cannon}
Cannon's matrix multiplication algorithm
\begin{tabbing}
jnk \= jnk \= jnk \= jnk \= jnk \kill
\> forall $i=0:N-1$ \\
\> \> Left circular shift row $i$ by $i$, so that $A^{(i,j)}$
\\
\> \> \> is
assigned $A^{(i,(j+i) \bmod N)}$. \\
\> forall $j=0:N-1$ \\
\> \> Upward circular shift column $j$ by $j$, so that $B^{(i,j)}$
\\
\> \> \> is
assigned $B^{((j+i) \bmod N , j)}$. \\
\> for $k=1:N$ \\
\> \> forall $i=0:N-1$, forall $j=0:N-1$ \\
\> \> \> $C^{(ij)} = C^{(ij)} + A^{(ij)} \cdot B^{(ij)}$ \\
\> \> \> Left circular shift each row of $A$ by 1, so
$A^{(i,j)}$
\\
\> \> \> \> is assigned $A^{(i,(j+1) \bmod N)}$. \\
\> \> \> Upward circular shift each column of $B$ by 1, so
$B^{(i,j)}$
\\
\> \> \> \> is assigned $B^{((i+1) \bmod N,j)}$.
\end{tabbing}
\end{algorithm}
% number of steps for skewing: N-1 * size_size^2
% number of steps for main loop: N-1 * size_size^2
Figure~\ref{figCannon} illustrates the functioning of this algorithm
for $N=3$.
\begin{figure}
\begin{center}
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|}
\multicolumn{11}{c}{} \\
\cline{1-3} \cline{5-7} \cline{9-11}
$A^{(00)}$ & $A^{(01)}$ & $A^{(02)}$ & &
$A^{(01)}$ & $A^{(02)}$ & $A^{(00)}$ & &
$A^{(02)}$ & $A^{(00)}$ & $A^{(01)}$ \rule{0pt}{13pt}\\
\cline{1-3} \cline{5-7} \cline{9-11}
$A^{(11)}$ & $A^{(12)}$ & $A^{(10)}$ & &
$A^{(12)}$ & $A^{(10)}$ & $A^{(11)}$ & &
$A^{(10)}$ & $A^{(11)}$ & $A^{(12)}$ \rule{0pt}{13pt}\\
\cline{1-3} \cline{5-7} \cline{9-11}
$A^{(22)}$ & $A^{(20)}$ & $A^{(21)}$ & &
$A^{(20)}$ & $A^{(21)}$ & $A^{(22)}$ & &
$A^{(21)}$ & $A^{(22)}$ & $A^{(20)}$ \rule{0pt}{13pt}\\
\cline{1-3} \cline{5-7} \cline{9-11}
\multicolumn{11}{c}{} \\
\cline{1-3} \cline{5-7} \cline{9-11}
$B^{(00)}$ & $B^{(11)}$ & $B^{(22)}$ & &
$B^{(10)}$ & $B^{(21)}$ & $B^{(02)}$ & &
$B^{(20)}$ & $B^{(01)}$ & $B^{(12)}$ \rule{0pt}{13pt}\\
\cline{1-3} \cline{5-7} \cline{9-11}
$B^{(10)}$ & $B^{(21)}$ & $B^{(02)}$ & &
$B^{(20)}$ & $B^{(01)}$ & $B^{(12)}$ & &
$B^{(00)}$ & $B^{(11)}$ & $B^{(22)}$ \rule{0pt}{13pt}\\
\cline{1-3} \cline{5-7} \cline{9-11}
$B^{(20)}$ & $B^{(01)}$ & $B^{(12)}$ & &
$B^{(00)}$ & $B^{(11)}$ & $B^{(22)}$ & &
$B^{(10)}$ & $B^{(21)}$ & $B^{(02)}$ \rule{0pt}{13pt}\\
\cline{1-3} \cline{5-7} \cline{9-11}
\multicolumn{11}{c}{} \\
\multicolumn{3}{c}{$A$, $B$ after skewing} & \multicolumn{1}{c}{} &
\multicolumn{3}{c}{$A$, $B$ after shift $k=1$} & \multicolumn{1}{c}{} &
\multicolumn{3}{c}{$A$, $B$ after shift $k=2$}
\end{tabular}
\caption{Cannon's algorithm for $N=3$.}\label{figCannon}
\end{center}
\end{figure}

The time spent by this algorithm is computed as follows. Assume we multiply
$n$-by-$n$ matrices on an $N$-by-$N$ processor mesh, where for simplicity
$N$ divides $n$ evenly. Assuming messages are sent between nearest neighbors
only, and that a processor can only send a single message at a time, the total 
number of messages sent (in parallel) is $4N-2$, the total number of words
sent (in parallel) is $(n/N)^2 \cdot (4N-2)$, and the total number of
flops (in parallel) is $2n^3/N^2$.

{\bf Exercise:} Implement Cannon's algorithm on a serial machine in your
favorite programming language and confirm that it works. $\Box$

{\bf Exercise:} Confirm this timing analysis for Cannon's method. $\Box$

{\bf Exercise:} Show how to reduce the number of messages sent in the
first two loops of the algorithm by sometimes shifting right (down)
instead of left (up). How does this reduce the overall cost? $\Box$

{\bf Exercise:} When Cannon's algorithm completes, $A$ and $B$ are not
in their original storage locations. Write an algorithm to return
$A$ and $B$ to their original positions. What is the running time of
your algorithm? $\Box$

A variation of this algorithm suitable for machines that are
efficient at {\em spreading\/} subblocks across rows (or down columns)
is to do this instead of the preshifting and rotation of $A$ (or $B$)
\cite{foxetal}.

Cannon's algorithm may also be easily adapted to a hypercube \cite{demmelheathvorst}.
The simplest way is to embed a grid (or two-dimensional torus) in a hypercube, 
i.e.\ map the processors in a grid to the processors in a hypercube, 
and the connections in a grid to a subset of the connections in a hypercube
\cite{ho,johnsson}. This approach (which is useful for more than matrix 
multiplication) uses only a subset of the connections in a hypercube, which 
makes the communication slower than it need be. Several
sophisticated improvements on this basic algorithm have been developed
\cite{dekel,johnssonho}, the latter of which fully utilizes the
available bandwidth of the hypercube to reduce the number of messages
sent by a factor of 2 and the number of words sent by a factor of nearly
$2n/N$.  This assumes each processor in the hypercube can send messages
to all its neighbors simultaneously, as was the case on the CM-2.
If the architecture permits us to overlap communication and computation,
we can save up to another factor of two in speed.

In this section
we have shown one can optimize matrix multiplication in a series of
steps tuning it
ever more highly for a particular computer architecture, until
essentially every communication
link and floating point unit is utilized. The algorithms are scalable,
in that they
continue to run efficiently on larger machines and larger problems,
with communication
costs becoming ever smaller with respect to computation.
On the other hand, let us ask what we lose by optimizing
so heavily for one architecture.
Our high performance depends on
the matrices having just the right dimensions, being laid
out just right in memory,
and leaving them in a scrambled final position at the end 
(although a modest amount of extra communication
could repair this).
It is unreasonable to expect users, who want to do several
computations of which this is but one, to satisfy all these requirements.
Therefore a practical
algorithm will have to deal with many irregularities, and be quite
complicated.
Our ability to do this extreme optimization is limited to a few simple and
regular problems like matrix multiplication on a hypercube,
as well as other heavily used
kernels like the BLAS, which have indeed been
highly optimized for many architectures.
We do not expect equal success
for more complicated algorithms on all architectures of
interest, at least within a reasonable amount of 
time.\addtocounter{footnote}{-1}\footnotemark\ 
Also, the algorithm is highly tuned to a particular
interconnection network topology, which
may require redesign for another machine. In view of this,
a number of recent machines try to
make communication time appear as independent of topology as possible,
so the user sees
essentially a completely connected topology.

\section{Data layouts on distributed memory machines}\label{sec_DistLayout}

Choosing a data layout may be described as choosing
a mapping $f(i,j)$ from location
$(i,j)$ in a matrix to the processor on which it is stored.
As discussed previously, we
hope to design $f$ so that it permits highly parallel
implementation of a variety of
matrix algorithms, limits communication cost as much as possible,
and retains these
attractive properties as we scale to larger matrices and larger machines.
For example,
the algorithm of the previous section uses the map
$f(i,j) = ( \lfloor i/r \rfloor , \lfloor j/r \rfloor )$,
where we subscript matrices
starting at 0, number processors by their coordinates
in a grid (also starting at (0,0)),
and store an $r \times r$ submatrix on each processor,
where $r = n/N$.

There is an emerging consensus about data layouts for distributed memory
machines. This is being implemented in several programming languages
\cite{fortranD1,HPFortran},
that will be available to programmers in the near future. 
We describe these layouts here.

High Performance Fortran (HPF) \cite{HPFortran}
permits the user to define a virtual
array of processors, align actual data structures like matrices and arrays with
this virtual array (and so with respect to each other), and then to layout the
virtual processor array on an actual machine. We describe the
layout functions $f$
offered for this last step. The range of $f$ is a
rectangular array of processors
numbered from $(0,0)$ up to $(p_1 -1 , p_2 -1)$.
Then all $f$ can be parameterized
by two integer parameters $b_1$ and $b_2$ as follows:
\[
f_{b_1,b_2}(i,j) = \left(
\left\lfloor \frac{i}{b_1} \right\rfloor \bmod p_1 ,
\left\lfloor \frac{j}{b_2} \right\rfloor \bmod p_2 \right)
\]
Suppose the matrix $A$ (or virtual processor array) is $m \times n$.
Then choosing $b_2 = n$
yields a column of processors, each containing some number
of complete rows of $A$.
Choosing $b_1 = m$ yields a row of processors. Choosing $b_1 = m/p_1$ and
$b_2 = n/p_2$ yields a {\em blocked layout}, where $A$ is
broken into $b_1 \times b_2$ subblocks,
each of which resides on a single processor.
This is the simplest two-dimensional layout one could
imagine (we used it in the previous section),
and by having large subblocks stored on each processor
it makes using the BLAS on
each processor attractive.
However, for straightforward matrix algorithms that process
the matrix from left to right
(including
Gaussian elimination, {\it QR\/} decomposition, reduction to tridiagonal form,
and so on), the
leftmost processors will become idle early in the computation
and make load balance poor.
Choosing $b_1 = b_2 = 1$ is called {\em scatter mapping\/} (also known as
{\em wrapped\/} or {\em cyclic\/} or {\em interleaved mapping}),
and optimizes load balance, since the
matrix entries stored on a single processor are as nearly
as possible uniformly distributed
throughout the matrix. On the other hand, this appears to
inhibit the use of the BLAS locally in
each processor, since the data owned by a processor are
not contiguous from the point of view
of the matrix. Finally, by choosing $1 < b_1 < m/p_1$
and $1 < b_2 < n/p_2$, we get a {\em block-scatter mapping\/}
which trades off load balance and
applicability of the BLAS. These
layouts are shown in Figures~\ref{fig_LayoutBlock} through
\ref{fig_LayoutBlockScatter}
for a $16 \times 16$ matrix laid out on a $4 \times 4$ processor grid;
each array entry is labeled
by the number of the processor that stores it.
\begin{figure}
\begin{center}
{\footnotesize \tabcolsep=5.3pt
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|}
\multicolumn{16}{c}{} \\
\cline{1-16}
0,0 & 0,0 & 0,0 & 0,0 & 0,1 & 0,1 & 0,1 & 0,1 &
0,2 & 0,2 & 0,2 & 0,2 & 0,3 & 0,3 & 0,3 & 0,3 \\
\cline{1-16}
0,0 & 0,0 & 0,0 & 0,0 & 0,1 & 0,1 & 0,1 & 0,1 &
0,2 & 0,2 & 0,2 & 0,2 & 0,3 & 0,3 & 0,3 & 0,3 \\
\cline{1-16}
0,0 & 0,0 & 0,0 & 0,0 & 0,1 & 0,1 & 0,1 & 0,1 &
0,2 & 0,2 & 0,2 & 0,2 & 0,3 & 0,3 & 0,3 & 0,3 \\
\cline{1-16}
0,0 & 0,0 & 0,0 & 0,0 & 0,1 & 0,1 & 0,1 & 0,1 &
0,2 & 0,2 & 0,2 & 0,2 & 0,3 & 0,3 & 0,3 & 0,3 \\
\cline{1-16}
1,0 & 1,0 & 1,0 & 1,0 & 1,1 & 1,1 & 1,1 & 1,1 &
1,2 & 1,2 & 1,2 & 1,2 & 1,3 & 1,3 & 1,3 & 1,3 \\
\cline{1-16}
1,0 & 1,0 & 1,0 & 1,0 & 1,1 & 1,1 & 1,1 & 1,1 &
1,2 & 1,2 & 1,2 & 1,2 & 1,3 & 1,3 & 1,3 & 1,3 \\
\cline{1-16}
1,0 & 1,0 & 1,0 & 1,0 & 1,1 & 1,1 & 1,1 & 1,1 &
1,2 & 1,2 & 1,2 & 1,2 & 1,3 & 1,3 & 1,3 & 1,3 \\
\cline{1-16}
1,0 & 1,0 & 1,0 & 1,0 & 1,1 & 1,1 & 1,1 & 1,1 &
1,2 & 1,2 & 1,2 & 1,2 & 1,3 & 1,3 & 1,3 & 1,3 \\
\cline{1-16}
2,0 & 2,0 & 2,0 & 2,0 & 2,1 & 2,1 & 2,1 & 2,1 &
2,2 & 2,2 & 2,2 & 2,2 & 2,3 & 2,3 & 2,3 & 2,3 \\
\cline{1-16}
2,0 & 2,0 & 2,0 & 2,0 & 2,1 & 2,1 & 2,1 & 2,1 &
2,2 & 2,2 & 2,2 & 2,2 & 2,3 & 2,3 & 2,3 & 2,3 \\
\cline{1-16}
2,0 & 2,0 & 2,0 & 2,0 & 2,1 & 2,1 & 2,1 & 2,1 &
2,2 & 2,2 & 2,2 & 2,2 & 2,3 & 2,3 & 2,3 & 2,3 \\
\cline{1-16}
2,0 & 2,0 & 2,0 & 2,0 & 2,1 & 2,1 & 2,1 & 2,1 &
2,2 & 2,2 & 2,2 & 2,2 & 2,3 & 2,3 & 2,3 & 2,3 \\
\cline{1-16}
3,0 & 3,0 & 3,0 & 3,0 & 3,1 & 3,1 & 3,1 & 3,1 &
3,2 & 3,2 & 3,2 & 3,2 & 3,3 & 3,3 & 3,3 & 3,3 \\
\cline{1-16}
3,0 & 3,0 & 3,0 & 3,0 & 3,1 & 3,1 & 3,1 & 3,1 &
3,2 & 3,2 & 3,2 & 3,2 & 3,3 & 3,3 & 3,3 & 3,3 \\
\cline{1-16}
3,0 & 3,0 & 3,0 & 3,0 & 3,1 & 3,1 & 3,1 & 3,1 &
3,2 & 3,2 & 3,2 & 3,2 & 3,3 & 3,3 & 3,3 & 3,3 \\
\cline{1-16}
3,0 & 3,0 & 3,0 & 3,0 & 3,1 & 3,1 & 3,1 & 3,1 &
3,2 & 3,2 & 3,2 & 3,2 & 3,3 & 3,3 & 3,3 & 3,3 \\
\cline{1-16}
\end{tabular}
}
\caption{Block layout of a $16 \times 16$ matrix on a $4 \times 4$
processor grid.}\label{fig_LayoutBlock}
\end{center}
\end{figure}
\begin{figure}
\begin{center}
{\footnotesize \tabcolsep=5.3pt
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|}
\multicolumn{16}{c}{} \\
\cline{1-16}
0,0 & 0,1 & 0,2 & 0,3 & 0,0 & 0,1 & 0,2 & 0,3 &
0,0 & 0,1 & 0,2 & 0,3 & 0,0 & 0,1 & 0,2 & 0,3 \\
\cline{1-16}
1,0 & 1,1 & 1,2 & 1,3 & 1,0 & 1,1 & 1,2 & 1,3 &
1,0 & 1,1 & 1,2 & 1,3 & 1,0 & 1,1 & 1,2 & 1,3 \\
\cline{1-16}
2,0 & 2,1 & 2,2 & 2,3 & 2,0 & 2,1 & 2,2 & 2,3 &
2,0 & 2,1 & 2,2 & 2,3 & 2,0 & 2,1 & 2,2 & 2,3 \\
\cline{1-16}
3,0 & 3,1 & 3,2 & 3,3 & 3,0 & 3,1 & 3,2 & 3,3 &
3,0 & 3,1 & 3,2 & 3,3 & 3,0 & 3,1 & 3,2 & 3,3 \\
\cline{1-16}
0,0 & 0,1 & 0,2 & 0,3 & 0,0 & 0,1 & 0,2 & 0,3 &
0,0 & 0,1 & 0,2 & 0,3 & 0,0 & 0,1 & 0,2 & 0,3 \\
\cline{1-16}
1,0 & 1,1 & 1,2 & 1,3 & 1,0 & 1,1 & 1,2 & 1,3 &
1,0 & 1,1 & 1,2 & 1,3 & 1,0 & 1,1 & 1,2 & 1,3 \\
\cline{1-16}
2,0 & 2,1 & 2,2 & 2,3 & 2,0 & 2,1 & 2,2 & 2,3 &
2,0 & 2,1 & 2,2 & 2,3 & 2,0 & 2,1 & 2,2 & 2,3 \\
\cline{1-16}
3,0 & 3,1 & 3,2 & 3,3 & 3,0 & 3,1 & 3,2 & 3,3 &
3,0 & 3,1 & 3,2 & 3,3 & 3,0 & 3,1 & 3,2 & 3,3 \\
\cline{1-16}
0,0 & 0,1 & 0,2 & 0,3 & 0,0 & 0,1 & 0,2 & 0,3 &
0,0 & 0,1 & 0,2 & 0,3 & 0,0 & 0,1 & 0,2 & 0,3 \\
\cline{1-16}
1,0 & 1,1 & 1,2 & 1,3 & 1,0 & 1,1 & 1,2 & 1,3 &
1,0 & 1,1 & 1,2 & 1,3 & 1,0 & 1,1 & 1,2 & 1,3 \\
\cline{1-16}
2,0 & 2,1 & 2,2 & 2,3 & 2,0 & 2,1 & 2,2 & 2,3 &
2,0 & 2,1 & 2,2 & 2,3 & 2,0 & 2,1 & 2,2 & 2,3 \\
\cline{1-16}
3,0 & 3,1 & 3,2 & 3,3 & 3,0 & 3,1 & 3,2 & 3,3 &
3,0 & 3,1 & 3,2 & 3,3 & 3,0 & 3,1 & 3,2 & 3,3 \\
\cline{1-16}
0,0 & 0,1 & 0,2 & 0,3 & 0,0 & 0,1 & 0,2 & 0,3 &
0,0 & 0,1 & 0,2 & 0,3 & 0,0 & 0,1 & 0,2 & 0,3 \\
\cline{1-16}
1,0 & 1,1 & 1,2 & 1,3 & 1,0 & 1,1 & 1,2 & 1,3 &
1,0 & 1,1 & 1,2 & 1,3 & 1,0 & 1,1 & 1,2 & 1,3 \\
\cline{1-16}
2,0 & 2,1 & 2,2 & 2,3 & 2,0 & 2,1 & 2,2 & 2,3 &
2,0 & 2,1 & 2,2 & 2,3 & 2,0 & 2,1 & 2,2 & 2,3 \\
\cline{1-16}
3,0 & 3,1 & 3,2 & 3,3 & 3,0 & 3,1 & 3,2 & 3,3 &
3,0 & 3,1 & 3,2 & 3,3 & 3,0 & 3,1 & 3,2 & 3,3 \\
\cline{1-16}
\end{tabular}
}
\caption{Scatter layout of a $16 \times 16$ matrix on a
$4 \times 4$ processor grid.}\label{fig_LayoutScatter}
\end{center}
\end{figure}
\begin{figure}
\begin{center}
{\footnotesize \tabcolsep=5.3pt
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|}
\multicolumn{16}{c}{} \\
\cline{1-16}
0,0 & 0,0 & 0,1 & 0,1 & 0,2 & 0,2 & 0,3 & 0,3 &
0,0 & 0,0 & 0,1 & 0,1 & 0,2 & 0,2 & 0,3 & 0,3 \\
\cline{1-16}
0,0 & 0,0 & 0,1 & 0,1 & 0,2 & 0,2 & 0,3 & 0,3 &
0,0 & 0,0 & 0,1 & 0,1 & 0,2 & 0,2 & 0,3 & 0,3 \\
\cline{1-16}
1,0 & 1,0 & 1,1 & 1,1 & 1,2 & 1,2 & 1,3 & 1,3 &
1,0 & 1,0 & 1,1 & 1,1 & 1,2 & 1,2 & 1,3 & 1,3 \\
\cline{1-16}
1,0 & 1,0 & 1,1 & 1,1 & 1,2 & 1,2 & 1,3 & 1,3 &
1,0 & 1,0 & 1,1 & 1,1 & 1,2 & 1,2 & 1,3 & 1,3 \\
\cline{1-16}
2,0 & 2,0 & 2,1 & 2,1 & 2,2 & 2,2 & 2,3 & 2,3 &
2,0 & 2,0 & 2,1 & 2,1 & 2,2 & 2,2 & 2,3 & 2,3 \\
\cline{1-16}
2,0 & 2,0 & 2,1 & 2,1 & 2,2 & 2,2 & 2,3 & 2,3 &
2,0 & 2,0 & 2,1 & 2,1 & 2,2 & 2,2 & 2,3 & 2,3 \\
\cline{1-16}
3,0 & 3,0 & 3,1 & 3,1 & 3,2 & 3,2 & 3,3 & 3,3 &
3,0 & 3,0 & 3,1 & 3,1 & 3,2 & 3,2 & 3,3 & 3,3 \\
\cline{1-16}
3,0 & 3,0 & 3,1 & 3,1 & 3,2 & 3,2 & 3,3 & 3,3 &
3,0 & 3,0 & 3,1 & 3,1 & 3,2 & 3,2 & 3,3 & 3,3 \\
\cline{1-16}
0,0 & 0,0 & 0,1 & 0,1 & 0,2 & 0,2 & 0,3 & 0,3 &
0,0 & 0,0 & 0,1 & 0,1 & 0,2 & 0,2 & 0,3 & 0,3 \\
\cline{1-16}
0,0 & 0,0 & 0,1 & 0,1 & 0,2 & 0,2 & 0,3 & 0,3 &
0,0 & 0,0 & 0,1 & 0,1 & 0,2 & 0,2 & 0,3 & 0,3 \\
\cline{1-16}
1,0 & 1,0 & 1,1 & 1,1 & 1,2 & 1,2 & 1,3 & 1,3 &
1,0 & 1,0 & 1,1 & 1,1 & 1,2 & 1,2 & 1,3 & 1,3 \\
\cline{1-16}
1,0 & 1,0 & 1,1 & 1,1 & 1,2 & 1,2 & 1,3 & 1,3 &
1,0 & 1,0 & 1,1 & 1,1 & 1,2 & 1,2 & 1,3 & 1,3 \\
\cline{1-16}
2,0 & 2,0 & 2,1 & 2,1 & 2,2 & 2,2 & 2,3 & 2,3 &
2,0 & 2,0 & 2,1 & 2,1 & 2,2 & 2,2 & 2,3 & 2,3 \\
\cline{1-16}
2,0 & 2,0 & 2,1 & 2,1 & 2,2 & 2,2 & 2,3 & 2,3 &
2,0 & 2,0 & 2,1 & 2,1 & 2,2 & 2,2 & 2,3 & 2,3 \\
\cline{1-16}
3,0 & 3,0 & 3,1 & 3,1 & 3,2 & 3,2 & 3,3 & 3,3 &
3,0 & 3,0 & 3,1 & 3,1 & 3,2 & 3,2 & 3,3 & 3,3 \\
\cline{1-16}
3,0 & 3,0 & 3,1 & 3,1 & 3,2 & 3,2 & 3,3 & 3,3 &
3,0 & 3,0 & 3,1 & 3,1 & 3,2 & 3,2 & 3,3 & 3,3 \\
\cline{1-16}
\end{tabular}
}
\caption{Block scatter layout of a $16 \times 16$ matrix on a
$4 \times 4$ processor grid with $2 \times 2$ blocks.}
\label{fig_LayoutBlockScatter}
\end{center}
\end{figure}

{\bf Exercise:} Illustrate the block scattered layout of a
$50 \times 60$ matrix on a $4 \times 8$ processor grid
with $5 \times 5$ blocks. $\Box$

By being a little more flexible about the algorithms we implement,
we can mitigate the
apparent tradeoff between load balance and applicability of BLAS.
For example, the layout of $A$ in
Figure~\ref{fig_LayoutScatter} is identical to the layout in
Figure~\ref{fig_LayoutBlock}
of $P^T A P$, where $P$ is a permutation matrix.
% More generally, if $A$ is a square matrix
% laid out by scattering in both dimensions, we get the
% identical distribution as storing
% $PAP^T$ by block mapping in both dimensions, where $P$ is
% a permutation matrix.
This shows that running Cannon's algorithm from the last section
to multiply $A$ times $B$ in scatter
layout is the same as multiplying
$PAP^T$ and $PBP^T$ to get $PABP^T$, which is the desired product.
Indeed, as long as
(1) $A$ and $B$ are both distributed over a square array of
processors;
(2) the permutations of the columns of $A$ and rows of $B$ are
identical; and
(3) for all $i$ the number of columns of $A$ stored by processor column $i$
is the same
as the number of rows of $B$ stored by processor row $i$, the algorithms
of the previous section will
correctly multiply $A$ and $B$. The distribution of the product
will be determined by the
distribution of the rows of $A$ and columns of $B$.
We will see a similar phenomenon for other distributed memory
algorithms later.

{\bf Exercise:} Using your implementation of Cannon's algorithm from an
earlier exercise, confirm by running it that it multiplies matrices
correctly even when they are stored in a scattered layout. $\Box$

A different approach is to write algorithms that work independently of the
location of the data, and rely on the underlying language or run-time
system to optimize the necessary communications. This makes code easier to
write, but puts a large burden on compiler and run-time system writers
\cite{vandevelde}. Even though compiler and run-time system writers
aspire to make programming this easy, they have not yet succeeded
(at least without large performance penalties), so the computational
scientist interested in top performance must currently pay attention to 
details of data layout and communication.

\section{Gaussian Elimination on Dense Matrices}\label{sec_DenseGaussElim}

We begin with a brief review of Gaussian elimination, 
Then we show how to reorganize it for shared memory parallel machines
(section~\ref{sec_GaussElimShared}) and distributed memory parallel machines
(section~\ref{sec_GaussElimDist}).
% Finally, section~\ref{sec_ErrorBoundsForAx=b} discusses error bounds.

To solve $Ax=b$, we first use
Gaussian elimination to factor the nonsingular matrix $A$ as $PA=LU$,
where $L$ is lower
triangular, $U$ is upper triangular, and $P$ is a permutation matrix,
i.e.\ $PA$ is the same as $A$ but with its rows in a permuted order.
Then we solve
the triangular systems $Ly=Pb$ and $Ux=y$ for the solution $x$.
These last two operations are already Level 2 BLAS, so
we concentrate on factoring $PA=LU$, which has the dominant number
of floating point operations, $2n^3/3 + {\cal O}(n^2)$. 
Reordering the rows of $A$ with $P$ is called {\em pivoting}, and is
necessary for numerical stability (more on this in section~\ref{sec_ErrorBoundsForAx=b}).
We use the standard {\em partial pivoting} scheme \cite{golubvanloan}:
this means
$L$ has ones on its diagonal and other entries bounded in absolute value by one.
The simplest version of Gaussian elimination algorithm involves
adding multiples of one row of $A$ to others to zero out subdiagonal entries,
and overwriting $A$ with $L$ and $U$:

\begin{algorithm}\label{alg_RowOrientedGaussElim_kij}
Row oriented Gaussian elimination ($kij$-{\it LU\/}
decomposition)
\begin{tabbing}
jnk \= jnk \= jnk \= jnk \= jnk \kill
\>  for $k=1:n-1$ \\
\> \> \{ Pivot by choosing $l$ so $|A(l,k)| = \max_{k \le i \le n} |A(i,k)|$, \\
\> \> \> and swapping rows $l$ and $k$ of $A$ \} \\
\> \> \{ Exit if $A(k,k) = 0$ \} \\
\> \> for $i=k+1:n$ \\
\> \> \> $A(i,k) = A(i,k) / A(k,k)$ \\
\> \> for $i=k+1:n$ \\
\> \> \> for $j=k+1:n$ \\
\> \> \> \> $A(i,j) = A(i,j) - A(i,k) \cdot A(k,j)$
\end{tabbing}
\end{algorithm}

When this algorithm terminates, the diagonal and upper triangle of $A$ contain $U$,
the part below the diagonal contains the corresponding part of $L$ (the diagonal of
$L$ is all ones, and so does not need to be stored. The permutation matrix $P$
is determined implicitly by the permutations in the second line of the algorithm.

If $A$ is stored by column, as is the case in Fortran (but not C), then since 
the inner loop combines rows of $A$, it
accesses memory locations (at least) $n$ locations apart. As described in
Chapter [@Computer Architecture], this does not respect locality, and
so is likely to be slower than an algorithm which accesses $A$ by
column (i.e.\ accesses consecutive memory locations).
Algorithm~\ref{alg_RowOrientedGaussElim_kij} is
also called $kij$-{\it LU\/} decomposition, because of the 
nesting order of its 
loops. All the rest of the $3!$ permutations of $i$, $j$ and $k$ lead to
valid algorithms, some of which access columns of $A$ in the innermost loop.
The next algorithm is one of these,
and is used in the LINPACK routine {\tt sgefa} \cite{linpack}.

\begin{algorithm}\label{alg_ColumnOrientedGaussElim_kji}
Column oriented Gaussian elimination ($kji$-{\it LU\/}
decomposition)
\begin{tabbing}
jnk \= jnk \= jnk \= jnk \= jnk \kill
\>  for $k=1:n-1$ \\
\> \> \{ Pivot by choosing $l$ so $|A(l,k)| = \max_{k \le i \le n} |A(i,k)|$, \\
\> \> \> and swapping $A(l,k)$ and $A(k,k)$ \} \\
\> \> \{ Exit if $A(k,k) = 0$ \} \\
\> \> for $i=k+1:n$ \\
\> \> \> $A(i,k) = A(i,k) / A(k,k)$ \\
\> \> for $j=k+1:n$ \\
\> \> \> \{ Pivot by swapping $A(l,j)$ and $A(k,j)$ \} \\
\> \> \> for $i=k+1:n$ \\
\> \> \> \> $A(i,j) = A(i,j) - A(i,k) \cdot A(k,j)$
\end{tabbing}
\end{algorithm}

\subsection{Gaussian Elimination on Shared Memory Machines}\label{sec_GaussElimShared}

The innermost loop of Algorithm~\ref{alg_ColumnOrientedGaussElim_kji}
can be performed by a single call to
the Level 1 BLAS operation {\tt saxpy}; this in done in LINPACK.
To achieve higher performance, we
modify this code first to use the Level 2 and then the Level 3 BLAS in
its innermost loops. Again, 3! versions of these algorithms are possible,
but we just describe the ones used in the LAPACK library \cite{lapackmanual}.
There is obvious parallelism in the innermost loop, since each
$A_{ij}$ can be updated independently.
To make the use of BLAS clear, we use Fortran 90 (or Matlab) notation:

\begin{algorithm}\label{alg_GaussElimBLAS2}
Gaussian elimination using Level 2 BLAS
\begin{tabbing}
jnk \= jnk \= jnk \= jnk \= jnk \kill
\>  for $k=1:n-1$ \\
\> \> \{ Pivot by choosing $l$ so $|A_{lk}| = \max_{k \le i \le n} |A_{ik}|$, \\
\> \> \> and swapping rows $l$ and $k$ of $A$ \} \\
\> \> \{ Exit if $A(k,k) = 0$ \} \\
\> \> $A(k+1:n,k) = A(k+1:n,k) / A_{kk}$ \\
\> \> $A(k+1:n,k+1:n) = A(k+1:n,k+1:n)$ \\
\> \> \> $- A(k+1:n,k) \cdot A(k,k+1:n)$ \\
\end{tabbing}
\end{algorithm}

The parallelism is evident:
most work is performed is a single rank-1 update of the
trailing $n-k \, \times \, n-k$ submatrix $A(k+1:n,k+1:n)$,
where each entry of $A(k+1:n,k+1:n)$ can be updated in parallel.
Other permutations of the
nested loops lead to different algorithms, which depend on the BLAS for
matrix--vector multiplication and solving a triangular system instead of
rank-1 updating \cite{lapwn19,robert};
which is faster depends on the relative speed of these on a given machine.

To convert to the Level 3 BLAS involves column blocking
\[
A=[A^{(1)}, \ldots , A^{(m)}]
\]
into $n \times n_b$ blocks, where
$n_b$ is the {\em block size\/} and $m=n/n_b$. The optimal choice of $n_b$
depends on the
memory hierarchy of the machine in question: our approach is to compute
the {\it LU\/} decomposition
of each $n \times n_b$ subblock of $A$ using Algorithm~\ref{alg_GaussElimBLAS2}
in the fast memory, and then use
Level 3 BLAS to update the rest of the matrix:

\begin{algorithm}\label{alg_GaussElimBLAS3}
Gaussian elimination using Level 3 BLAS (we assume $n_b$ divides $n$)
\begin{tabbing}
jnk \= jnk \= jnk \= jnk \= jnk \kill
\>  for $l=1:m$  \\
\> \> $k= (l-1) \cdot n_b + 1$ \\
\> \> Use Algorithm~\ref{alg_GaussElimBLAS2} to factorize $PA^{(l)} = LU$ in place \\
\> \> Apply $P$ to prior columns $A(1:n,1:k-1)$ and later columns
\\
\> \> \> $A(1:n,k+n_b:n)$ \\
\> \> Update block row of $U$: \\
\> \> \> Replace $A(k:k+n_b -1,k+n_b:n)$ by the solution $X$ of \\
\> \> \> $TX=A(k:k+n_b -1,k+n_b:n)$, where $T$ is the lower \\
\> \> \> triangular matrix in $A(k:k+n_b-1,k:k+n_b-1)$ \\
\> \> $A(k+n_b:n,k+n_b:n) = A(k+n_b:n,k+n_b:n) - $ \\
\> \> \> $A(k+n_b:n,k:k+n_b-1) \cdot A(k:k+n_b-1,k+n_b:n)$
\end{tabbing}
\end{algorithm}

Most of the work is performed in the last two lines, solving a triangular
system with
many right-hand sides, and matrix multiplication. Other similar algorithms
may be derived
by conformally partitioning $L$, $U$ and $A$, and equating partitions
in $A=LU$.
Algorithms~\ref{alg_GaussElimBLAS2} and \ref{alg_GaussElimBLAS3} are available as, 
respectively, subroutines {\tt sgetf2} and
{\tt sgetrf} in LAPACK \cite{lapackmanual}.

{\bf Exercise:} The {\em Cholesky factorization} of a symmetric positive
definite matrix $A$ is $A=LL^T$, where $L$ is a lower triangular matrix.
The algorithm is very similar to Gaussian elimination, but the special
properties of $A$ mean only half as much storage and half as many flops
are needed as for standard Gaussian elimination. Here is the analog of
Algorithm~\ref{alg_RowOrientedGaussElim_kij} for Cholesky:

\begin{algorithm}\label{alg_RowOrientedCholesky_kij}
Row oriented Cholesky (access lower triangle of $A$ only) 
% ($kij$-{\it LU\/} decomposition)
\begin{tabbing}
jnk \= jnk \= jnk \= jnk \= jnk \kill
\>  for $k=1:n$ \\
\> \> for $i=k:n$ \\
\> \> \> for $j=1:k-1$ \\
\> \> \> \> $A(i,k) = A(i,k) - A(i,j) \cdot A(k,j)$ \\
\> \> Exit if $A(k,k) \leq 0$ \\
\> \> $A(k,k) = \sqrt{A(k,k)}$ \\
\> \> for $i=k:n$ \\
\> \> \> $A(i,k) = A(i,k) / A(k,k)$ 
\end{tabbing}
\end{algorithm}

Note that Cholesky does not require pivoting.
Derive analogs of
Algorithms~\ref{alg_ColumnOrientedGaussElim_kji}
through
\ref{alg_GaussElimBLAS3} for Cholesky. Ideally your algorithm
should only need to read and write the lower (or upper) triangular 
part of $A$. $\Box$

The LAPACK routine implementing Cholesky is {\tt sposv}. Other matrix
structures handled by LAPACK include 
\begin{itemize}
\item {\em packed symmetric positive definite matrices}, i.e.\ stored in half the
space of a full dense matrix ({\tt sppsv})
\item {\em symmetric indefinite matrices}, 
i.e.\ symmetric but not necessarily positive definite matrices ({\tt ssysv}), 
\item {\em packed symmetric indefinite matrices}, i.e. stored in half the space of
a full dense matrix ({\tt sspsv}),
\item {\em band matrices}, i.e.\ those nonzero only within a fixed distance from the 
main diagonal ({\tt sgbsv}),
\item {\em tridiagonal matrices}, i.e.\ a special case of band matrices which are
nonzero only on the diagonal, immediately above it and immediately below it ({\tt sgtsv}),
\item {\em symmetric positive definite band matrices} ({\tt spbsv}), and
\item {\em symmetric positive definite tridiagonal matrices} ({\tt sptsv}).
\end{itemize}
There are also double precision, complex, and double precision complex counterparts of
all LAPACK routines (replace the leading 's' in the subroutine name by 'd', 'c', and
'z', respectively).

\subsection{Gaussian Elimination on Distributed Memory Machines}\label{sec_GaussElimDist}

As described earlier, data layout influences the algorithm.
We show the algorithm for
a block scatter mapping in both dimensions, and then discuss how other
layouts may be handled.
The algorithm is essentially the same as Algorithm~\ref{alg_GaussElimBLAS3}
with interprocessor communication inserted as
necessary. The block size $n_b$ equals $b_2$, which determines the
layout in the horizontal direction.

Communication is required in Algorithm~\ref{alg_GaussElimBLAS2} 
to find the pivot
entry at each step and swap rows if necessary; then each processor
can perform the
scaling and rank-1 updates independently. The pivot search is a
{\em reduction\/} operation, meaning that values from all processors
must be reduced to a single value, a pointer to the row containing
the largest pivot.
After the block column is
fully factorized,
the pivot information must be {\em broadcast\/} so other processors
can permute their own
data, as well as permute among different processors.

In Algorithm~\ref{alg_GaussElimBLAS3}, 
the $n_b \times n_b$ $L$ matrix
stored on the diagonal must be {\em spread\/} rightward to other processors
in the same
row, so they can compute their entries of $U$. Finally,
the processors holding the rest of $L$ below
the diagonal must {\em spread\/} their submatrices to the right,
and the processors holding
the new entries of $U$ just computed must {\em spread\/} their submatrices
downward, before the
final rank-$n_b$ update in the last line of Algorithm~\ref{alg_GaussElimBLAS3}
can take place.

The optimal choice of block sizes $b_1$ and $b_2$ depends on the
cost of communication
{\em versus\/} computation. For example, if the communication required to do
pivot search and
swapping of rows is expensive, $b_1$ should be large.
The execution time is a function
of dimension $n$, block sizes $b_1$ and $b_2$,
processor counts $p_1$ and $p_2$,
and the cost of computation and communication
(from Chapter [@Computer Architecture], we know how to model these).
Given this function, it
may be minimized as a function of $b_1$, $b_2$, $p_1$ and $p_2$.
Some theoretical analyses of this sort
for special cases may be found in \cite{robert} and the references
therein.
See also \cite{lapwn24} and \cite{lapwn37}.  As an example of the
performance that can be
attained in practice, on an Intel Delta with 512 processors the speed
of {\it LU\/} ranged from a little over 1 gigaflop for $n=2000$ to nearly 12
gigaflops for $n=25000$.

Even if the layout is not block scatter as described so far,
essentially the same
algorithm may be used. As described in Section~\ref{sec_DistLayout},
many possible layouts
are related by permutation matrices. So simply performing
the algorithm just described
with (optimal) block sizes $b_1$ and $b_2$ on the matrix $A$
as stored is equivalent to
performing the {\it LU\/} decomposition of $P_1 A P_2$ where $P_1$
and $P_2$ are permutation
matrices. Thus at the cost of keeping track of these permutations
(a possibly nontrivial software
issue), a single algorithm suffices for a wide variety of layouts.
% For example, on an Intel Delta with 512 processors,
% the speed of {\it LU\/} ranged from
% a little over 1 gigaflop for $n=2000$ to nearly 12 gigaflops for $n=25000$.

Finally, we need to solve the triangular systems $Ly=b$ and $Ux=y$
arising from the
{\it LU\/} decomposition. On a shared memory machine, this is accomplished by
two calls to the
Level 2 BLAS. Designing such an algorithm on a distributed
memory machine is harder,
because the fewer floating point operations performed (${\cal O}(n^2)$
instead of ${\cal O}(n^3)$) make
it harder to mask the communication
\cite{eisenstatetal,heathromine,licoleman,romineortega}.

\section{Gaussian Elimination on Band Matrices}\label{sec_BandGaussElim}

Recall that a {\em band matrix} is one which is nonzero only on certain
diagonals. We say a matrix $A$ has {\em lower bandwidth $lb$} if
$a_{ij} = 0$ for $i>j+lb$, and {\em upper bandwidth $ub$} if
$a_{ij} = 0$ for $i<j-ub$. If $lb$ and/or $ub$ are large, then the
techniques of the Section~\ref{sec_DenseGaussElim} are still applicable,
and the LAPACK routines for band matrices ({\tt sgbsv} and {\tt spbsv})
have been optimized for this situation \cite{lapwn21,foxetal}.
Different techniques are needed when the bandwidth is small.
The reason is that proportionately less and less parallel
work is available in updating the trailing submatrix, which is where
we use the Level 3 BLAS.  In the limiting case of tridiagonal
matrices, the parallel algorithm derived as in section~\ref{sec_GaussElimShared}
and the standard serial algorithm are nearly identical. 

The problem of solving banded linear systems with a narrow band has been
studied by many authors, see for instance the references in
\cite{gallivanetal1,ortega88}.
We will only sketch some of the main ideas and we will do so for
rather simple problems. The reader should keep in mind that these ideas
can easily be generalized for more complicated situations, and many
have appeared in the literature.

Most of the parallel approaches perform more arithmetic operations
than standard (sequential) Gaussian elimination (typically $2.5$
times as many),
{\em twisted factorization\/} being the only exception.
In twisted factorization the Gaussian elimination process is carried
out in parallel from both sides. This method was first proposed in
\cite{babu72}
for tridiagonal systems $Tx=b$ as a means to compute a specified component of
$x$ more accurately. For a tridiagonal matrix twisted factorization leads
to the following decomposition of $T$:
\begin{equation}
\left(\begin{array}{cccccccc}
     \star & \star &      &      &      &       &      & \\
     \star & \star & \star&      &      &       &      &  \\
           & \star & \star& \star&      &       &      &  \\
           &       &\star & \ddots&\ddots&      &      &  \\
           &       &      &\ddots &\ddots&\ddots&      &   \\
           &       &      &       & \ddots&\star& \star &    \\
           &       &      &       &       &\star& \star &\star \\
           &       &      &       &       &     & \star &\star
      \end{array}\right)
\end{equation}
\[
      = \left(\begin{array}{cccccccc}
     \star &       &      &       &     &       &        &  \\
     \star & \star &      &       &     &       &        &  \\
           & \star & \ddots&      &     &       &        &  \\
           &       &\ddots& \ddots&     &       &        &   \\
           &       &      & \star &\star& \star &        &  \\
           &       &      &       &     &\ddots & \ddots &   \\
           &       &      &       &     &       & \star  &\star \\
           &       &      &       &     &       &        &\star
      \end{array}\right)  \left(\begin{array}{cccccccc}
     \star & \star &      &      &      &       &      & \\
           & \star & \star&      &      &       &      &  \\
           &       & \star& \ddots&      &       &      &  \\
           &       &      & \ddots&\star &      &      &  \\
           &       &      &      &\star  &      &      &   \\
           &       &      &      &\star  &\ddots &      &    \\
           &       &      &      &       &\ddots & \star&   \\
           &       &      &       &       &     & \star &\star
       \end{array}\right)   ,
\]
or $T=PQ$, where we have assumed that no zero diagonal element is created
in $P$ or $Q$. Such decompositions exist if $A$ is symmetric positive
definite, or if $A$ is an $M$-matrix, or when $A$ is diagonally dominant.
The twisted factorization and subsequent forward and back substitutions with
$P$ and $Q$ take as many arithmetic operations as the standard factorization,
and
can be carried out with twofold parallelism by working from both ends of the
matrix simultaneously. For an analysis of this
process for tridiagonal systems, see \cite{vors87a}. Twisted
factorization
can be combined with any of the following techniques, often doubling the
parallelism.

The other techniques we will discuss can all be applied to general banded
systems, for which most were originally proposed, but for ease of exposition
we will illustrate them just with a lower unit bidiagonal system $Lx=b$.
A straight forward parallelization approach is to eliminate
the unknown $x_{i-1}$ from equation $i$ using
equation $i-1$, for all $i$ in parallel. This leads to a new system in which
each $x_i$ is coupled only with $x_{i-2}$. Thus, the original system splits
in two independent lower bidiagonal systems of half the size,
one for the odd-numbered
unknowns, and one for the even-numbered unknowns. This process
can be repeated recursively for both new systems,
leading to an algorithm known as {\em recursive doubling\/} \cite{stone73}.
% In Algorithm 2 (Section~\ref{secParSysExamples}) it was presented as
% a special case of parallel prefix.
It has been analyzed and generalized for banded
systems in \cite{duro77}. Its significance for modern parallel computers
is limited, which we illustrate with the following examples.

Suppose we perform a single step of recursive doubling. This
step can be done in parallel, but it involves slightly more arithmetic
than the serial elimination process for solving $Lx=b$.
The two resulting lower bidiagonal systems can be solved in parallel. This
implies that on a two-processor system the time for a single step of
recursive doubling will be slightly more than the time for solving the original
system with only one processor. If we have $n$ processors (where $n$ is
the dimension of $L$), then the elimination step can be done in very few
time steps, and the two resulting systems can be solved in parallel, so that
we have a speedup of about $2$. However, this is not very practical, since
during most of the time $n-2$ processors are idle, or formulated differently,
the efficiency of the processors is rather low.

If we use $n$ processors to apply this algorithm recursively instead of
splitting into just two systems, we can solve in ${\cal O}( \log n )$ steps, a
speedup of ${\cal O}(n/ \log n)$, but the efficiency decreases like ${\cal
O}(1/ \log n)$.
This is theoretically attractive but inefficient. Because of the data
movement required, it is unlikely to be fast without system support
for this communication pattern.

A related approach, which avoids the two subsystems, is to eliminate
only the odd-numbered unknowns $x_{i-1}$ from the even-numbered equations $i$.
Again, this can be done in parallel, or in vector mode, and it results in
a new system in which only the even-numbered unknowns are coupled. After having
solved this reduced system, the odd-numbered unknowns can be computed in
parallel
from the odd-numbered equations. Of course, the trick can be repeated for the
subsystem of half size, and this process is known as {\em cyclic reduction\/}
\cite{lavo74,hell78}.
Since the amount of serial work is halved in each step
by completely parallel (or vectorizable) operations, this approach has been
successfully applied on vector supercomputers, especially when the vector
speed of the machine is significantly greater than the scalar speed
\cite{ortega88,degr91,scvo87}.
For distributed memory computers
the method requires too much data movement for the reduced system to be
practical.

However, the method is easily generalized to one with more parallelism.
Cyclic reduction can be viewed as an approach in
which the given matrix $L$ is written as a lower block bidiagonal matrix with
$2\times 2$ blocks along the diagonal. In the elimination process all
$(2,1)$~positions in the diagonal blocks are eliminated in parallel.
An obvious idea is to subdivide the matrix into larger blocks, i.e.\ we
write $L$ as a block bidiagonal matrix with $k\times k$ blocks along the
diagonal (for simplicity we assume that $n$ is a multiple of $k$).
In practical cases $k$ is chosen so large that the process is not repeated
for the resulting subsystems, as for cyclic reduction (where $k=2$). This
approach is referred to as a {\em divide-and-conquer\/} approach. For banded
triangular systems it was first suggested in \cite{chku78}, 
for tridiagonal systems it was proposed in \cite{wang81}.

To illustrate, let us apply one parallel elimination step to the
lower bidiagonal system $Lx=b$ to eliminate all
subdiagonal elements in all diagonal blocks.
This yields a system $\tilde{L}x=\tilde{b}$, where for $k=4$ and $n=16$
we get
\begin{equation}
\tilde{L}= \left(\begin{array}{cccccccccccccccc}
1  &   &  &      &   &   &   &   &   &   &   &  &   &   &    &   \\
   & 1 &  &      &   &   &   &   &   &   &   &  &   &   &    &   \\
   &   &1 &      &   &   &   &   &   &   &   &  &   &   &    &   \\
   &   &  & 1    &   &   &   &   &   &   &   &  &   &   &    &   \\
   &   &  &\star & 1 &   &   &   &   &   &   &  &   &   &    &   \\
   &   &  &\star &   & 1 &   &   &   &   &   &  &   &   &    &   \\
   &   &  &\star &   &   & 1 &   &   &   &   &  &   &   &    &   \\
   &   &  &\star &   &   &   & 1 &   &   &   &  &   &   &    &   \\
   &   &  &      &   &   &   &\star  & 1 &   &   &  &   &   &    &   \\
   &   &  &      &   &   &   &\star  &   & 1 &   &  &   &   &    &   \\
   &   &  &      &   &   &   &\star  &   &   & 1 &  &   &   &    &   \\
   &   &  &      &   &   &   &\star  &   &   &   &1 &   &   &    &   \\
   &   &  &      &   &   &   &       &   &   &   &\star  &1  &   &    &   \\
   &   &  &      &   &   &   &       &   &   &   &\star  &   & 1 &    &   \\
   &   &  &      &   &   &   &       &   &   &   &\star  &   &   & 1  &   \\
   &   &  &      &   &   &   &       &   &   &   &\star  &   &   &    & 1
      \end{array}\right) .
\end{equation}
There are two possibilities for the next step. In the original approach
\cite{wang81}, the fill-in in the subdiagonal blocks is eliminated in parallel,
or vector mode, for each subdiagonal block (note that each subdiagonal block
has only one column
with nonzero elements). It has been shown in \cite{vode89} that this
leads to
very efficient vectorized code for machines such as Cray, Fujitsu, etc.

For parallel computers, the parallelism in eliminating these subdiagonal
blocks
is relatively fine-grained compared with the more coarse-grained parallelism
in the first step of the algorithm. Furthermore, on distributed memory
machines the data for each subdiagonal block has to be spread over all
processors. In \cite{mivo88} it has been shown that this limits the
performance
of the algorithm, the speedup being bounded by the ratio of computational
speed and communication speed. This ratio is often very low \cite{mivo88}.

The other approach is first to eliminate successively the last nonzero elements
in the subdiagonal blocks $\tilde{L}_{j,j-1}$. This can be done with a short
recurrence of length $n/k-1$, after which all fill-in can be eliminated in
parallel. For the recurrence we need some data communication between
processors.
However, for $k$ large enough with respect to $n/k$, one can attain speedups
close to $2k/5$ for this algorithm on a $k$ processor system 
\cite{vors89c}.
For a generalization of the divide-and-conquer approach for banded systems,
see \cite{meie85}; the data transport aspects for distributed memory
machines
have been discussed in \cite{mivo88}.

There are other variants of the divide-and-conquer approach that move
the fill-in into other columns of the subblocks or are
more stable numerically. For example,
in \cite{mehr91} the matrix is split into a block diagonal
matrix and a remainder via rank-1 updates.

\section{Gaussian Elimination on Sparse Matrices}\label{sec_SparseGaussElim}

% \section{Direct methods for sparse linear systems}\label{secMike}

% From heath@ncsa.uiuc.edu Mon Sep 28 11:56:50 1992
% Jim, here is my revised section on sparse direct methods.  It supercedes
% the version I sent you last week.  It still has the bibliography
% hard-wired in rather than coming from a bib file.  I'll try to get the
% latter to you right away.  Also, I'll remind you to let me know if you
% need pictex in order to process the first figure in my section.
% Finally, I never really asked you point blank in Daytona, but I assume
% from your lack of complaints that you found my section to be suitable
% for the purpose.  Please let me know if there is any problem with it.
% Thanks.
% -Mike
% \documentstyle[11pt]{article}
% \input{prepictex.tex}
% From heath@ncsa.uiuc.edu Tue Sep 29 07:18:45 1992
% This is prepictex.tex  Version 1.1  9/10/87
% To use the PiCTeX macros under LaTeX, you first need to \input this
% file, then the main corpus of PiCTeX macros (pictex.tex), and then
% the file postpictex.tex. Do not \input the file latexpicobjs.tex.
\catcode`@=11 \catcode`!=11
% First of all, see if  \fiverm  is defined. If so do nothing;
% if not, let  \fiverm  take on meaning of LaTeX's  \fivrm.
\expandafter\ifx\csname fiverm\endcsname\relax
  \let\fiverm\fivrm
\fi

% Save meanings of LaTeX keywords that duplicate PiCTeX keywords
\let\!latexendpicture=\endpicture
\let\!latexframe=\frame
\let\!latexlinethickness=\linethickness
\let\!latexmultiput=\multiput
\let\!latexput=\put

% Redefine the LaTeX \@picture macro
\def\@picture(#1,#2)(#3,#4){%
  \@picht #2\unitlength
  \setbox\@picbox\hbox to #1\unitlength\bgroup
  \let\endpicture=\!latexendpicture
  \let\frame=\!latexframe
  \let\linethickness=\!latexlinethickness
  \let\multiput=\!latexmultiput
  \let\put=\!latexput
  \hskip -#3\unitlength \lower #4\unitlength \hbox\bgroup}
\catcode`@=12 \catcode`!=12
% \input{pictex.tex}
% From heath@ncsa.uiuc.edu Tue Sep 29 07:19:41 1992
% This is PiCTeX, Version 1.1   9/21/87

% CAVEAT: The PiCTeX manual often has a more lucid explanation
%   of any given topic than you will find in the internal documentation
%   of the macros.

% PiCTeX's commands can be classified into two groups: (1) public (or
%   external), and (2) private (or internal). The public macros are
%   discussed at length in the manual. The only discussion of the private
%   macros is the internal documentation. The private macros all have
%   names beginning with an exclamation point (!) of category code 11.
%   Since in normal usage "!" has category code 12, these macros can't
%   be accessed or modified by the general user.

% The macros are organized into thematically related groups. For example,
%   the macros dealing with dots & dashes are all in the DASHPATTERN group.
%   The table below shows which macros are in which groups. The table
%   covers all public macros, and many (but not all) of PiCTeX's upper level
%   private macros. Following the table, the various groups are listed
%   in the order in which they appear in the table.

% *********************** TABLE OF GROUPS OF MACROS **********************

% HACKS:  Utility macros
%    \PiC
%    \PiCTeX
%    \placevalueinpts
%    \!!loop
%    \!cfor
%    \!copylist
%    \!ecfor
%    \!etfor
%    \!getnext
%    \!getnextvalueof
%    \!ifempty
%    \!ifnextchar
%    \!leftappend
%    \!listaddon
%    \!loop
%    \!lop
%    \!mlap
%    \!not
%    \!removept
%    \!rightappend
%    \!tfor
%    \!vmlap
%    \!wlet

% ALLOCATION:  Allocates registers

% AREAS: Deals with plot areas
%    \axis
%    \grid
%    \invisibleaxes
%    \normalgraphs
%    \plotheading
%    \setplotarea
%    \visibleaxes

% ARROWS:  Draws arrows
%    \arrow
%    \betweenarrows

% BARS:  Draws bars
%    \putbar
%    \setbars

% BOXES:  Draws rectangles
%    \frame
%    \putrectangle
%    \rectangle
%    \shaderectangleson
%    \shaderectanglesoff

% CURVES:  Upper level plot commands
%    \hshade
%    \plot
%    \sethistograms
%    \setlinear
%    \setquadratic
%    \vshade

% DASHPATTERNS:  Sets up dash patterns
%    \findlength
%    \setdashes
%    \setdashesnear
%    \setdashpattern
%    \setdots
%    \setdotsnear
%    \setsolid
%    \!dashingoff
%    \!dashingon

% DIVISION:  Does long division of dimension registers
%    \Divide
%    \!divide

% ELLIPSES:  Draws ellipses and circles
%    \circulararc
%    \ellipticalarc

% RULES:  Draws rules, i.e., horizontal & vertical lines
%    \putrule
%    \!putdashedhline
%    \!putdashedvline
%    \!puthline
%    \!putsolidhline
%    \!putsolidvline
%    \!putvline

% LINEAR ARC:  Draws straight lines -- solid and dashed
%    \inboundscheckoff
%    \inboundscheckon
%    \!advancedashing
%    \!drawlinearsegment
%    \!initinboundscheck
%    \!linearsolid
%    \!lineardashed
%    \!ljoin
%    \!plotifinbounds
%    \!start

% LOGTEN:  Log_10 function
%    \!logten

% PICTURES:  Basic setups for PiCtures; \put commands
%    \accountingoff
%    \accountingon
%    \beginpicture
%    \endpicture
%    \endpicturesave
%    \lines
%    \multiput
%    \put
%    \setcoordinatemode
%    \setcoordinatesystem
%    \setdimensionmode
%    \stack
%    \Lines
%    \Xdistance
%    \Ydistance
%    \!dimenput
%    \!ifcoordmode
%    \!ifdimenmode
%    \!setcoordmode
%    \!setdimenmode
%    \!setputobject

% PLOTTING:  Things to do with plotting
%    \dontsavelinesandcurves
%    \replot
%    \savelinesandcurves
%    \setplotsymbol
%    \writesavefile
%    \!plot

% PYTHAGORAS:  Euclidean distance function
%    \placehypotenuse
%    \!Pythag

% QUADRATIC ARC:  Draws a quadratic arc
%    \!qjoin

% ROTATIONS:  Handles rotations
%    \startrotation
%    \stoprotation
%    \!rotateaboutpivot
%    \!rotateonly

% SHADING:  Handles shading
%    \setshadegrid
%    \setshadesymbol
%    \!lshade
%    \!qshade
%    \!starthshade
%    \!startvshade

% TICKS:  Draws ticks on graphs
%    \gridlines
%    \loggedticks
%    \nogridlines
%    \ticksin
%    \ticksout
%    \unloggesticks

% ***************** END OF TABLE OF GROUPS OF MACROS ********************

\catcode`!=11 %  ***** THIS MUST NEVER BE OMITTED
% *******************************
% *** HACKS  (Utility macros) ***
% *******************************

% ** User commands
% **   \PiC{P\kern-.12em\lower.5ex\hbox{I}\kern-.075emC}
% **   \PiCTeX{\PiC\kern-.11em\TeX}
% **   \placevalueinpts of <DIMENSION REGISTER> in {CONTROL SEQUENCE}

% ** Internal commands
% **   \!ifnextchar{CHARACTER}{TRUE ACTION}{FALSE ACTION}
% **   \!tfor NAME := LIST \do {BODY}
% **   \!etfor NAME:= LIST \do {BODY}
% **   \!cfor NAME := LIST \do {BODY}
% **   \!ecfor NAME:= LIST \do {BODY}
% **   \!ifempty{MACRO}{TRUE ACTION}{FALSE ACTION}
% **   \!getnext\\ITEMfrom\LIST
% **   \!getnextvalueof\DIMEN\from\LIST
% **   \!copylist\LISTMACRO_A\to\LISTMACRO_B
% **   \!wlet\CONTROL_SEQUENCE_A=\CONTROL_SEQUENCE_B
% **   \!listaddon ITEM LIST
% **   \!rightappendITEM\withCS\to\LISTMACRO
% **   \!leftappendITEM\withCS\to\LISTMACRO
% **   \!lop\LISTMACRO\to\ITEM
% **   \!loop ... repeat
% **   \!!loop ... repeat
% **   \!removept{DIMENSION REGISTER}{CONTROL SEQUENCE}
% **   \!mlap{...}
% **   \!vmlap{...}
% **   \!not{TEK if-CONDITION}
% ** First, here are the the PiCTeX logo, and the syllable PiC:
\def\PiC{P\kern-.12em\lower.5ex\hbox{I}\kern-.075emC}
\def\PiCTeX{\PiC\kern-.11em\TeX}
% ** The following macro expands to parameter #2 or parameter #3 according to
% ** whether the next non-blank character following the macro is or is not #1.
% ** Blanks following the macro are gobbled.
\def\!ifnextchar#1#2#3{%
  \let\!testchar=#1%
  \def\!first{#2}%
  \def\!second{#3}%
  \futurelet\!nextchar\!testnext}
\def\!testnext{%
  \ifx \!nextchar \!spacetoken
    \let\!next=\!skipspacetestagain
  \else
    \ifx \!nextchar \!testchar
      \let\!next=\!first
    \else
      \let\!next=\!second
    \fi
  \fi
  \!next}
\def\\{\!skipspacetestagain}
  \expandafter\def\\ {\futurelet\!nextchar\!testnext}
\def\\{\let\!spacetoken= } \\  %  ** set \spacetoken to a space token


% ** Borrow the "tfor" macro from Latex:
% **   \!tfor NAME := LIST \do {BODY}
% **   if, before expansion, LIST = T1 ... Tn,  where each  Ti  is a token
% **   or  {...},  then executes  BODY  n  times, with  NAME = Ti  on the
% **   i-th iteration.  Works for  n=0.
\def\!tfor#1:=#2\do#3{%
  \edef\!fortemp{#2}%
  \ifx\!fortemp\!empty
    \else
    \!tforloop#2\!nil\!nil\!!#1{#3}%
  \fi}
\def\!tforloop#1#2\!!#3#4{%
  \def#3{#1}%
  \ifx #3\!nnil
    \let\!nextwhile=\!fornoop
  \else
    #4\relax
    \let\!nextwhile=\!tforloop
  \fi
  \!nextwhile#2\!!#3{#4}}


% **   \!etfor NAME:= LIST\do {BODY}
% **   This is like \!cfor, but LIST is any balanced token list whose complete
% **     expansion has the form  T1 ... Tn
\def\!etfor#1:=#2\do#3{%
  \def\!!tfor{\!tfor#1:=}%
  \edef\!!!tfor{#2}%
  \expandafter\!!tfor\!!!tfor\do{#3}}


% **   modify the Latex \tfor (token-for) loop to a \cfor (comma-for) loop.
% **   \!cfor NAME := LIST \do {BODY}
% **     if, before expansion, LIST = a1,a2,...an, then executes  BODY n times,
% **     with  NAME = ai  on the i-th iteration.  Works for  n=0.
\def\!cfor#1:=#2\do#3{%
  \edef\!fortemp{#2}%
  \ifx\!fortemp\!empty
  \else
    \!cforloop#2,\!nil,\!nil\!!#1{#3}%
  \fi}
\def\!cforloop#1,#2\!!#3#4{%
  \def#3{#1}%
  \ifx #3\!nnil
    \let\!nextwhile=\!fornoop
  \else
    #4\relax
    \let\!nextwhile=\!cforloop
  \fi
  \!nextwhile#2\!!#3{#4}}


% **   \!ecfor NAME:= LIST\do {BODY}
% **   This is like \!cfor, but LIST is any balanced token list whose complete
% **     expansion has the form  a1,a2,...,an.
\def\!ecfor#1:=#2\do#3{%
  \def\!!cfor{\!cfor#1:=}%
  \edef\!!!cfor{#2}%
  \expandafter\!!cfor\!!!cfor\do{#3}}


\def\!empty{}
\def\!nnil{\!nil}
\def\!fornoop#1\!!#2#3{}


% **  \!ifempty{ARG}{TRUE ACTION}{FALSE ACTION}
\def\!ifempty#1#2#3{%
  \edef\!emptyarg{#1}%
  \ifx\!emptyarg\!empty
    #2%
  \else
    #3%
  \fi}

% **  \!getnext\\ITEMfrom\LIST
% **    \LIST has the form \\{item1}\\{item2}\\{item3}...\\{itemk}
% **    This routine sets \ITEM to item1, and cycles \LIST to
% **    \\{item2}\\{item3}...\\{itemk}\\{item1}
\def\!getnext#1\from#2{%
  \expandafter\!gnext#2\!#1#2}%
\def\!gnext\\#1#2\!#3#4{%
  \def#3{#1}%
  \def#4{#2\\{#1}}%
  \ignorespaces}


% ** \!getnextvalueof\DIMEN\from\LIST
% **   Similar to !getnext.
% **   \LIST has the form \\{dimen1}\\{dimen2}\\{dimen3} ...
% **   \DIMEN is a dimension register
% **   Works also for counts
%
\def\!getnextvalueof#1\from#2{%
  \expandafter\!gnextv#2\!#1#2}%
\def\!gnextv\\#1#2\!#3#4{%
  #3=#1%
  \def#4{#2\\{#1}}%
  \ignorespaces}


% ** \!copylist\LISTMACROA\to\LISTMACROB
% **   makes the replacement text of LISTMACRO B identical to that of
% **   list macro A.
\def\!copylist#1\to#2{%
  \expandafter\!!copylist#1\!#2}
\def\!!copylist#1\!#2{%
  \def#2{#1}\ignorespaces}


% **  \!wlet\CSA=\CSB
% **  lets control sequence \CSB = control sequence \CSA, and writes a
% **    message to that effect in the log file using plain TEK's \wlog
\def\!wlet#1=#2{%
  \let#1=#2
  \wlog{\string#1=\string#2}}

% ** \!listaddon ITEM LIST
% ** LIST <-- LIST \\ ITEM
\def\!listaddon#1#2{%
  \expandafter\!!listaddon#2\!{#1}#2}
\def\!!listaddon#1\!#2#3{%
  \def#3{#1\\#2}}

% **  \!rightappendITEM\to\LISTMACRO
% **    \LISTMACRO --> \LISTMACRO\\{ITEM}
%\def\!rightappend#1\to#2{\expandafter\!!rightappend#2\!{#1}#2}
%\def\!!rightappend#1\!#2#3{\def#3{#1\\{#2}}}


% **  \!rightappendITEM\withCS\to\LISTMACRO
% **    \LISTMACRO --> \LISTMACRO||CS||{ITEM}
\def\!rightappend#1\withCS#2\to#3{\expandafter\!!rightappend#3\!#2{#1}#3}
\def\!!rightappend#1\!#2#3#4{\def#4{#1#2{#3}}}


% **  \!leftappendITEM\withCS\to\LISTMACRO
% **    \LISTMACRO --> CS||{ITEM}||\LISTMACRO
\def\!leftappend#1\withCS#2\to#3{\expandafter\!!leftappend#3\!#2{#1}#3}
\def\!!leftappend#1\!#2#3#4{\def#4{#2{#3}#1}}


% **  \!lop\LISTMACRO\to\ITEM
% **    \\{item1}\\{item2}\\{item3} ... --> \\{item2}\\{item3} ...
% **    item1 --> \ITEM
\def\!lop#1\to#2{\expandafter\!!lop#1\!#1#2}
\def\!!lop\\#1#2\!#3#4{\def#4{#1}\def#3{#2}}


% **  \!placeNUMBER\of\LISTMACRO\in\ITEM
% **    the NUMBERth item of \LISTMACRO --> replacement text of \ITEM
%\def\!place#1\of#2\in#3{\def#3{\outofrange}%
%{\count0=#1\def\\##1{\advance\count0-1 \ifnum\count0=0 \gdef#3{##1}\fi}#2}}


% **  Following code converts a commalist to a list macro, with all items
% **    fully expanded.
%\!ecfor\item:=\commalist\do{\expandafter\!rightappend\item\to\list}


% ** \!loop ... repeat
% ** This is exactly like TEX's \loop ... repeat.  It can be used in nesting
% ** two loops, without puting the inner one inside a group.
\def\!loop#1\repeat{\def\!body{#1}\!iterate}
\def\!iterate{\!body\let\!next=\!iterate\else\let\!next=\relax\fi\!next}

% ** \!!loop ... repeat
% ** This is exactly like TEX's \loop ... repeat.  It can be used in nesting
% ** two loops, without puting the inner one inside a group.
\def\!!loop#1\repeat{\def\!!body{#1}\!!iterate}
\def\!!iterate{\!!body\let\!!next=\!!iterate\else\let\!!next=\relax\fi\!!next}
%  (\multiput uses \!!loop)

% ** \!removept{DIMENREG}{\CS}
% ** Defines the control sequence CS to be the value (in points) in the
% ** dimension register DIMENREG (but without the "pt" TEK usually adds)
% ** E.g., after  \dimen0=12.3pt \!removept\dimen0\A, \A expands to 12.3
\def\!removept#1#2{\edef#2{\expandafter\!!removePT\the#1}}
{\catcode`p=12 \catcode`t=12 \gdef\!!removePT#1pt{#1}}
% ** \pladevalueinpts of <DIMENSION REGISTER> in {CONTROL SEQUENCE}
\def\placevalueinpts of <#1> in #2 {%
  \!removept{#1}{#2}}

% ** \!mlap{...}  \!vmlap{...}
% ** Center  ...  in a box of width 0.
\def\!mlap#1{\hbox to 0pt{\hss#1\hss}}
\def\!vmlap#1{\vbox to 0pt{\vss#1\vss}}

% ** \!not{TEK if-CONDITION}
% ** By a TEK if-CONDITION is meant something like
% **     \ifnum\N<0,   or   \ifdim\A>\B
% ** \!not produces an if-condition which is false if the original condition
% ** is true, and true if the original condition is false.
\def\!not#1{%
  #1\relax
    \!switchfalse
  \else
    \!switchtrue
  \fi
  \if!switch
  \ignorespaces}

% *******************
% *** ALLOCATIONS ***
% *******************
% This section allocates all the registers PiCTeX uses. Following
% each allocation is a string of the form  ....N.D...L......... ;
% the various letters show which sections of PiCTeX make explicit
% reference to that register, according to the following code:

%   H Hacks
%   A Areas
%   W arroWs
%   B Bars
%   X boXes
%   C Curves
%   D Dashpattterns
%   V diVision
%   E Ellipses
%   U rUles
%   L Linear arc
%   G loGten
%   P Pictures
%   O plOtting
%   Y pYthagoras
%   Q Quadratic arc
%   R Rotations
%   S Shading
%   T Ticks
% Turn off messages from TeX's allocation macros
\let\!!!wlog=\wlog              % "\wlog" is defined in plain TeX
\def\wlog#1{}
\newdimen\headingtoplotskip     %.A.................
\newdimen\linethickness         %.A..X....U........T
\newdimen\longticklength        %.A................T
\newdimen\plotsymbolspacing     %......D...L....Q...
\newdimen\shortticklength       %.A................T
\newdimen\stackleading          %.A..........P......
\newdimen\tickstovaluesleading  %.A................T
\newdimen\totalarclength        %......D...L....Q...
\newdimen\valuestolabelleading  %.A.................
\newbox\!boxA                   %.AW...............T
\newbox\!boxB                   %..W................
\newbox\!picbox                 %............P......
\newbox\!plotsymbol             %..........L..O.....
\newbox\!putobject              %............PO...S.
\newbox\!shadesymbol            %.................S.
\newcount\!countA               %.A....D..UL....Q.ST
\newcount\!countB               %......D..U.....Q.ST
\newcount\!countC               %...............Q..T
\newcount\!countD               %...................
\newcount\!countE               %.............O....T
\newcount\!countF               %.............O....T
\newcount\!countG               %..................T
\newcount\!fiftypt              %.........U.........
\newcount\!intervalno           %..........L....Q...
\newcount\!npoints              %..........L........
\newcount\!nsegments            %.........U.........
\newcount\!ntemp                %............P......
\newcount\!parity               %.................S.
\newcount\!scalefactor          %..................T
\newcount\!tfs                  %.......V...........
\newcount\!tickcase             %..................T
\newdimen\!Xleft                %............P......
\newdimen\!Xright               %............P......
\newdimen\!Xsave                %.A................T
\newdimen\!Ybot                 %............P......
\newdimen\!Ysave                %.A................T
\newdimen\!Ytop                 %............P......
\newdimen\!angle                %........E..........
\newdimen\!arclength            %..W......UL....Q...
\newdimen\!areabloc             %.A........L........
\newdimen\!arealloc             %.A........L........
\newdimen\!arearloc             %.A........L........
\newdimen\!areatloc             %.A........L........
\newdimen\!bshrinkage           %.................S.
\newdimen\!checkbot             %..........L........
\newdimen\!checkleft            %..........L........
\newdimen\!checkright           %..........L........
\newdimen\!checktop             %..........L........
\newdimen\!dimenA               %.AW.X.DVEUL..OYQRST
\newdimen\!dimenB               %....X.DVEU...O.QRS.
\newdimen\!dimenC               %..W.X.DVEU......RS.
\newdimen\!dimenD               %..W.X.DVEU....Y.RS.
\newdimen\!dimenE               %..W........G..YQ.S.
\newdimen\!dimenF               %...........G..YQ.S.
\newdimen\!dimenG               %...........G..YQ.S.
\newdimen\!dimenH               %...........G..Y..S.
\newdimen\!dimenI               %...BX.........Y....
\newdimen\!distacross           %..........L....Q...
\newdimen\!downlength           %..........L........
\newdimen\!dp                   %.A..X.......P....S.
\newdimen\!dshade               %.................S.
\newdimen\!dxpos                %..W......U..P....S.
\newdimen\!dxprime              %...............Q...
\newdimen\!dypos                %..WB.....U..P......
\newdimen\!dyprime              %...............Q...
\newdimen\!ht                   %.A..X.......P....S.
\newdimen\!leaderlength         %......D..U.........
\newdimen\!lshrinkage           %.................S.
\newdimen\!midarclength         %...............Q...
\newdimen\!offset               %.A................T
\newdimen\!plotheadingoffset    %.A.................
\newdimen\!plotsymbolxshift     %..........L..O.....
\newdimen\!plotsymbolyshift     %..........L..O.....
\newdimen\!plotxorigin          %..........L..O.....
\newdimen\!plotyorigin          %..........L..O.....
\newdimen\!rootten              %...........G.......
\newdimen\!rshrinkage           %.................S.
\newdimen\!shadesymbolxshift    %.................S.
\newdimen\!shadesymbolyshift    %.................S.
\newdimen\!tenAa                %...........G.......
\newdimen\!tenAc                %...........G.......
\newdimen\!tenAe                %...........G.......
\newdimen\!tshrinkage           %.................S.
\newdimen\!uplength             %..........L........
\newdimen\!wd                   %....X.......P....S.
\newdimen\!wmax                 %...............Q...
\newdimen\!wmin                 %...............Q...
\newdimen\!xB                   %...............Q...
\newdimen\!xC                   %...............Q...
\newdimen\!xE                   %..W.....E.L....Q.S.
\newdimen\!xM                   %..W.....E......Q.S.
\newdimen\!xS                   %..W.....E.L....Q.S.
\newdimen\!xaxislength          %.A................T
\newdimen\!xdiff                %..........L........
\newdimen\!xleft                %............P......
\newdimen\!xloc                 %..WB.....U.......S.
\newdimen\!xorigin              %.A........L.P....S.
\newdimen\!xpivot               %................R..
\newdimen\!xpos                 %..........L.P..Q.ST
\newdimen\!xprime               %...............Q...
\newdimen\!xright               %............P......
\newdimen\!xshade               %.................S.
\newdimen\!xshift               %..W.........PO...S.
\newdimen\!xtemp                %............P......
\newdimen\!xunit                %.AWBX...EUL.P..QRS.
\newdimen\!xxE                  %........E..........
\newdimen\!xxM                  %........E..........
\newdimen\!xxS                  %........E..........
\newdimen\!xxloc                %..WB....EU.........
\newdimen\!yB                   %...............Q...
\newdimen\!yC                   %...............Q...
\newdimen\!yE                   %..W.....E.L....Q...
\newdimen\!yM                   %..W.....E......Q...
\newdimen\!yS                   %..W.....E.L....Q...
\newdimen\!yaxislength          %.A................T
\newdimen\!ybot                 %............P......
\newdimen\!ydiff                %..........L........
\newdimen\!yloc                 %..WB.....U.......S.
\newdimen\!yorigin              %.A........L.P....S.
\newdimen\!ypivot               %................R..
\newdimen\!ypos                 %..........L.P..Q.ST
\newdimen\!yprime               %...............Q...
\newdimen\!yshade               %.................S.
\newdimen\!yshift               %..W.........PO...S.
\newdimen\!ytemp                %............P......
\newdimen\!ytop                 %............P......
\newdimen\!yunit                %.AWBX...EUL.P..QRS.
\newdimen\!yyE                  %........E..........
\newdimen\!yyM                  %........E..........
\newdimen\!yyS                  %........E..........
\newdimen\!yyloc                %..WB....EU.........
\newdimen\!zpt                  %.AWBX.DVEULGP.YQ.ST
\newif\if!axisvisible           %.A.................
\newif\if!gridlinestoo          %..................T
\newif\if!keepPO                %...................
\newif\if!placeaxislabel        %.A.................
\newif\if!switch                %H..................
\newif\if!xswitch               %.A................T
\newtoks\!axisLaBeL             %.A.................
\newtoks\!keywordtoks           %.A.................
\newwrite\!replotfile           %.............O.....
\newhelp\!keywordhelp{The keyword mentioned in the error message in unknown.
Replace NEW KEYWORD in the indicated response by the keyword that
should have been specified.}    %.A.................
% The following commands assign alternate names to some of the
% above registers.  "\!wlet"  is defined in  Hacks.
\!wlet\!!origin=\!xM                   %.A................T
\!wlet\!!unit=\!uplength               %.A................T
\!wlet\!Lresiduallength=\!dimenG       %.........U.........
\!wlet\!Rresiduallength=\!dimenF       %.........U.........
\!wlet\!axisLength=\!distacross        %.A................T
\!wlet\!axisend=\!ydiff                %.A................T
\!wlet\!axisstart=\!xdiff              %.A................T
\!wlet\!axisxlevel=\!arclength         %.A................T
\!wlet\!axisylevel=\!downlength        %.A................T
\!wlet\!beta=\!dimenE                  %...............Q...
\!wlet\!gamma=\!dimenF                 %...............Q...
\!wlet\!shadexorigin=\!plotxorigin     %.................S.
\!wlet\!shadeyorigin=\!plotyorigin     %.................S.
\!wlet\!ticklength=\!xS                %..................T
\!wlet\!ticklocation=\!xE              %..................T
\!wlet\!ticklocationincr=\!yE          %..................T
\!wlet\!tickwidth=\!yS                 %..................T
\!wlet\!totalleaderlength=\!dimenE     %.........U.........
\!wlet\!xone=\!xprime                  %....X..............
\!wlet\!xtwo=\!dxprime                 %....X..............
\!wlet\!ySsave=\!yM                    %...................
\!wlet\!ybB=\!yB                       %.................S.
\!wlet\!ybC=\!yC                       %.................S.
\!wlet\!ybE=\!yE                       %.................S.
\!wlet\!ybM=\!yM                       %.................S.
\!wlet\!ybS=\!yS                       %.................S.
\!wlet\!ybpos=\!yyloc                  %.................S.
\!wlet\!yone=\!yprime                  %....X..............
\!wlet\!ytB=\!xB                       %.................S.
\!wlet\!ytC=\!xC                       %.................S.
\!wlet\!ytE=\!downlength               %.................S.
\!wlet\!ytM=\!arclength                %.................S.
\!wlet\!ytS=\!distacross               %.................S.
\!wlet\!ytpos=\!xxloc                  %.................S.
\!wlet\!ytwo=\!dyprime                 %....X..............

% Initial values for registers
\!zpt=0pt                              % static
\!xunit=1pt
\!yunit=1pt
\!arearloc=\!xunit
\!areatloc=\!yunit
\!dshade=5pt
\!leaderlength=24in
\!tfs=256                              % static
\!wmax=5.3pt                           % static
\!wmin=2.7pt                           % static
\!xaxislength=\!xunit
\!xpivot=\!zpt
\!yaxislength=\!yunit
\!ypivot=\!zpt
\plotsymbolspacing=.4pt
  \!dimenA=50pt \!fiftypt=\!dimenA     % static
\!rootten=3.162278pt                   % static
\!tenAa=8.690286pt                     % static  (A5)
\!tenAc=2.773839pt                     % static  (A3)
\!tenAe=2.543275pt                     % static  (A1)
% Initial values for control sequences
\def\!cosrotationangle{1}      %................R..
\def\!sinrotationangle{0}      %................R..
\def\!xpivotcoord{0}           %................R..
\def\!xref{0}                  %............P......
\def\!xshadesave{0}            %.................S.
\def\!ypivotcoord{0}           %................R..
\def\!yref{0}                  %............P......
\def\!yshadesave{0}            %.................S.
\def\!zero{0}                  %..................T
% Reset TeX to report allocations
\let\wlog=\!!!wlog
%  *************************************
%  ***  AREAS: Deals with plot areas ***
%  *************************************
%
%  ** User commands
%  **   \setplotarea x from LEFT XCOORD to RIGTH XCOORD, y from BOTTOM YCOORD
%  **      to TOP YCOORD
%  **   \axis BOTTOM-LEFT-TOP-RIGHT  [SHIFTEDTO xy=COORD] [VISIBLE-INVISIBLE]
%  **      [LABEL {label}] [TICKS] /
%  **   \visibleaxes
%  **   \invisibleaxes
%  **   \plotheading {HEADING}
%  **   \grid {# of columns} {# of rows}
%  **   \normalgraphs

%  **  \normalgraphs
%  **    Sets defaults for graph setup. See Subsection 3.4 of manual.
\def\normalgraphs{%
  \longticklength=.4\baselineskip
  \shortticklength=.25\baselineskip
  \tickstovaluesleading=.25\baselineskip
  \valuestolabelleading=.8\baselineskip
  \linethickness=.4pt
  \stackleading=.17\baselineskip
  \headingtoplotskip=1.5\baselineskip
  \visibleaxes
  \ticksout
  \nogridlines
  \unloggedticks}
%
% **  \setplotarea x from LEFT XCOORD to RIGTH XCOORD, y from BOTTOM YCOORD
% **    to TOP YCOORD
% **  Reserves space in PICBOX for a rectangular box with the indicated
% **   coordinates.  Must be specified before calls to  \axis,
% **   \grid, \plotheading.
% **  See Subsection 3.1 of the manual.
\def\setplotarea x from #1 to #2, y from #3 to #4 {%
  \!arealloc=\!M{#1}\!xunit \advance \!arealloc -\!xorigin
  \!areabloc=\!M{#3}\!yunit \advance \!areabloc -\!yorigin
  \!arearloc=\!M{#2}\!xunit \advance \!arearloc -\!xorigin
  \!areatloc=\!M{#4}\!yunit \advance \!areatloc -\!yorigin
  \!initinboundscheck
  \!xaxislength=\!arearloc  \advance\!xaxislength -\!arealloc
  \!yaxislength=\!areatloc  \advance\!yaxislength -\!areabloc
  \!plotheadingoffset=\!zpt
  \!dimenput {{\setbox0=\hbox{}\wd0=\!xaxislength\ht0=\!yaxislength\box0}}
     [bl] (\!arealloc,\!areabloc)}
%
% ** \visibleaxes, \invisibleaxes
% ** Switches for setting visibility of subsequent axes.
% ** See Subsection 3.2 of the manual.
\def\visibleaxes{%
  \def\!axisvisibility{\!axisvisibletrue}}
\def\invisibleaxes{%
  \def\!axisvisibility{\!axisvisiblefalse}}
%
% ** The next few macros enable the user to fix up an erroneous keyword
% **   in the \axis command.
%  \newhelp is in ALLOCATIONS
%  \newhelp\!keywordhelp{The keyword mentioned in the error message in unknown.
%  Replace NEW KEYWORD in the indicated response by the keyword that
%  should have been specified.}
\def\!fixkeyword#1{%
  \errhelp=\!keywordhelp
  \errmessage{Unrecognized keyword `#1': \the\!keywordtoks{NEW KEYWORD}'}}
%  \newtoks\!keywordtoks    In ALLOCATIONS.
\!keywordtoks={enter `i\fixkeyword}
\def\fixkeyword#1{%
  \!nextkeyword#1 }
% ** \axis BOTTOM-LEFT-TOP-RIGHT  [SHIFTEDTO xy=COORD] [VISIBLE-INVISIBLE]
% **   [LABEL {label}] [TICKS] /
% ** Exactly one of the keywords BOTTOM, LEFT, TOP, RIGHT must be
% ** specified. Axis is drawn along the indicated edge of the current
% ** plot area, shifted if the SHIFTEDTO option is used, visible or
% ** invisible according the selected option, with an optional LABEL,
% ** and optional TICKS (see ticks.tex for the options avialabel with
% ** TICKS). The TICKS option must be the last one specified. The \axis
% ** MUST be terminated with a / followed by a space.
% ** See Subsection 3.2 of the manual for more information.
% ** The various options of the \axis command are processed by the
% ** \!nextkeyword macro defined below. For example,
% ** `\!nextkeyword shiftedto ' expands to `\!axisshiftedto'.
\def\axis {%
  \def\!nextkeyword##1 {%
    \expandafter\ifx\csname !axis##1\endcsname \relax
      \def\!next{\!fixkeyword{##1}}%
    \else
      \def\!next{\csname !axis##1\endcsname}%
    \fi
    \!next}%
  \!offset=\!zpt
  \!axisvisibility
  \!placeaxislabelfalse
  \!nextkeyword}
% ** This and the various macros that follow handle the keyword
% ** specifications on the \axis command
% ** See Subsection 3.2 of the manual.
\def\!axisbottom{%
  \!axisylevel=\!areabloc
  \def\!tickxsign{0}%
  \def\!tickysign{-}%
  \def\!axissetup{\!axisxsetup}%
  \def\!axislabeltbrl{t}%
  \!nextkeyword}
\def\!axistop{%
  \!axisylevel=\!areatloc
  \def\!tickxsign{0}%
  \def\!tickysign{+}%
  \def\!axissetup{\!axisxsetup}%
  \def\!axislabeltbrl{b}%
  \!nextkeyword}
\def\!axisleft{%
  \!axisxlevel=\!arealloc
  \def\!tickxsign{-}%
  \def\!tickysign{0}%
  \def\!axissetup{\!axisysetup}%
  \def\!axislabeltbrl{r}%
  \!nextkeyword}
\def\!axisright{%
  \!axisxlevel=\!arearloc
  \def\!tickxsign{+}%
  \def\!tickysign{0}%
  \def\!axissetup{\!axisysetup}%
  \def\!axislabeltbrl{l}%
  \!nextkeyword}
\def\!axisshiftedto#1=#2 {%
  \if 0\!tickxsign
    \!axisylevel=\!M{#2}\!yunit
    \advance\!axisylevel -\!yorigin
  \else
    \!axisxlevel=\!M{#2}\!xunit
    \advance\!axisxlevel -\!xorigin
  \fi
  \!nextkeyword}
\def\!axisvisible{%
  \!axisvisibletrue
  \!nextkeyword}
\def\!axisinvisible{%
  \!axisvisiblefalse
  \!nextkeyword}
\def\!axislabel#1 {%
  \!axisLaBeL={#1}%
  \!placeaxislabeltrue
  \!nextkeyword}
\expandafter\def\csname !axis/\endcsname{%
  \!axissetup % This could done already by "ticks"; if so, now \relax
  \if!placeaxislabel
    \!placeaxislabel
  \fi
  \if +\!tickysign %                 ** (A "top" axis)
    \!dimenA=\!axisylevel
    \advance\!dimenA \!offset %      ** dimA = top of the axis structure
    \advance\!dimenA -\!areatloc %   ** dimA = excess over the plot area
    \ifdim \!dimenA>\!plotheadingoffset
      \!plotheadingoffset=\!dimenA % ** Greatest excess over the plot area
    \fi
  \fi}
% ** \grid {c} {r}
% ** Partitions the plot area into c columns and r rows; see Subsection 3.3
% ** of the manual.
% ** (Other grid patterns can be drawn with the TICKS option of the \axis
% ** command.
\def\grid #1 #2 {%
  \!countA=#1\advance\!countA 1
  \axis bottom invisible ticks length <\!zpt> andacross quantity {\!countA} /
  \!countA=#2\advance\!countA 1
  \axis left   invisible ticks length <\!zpt> andacross quantity {\!countA} / }
% ** \plotheading{HEADING}
% ** Places HEADING centered above the top of the plotarea (and above
% ** any top axis ticks marks, tick labels, and axis label); see
% ** Subsection 3.3 of the manual.
\def\plotheading#1 {%
  \advance\!plotheadingoffset \headingtoplotskip
  \!dimenput {#1} [B] <.5\!xaxislength,\!plotheadingoffset>
    (\!arealloc,\!areatloc)}
% ** From here on, the routines are internal.
\def\!axisxsetup{%
  \!axisxlevel=\!arealloc
  \!axisstart=\!arealloc
  \!axisend=\!arearloc
  \!axisLength=\!xaxislength
  \!!origin=\!xorigin
  \!!unit=\!xunit
  \!xswitchtrue
  \if!axisvisible
    \!makeaxis
  \fi}
\def\!axisysetup{%
  \!axisylevel=\!areabloc
  \!axisstart=\!areabloc
  \!axisend=\!areatloc
  \!axisLength=\!yaxislength
  \!!origin=\!yorigin
  \!!unit=\!yunit
  \!xswitchfalse
  \if!axisvisible
    \!makeaxis
  \fi}
\def\!makeaxis{%
  \setbox\!boxA=\hbox{% (Make a pseudo-y[x] tick for an x[y]-axis)
    \beginpicture
      \!setdimenmode
      \setcoordinatesystem point at {\!zpt} {\!zpt}
      \putrule from {\!zpt} {\!zpt} to
	{\!tickysign\!tickysign\!axisLength}
	{\!tickxsign\!tickxsign\!axisLength}
    \endpicturesave <\!Xsave,\!Ysave>}%
    \wd\!boxA=\!zpt
    \!placetick\!axisstart}
\def\!placeaxislabel{%
  \advance\!offset \valuestolabelleading
  \if!xswitch
    \!dimenput {\the\!axisLaBeL} [\!axislabeltbrl]
      <.5\!axisLength,\!tickysign\!offset> (\!axisxlevel,\!axisylevel)
    \advance\!offset \!dp  % ** advance offset by the "tallness"
    \advance\!offset \!ht  % ** of the label
  \else
    \!dimenput {\the\!axisLaBeL} [\!axislabeltbrl]
      <\!tickxsign\!offset,.5\!axisLength> (\!axisxlevel,\!axisylevel)
  \fi
  \!axisLaBeL={}}

% *******************************
% *** ARROWS  (Draws arrows)  ***
% *******************************
%
% ** User commands
% **  \arrow <ARROW HEAD LENGTH> [MID FRACTION, BASE FRACTION]
% **    [<XSHIFT,YSHIFT>] from XFROM YFROM to XTO YTO
% **  \betweenarrows {TEXT} [orientation & shift] from XFROM YFROM to XTO YTO
% ** \arrow <ARROW HEAD LENGTH> [MID FRACTION, BASE FRACTION]
% **    [<XSHIFT,YSHIFT>] from XFROM YFROM to XTO YTO
% ** Draws an arrow from (XFROM,YFROM) to (XTO,YTO).  The arrow head
% ** is constructed two quadratic arcs, which extend back a distance
% ** ARROW HEAD LENGTH (a dimension) on both sides of the arrow shaft.
% ** All the way back the arcs are a distance BASE FRACTION*ARROW HEAD
% ** LENGTH apart, while half-way back they are a distance MID FRACTION*
% ** ARROW HEAD LENGTH apart. <XSHIFT,YSHIFT> is optional, and has
% ** its usual interpreation. See Subsection 5.4 of the manual.
\def\arrow <#1> [#2,#3]{%
  \!ifnextchar<{\!arrow{#1}{#2}{#3}}{\!arrow{#1}{#2}{#3}<\!zpt,\!zpt> }}
\def\!arrow#1#2#3<#4,#5> from #6 #7 to #8 #9 {%
%
% ** convert to dimensions
  \!xloc=\!M{#8}\!xunit
  \!yloc=\!M{#9}\!yunit
  \!dxpos=\!xloc  \!dimenA=\!M{#6}\!xunit  \advance \!dxpos -\!dimenA
  \!dypos=\!yloc  \!dimenA=\!M{#7}\!yunit  \advance \!dypos -\!dimenA
  \let\!MAH=\!M%                         ** save current c/d mode
  \!setdimenmode%                        ** go into dimension mode
%
  \!xshift=#4\relax  \!yshift=#5\relax%  ** pick up shift
  \!reverserotateonly\!xshift\!yshift%   ** back rotate shift
  \advance\!xshift\!xloc  \advance\!yshift\!yloc
%
% **  draw shaft of arrow
  \!xS=-\!dxpos  \advance\!xS\!xshift
  \!yS=-\!dypos  \advance\!yS\!yshift
  \!start (\!xS,\!yS)
  \!ljoin (\!xshift,\!yshift)
%
% ** find 32*cosine and 32*sine of angle of rotation
  \!Pythag\!dxpos\!dypos\!arclength
  \!divide\!dxpos\!arclength\!dxpos
  \!dxpos=32\!dxpos  \!removept\!dxpos\!!cos
  \!divide\!dypos\!arclength\!dypos
  \!dypos=32\!dypos  \!removept\!dypos\!!sin
%
% ** construct arrowhead
  \!halfhead{#1}{#2}{#3}%                ** draw half of arrow head
  \!halfhead{#1}{-#2}{-#3}%              ** draw other half
%
  \let\!M=\!MAH%                         ** restore old c/d mode
  \ignorespaces}
%
% ** draw half of arrow head
  \def\!halfhead#1#2#3{%
    \!dimenC=-#1%
    \divide \!dimenC 2 %                 ** half way back
    \!dimenD=#2\!dimenC%                 ** half the mid width
    \!rotate(\!dimenC,\!dimenD)by(\!!cos,\!!sin)to(\!xM,\!yM)
    \!dimenC=-#1%                        ** all the way back
    \!dimenD=#3\!dimenC
    \!dimenD=.5\!dimenD%                 ** half the full width
    \!rotate(\!dimenC,\!dimenD)by(\!!cos,\!!sin)to(\!xE,\!yE)
    \!start (\!xshift,\!yshift)
    \advance\!xM\!xshift  \advance\!yM\!yshift
    \advance\!xE\!xshift  \advance\!yE\!yshift
    \!qjoin (\!xM,\!yM) (\!xE,\!yE)
    \ignorespaces}

% ** \betweenarrows {TEXT} [orientation & shift] from XFROM YFROM to XTO YTO
% **   Makes things like <--- text --->, using arrow heads from TeX's fonts.
% **   See Subsection 5.4 of the manual.
\def\betweenarrows #1#2 from #3 #4 to #5 #6 {%
  \!xloc=\!M{#3}\!xunit  \!xxloc=\!M{#5}\!xunit%
  \!yloc=\!M{#4}\!yunit  \!yyloc=\!M{#6}\!yunit%
  \!dxpos=\!xxloc  \advance\!dxpos by -\!xloc
  \!dypos=\!yyloc  \advance\!dypos by -\!yloc
  \advance\!xloc .5\!dxpos
  \advance\!yloc .5\!dypos
%
  \let\!MBA=\!M%           ** save current coord\dimen mode
  \!setdimenmode%          ** express locations in dimens
  \ifdim\!dypos=\!zpt
    \ifdim\!dxpos<\!zpt \!dxpos=-\!dxpos \fi
    \put {\!lrarrows{\!dxpos}{#1}}#2{} at {\!xloc} {\!yloc}
  \else
    \ifdim\!dxpos=\!zpt
      \ifdim\!dypos<\!zpt \!dypos=-\!zpt \fi
      \put {\!udarrows{\!dypos}{#1}}#2{} at {\!xloc} {\!yloc}
    \fi
  \fi
  \let\!M=\!MBA%           ** restore previous c/d mode
  \ignorespaces}
% ** Subroutine for left-right between arrows
\def\!lrarrows#1#2{% #1=width, #2=text
  {\setbox\!boxA=\hbox{$\mkern-2mu\mathord-\mkern-2mu$}%
   \setbox\!boxB=\hbox{$\leftarrow$}\!dimenE=\ht\!boxB
   \setbox\!boxB=\hbox{}\ht\!boxB=2\!dimenE
   \hbox to #1{$\mathord\leftarrow\mkern-6mu
     \cleaders\copy\!boxA\hfil
     \mkern-6mu\mathord-$%
     \kern.4em $\vcenter{\box\!boxB}$$\vcenter{\hbox{#2}}$\kern.4em
     $\mathord-\mkern-6mu
     \cleaders\copy\!boxA\hfil
     \mkern-6mu\mathord\rightarrow$}}}
% ** Subroutine for up-down between arrows
\def\!udarrows#1#2{% #1=width, #2=text
  {\setbox\!boxB=\hbox{#2}%
   \setbox\!boxA=\hbox to \wd\!boxB{\hss$\vert$\hss}%
   \!dimenE=\ht\!boxA \advance\!dimenE \dp\!boxA \divide\!dimenE 2
   \vbox to #1{\offinterlineskip
      \vskip .05556\!dimenE
      \hbox to \wd\!boxB{\hss$\mkern.4mu\uparrow$\hss}\vskip-\!dimenE
      \cleaders\copy\!boxA\vfil
      \vskip-\!dimenE\copy\!boxA
      \vskip\!dimenE\copy\!boxB\vskip.4em
      \copy\!boxA\vskip-\!dimenE
      \cleaders\copy\!boxA\vfil
      \vskip-\!dimenE \hbox to \wd\!boxB{\hss$\mkern.4mu\downarrow$\hss}
      \vskip .05556\!dimenE}}}

% ***************************
% *** BARS  (Draws bars)  ***
% ***************************
%
% ** User commands:
% ** \putbar [<XSHIFT,YSHIFT>] breadth <BREADTH> from XSTART YSTART
% **   to XEND YEND
% ** \setbars [<XSHIFT,YSHIFT>] breadth <BREADTH> baseline at XY = COORD
% **   [baselabels ([B_ORIENTATION_x,B_ORIENTATION_y] <B_XSHIFT,B_YSHIFT>)]
% **   [endlabels  ([E_ORIENTATION_x,E_ORIENTATION_y] <E_XSHIFT,E_YSHIFT>)]

% ** \putbar [<XSHIFT,YSHIFT>] breadth <BREADTH> from XSTART YSTART
% **   to XEND YEND
% ** Either XSTART=XEND or YSTART=YEND. Draws a rectangle between
% **   (XSTART,YSTART) & (XEND,YEND). The "depth" of the rectangle
% **   is determined by those two plot positions; its other
% **   dimension "breadth" is specified by the dimension BREADTH.
% ** See Subsection 4.2 of the manual.
\def\putbar#1breadth <#2> from #3 #4 to #5 #6 {%
  \!xloc=\!M{#3}\!xunit  \!xxloc=\!M{#5}\!xunit%
  \!yloc=\!M{#4}\!yunit  \!yyloc=\!M{#6}\!yunit%
  \!dypos=\!yyloc  \advance\!dypos by -\!yloc
  \!dimenI=#2
%
  \ifdim \!dimenI=\!zpt %            ** If 0 breadth
    \putrule#1from {#3} {#4} to {#5} {#6} % ** Then draw line
  \else %                            ** Else, put in a rectangle
    \let\!MBar=\!M%                  ** save current c/d mode
    \!setdimenmode %                 ** go into dimension mode
    \divide\!dimenI 2
    \ifdim \!dypos=\!zpt
      \advance \!yloc -\!dimenI %    ** Equal y coordinates
      \advance \!yyloc \!dimenI
    \else
      \advance \!xloc -\!dimenI %    ** Equal x coordinates
      \advance \!xxloc \!dimenI
    \fi
    \putrectangle#1corners at {\!xloc} {\!yloc} and {\!xxloc} {\!yyloc}
    \let\!M=\!MBar %                 ** restore c/d mode
  \fi
  \ignorespaces}

% ** \setbars [<XSHIFT,YSHIFT>] breadth <BREADTH> baseline at XY = COORD
% **   [baselabels ([B_ORIENTATION_x,B_ORIENTATION_y] <B_XSHIFT,B_YSHIFT>)]
% **   [endlabels  ([E_ORIENTATION_x,E_ORIENTATION_y] <E_XSHIFT,E_YSHIFT>)]
% ** This command puts PiCTeX into the bar graph drawing mode described
% **   in Subsection 4.4 of the manual.
\def\setbars#1breadth <#2> baseline at #3 = #4 {%
  \edef\!barshift{#1}%
  \edef\!barbreadth{#2}%
  \edef\!barorientation{#3}%
  \edef\!barbaseline{#4}%
  \def\!bardobaselabel{\!bardoendlabel}%
  \def\!bardoendlabel{\!barfinish}%
  \let\!drawcurve=\!barcurve
  \!setbars}
\def\!setbars{%
  \futurelet\!nextchar\!!setbars}
\def\!!setbars{%
  \if b\!nextchar
    \def\!!!setbars{\!setbarsbget}%
  \else
    \if e\!nextchar
      \def\!!!setbars{\!setbarseget}%
    \else
      \def\!!!setbars{\relax}%
    \fi
  \fi
  \!!!setbars}
\def\!setbarsbget baselabels (#1) {%
  \def\!barbaselabelorientation{#1}%
  \def\!bardobaselabel{\!!bardobaselabel}%
  \!setbars}
\def\!setbarseget endlabels (#1) {%
  \edef\!barendlabelorientation{#1}%
  \def\!bardoendlabel{\!!bardoendlabel}%
  \!setbars}
% ** \!barcurve
% ** Draws a bargraph with preset values of barshift, barbreadth,
% ** barorientation (x or y) and barbaseline (coordinate)
\def\!barcurve #1 #2 {%
  \if y\!barorientation
    \def\!basexarg{#1}%
    \def\!baseyarg{\!barbaseline}%
  \else
    \def\!basexarg{\!barbaseline}%
    \def\!baseyarg{#2}%
  \fi
  \expandafter\putbar\!barshift breadth <\!barbreadth> from {\!basexarg}
    {\!baseyarg} to {#1} {#2}
  \def\!endxarg{#1}%
  \def\!endyarg{#2}%
  \!bardobaselabel}
\def\!!bardobaselabel "#1" {%
  \put {#1}\!barbaselabelorientation{} at {\!basexarg} {\!baseyarg}
  \!bardoendlabel}

\def\!!bardoendlabel "#1" {%
  \put {#1}\!barendlabelorientation{} at {\!endxarg} {\!endyarg}
  \!barfinish}
\def\!barfinish{%
  \!ifnextchar/{\!finish}{\!barcurve}}

% ********************************
% *** BOXES (Draws rectangles) ***
% ********************************
%
% ** User commands:
% **   \putrectangle [<XSHIFT,YSHIFT>] corners at  XCOORD1 YCOORD1
% **     and  XCOORD2 YCOORD2
% **   \shaderectangleson
% **   \shaderectanglesoff
% **   \frame [<SEPARATION>] {TEXT}
% **   \rectangle <WIDTH> <HEIGHT>
%
%
% **  \putrectangle [<XSHIFT,YSHIFT>] corners at XCOORD1 YCOORD1
% **    and  XCOORD2 YCOORD2
% **  Draws a rectangle with corners at (X1,Y1), (X2,Y1), (X1,Y2), (X2,Y2)
% **  Lines have thickness \linethickness, and overlap at the corners.
% **  The optional field  <XSHIFT,YSHIFT>  functions as with a \put command.
% **  See Subsection 4.2 of the manual.
\def\putrectangle{%
  \!ifnextchar<{\!putrectangle}{\!putrectangle<\!zpt,\!zpt> }}
\def\!putrectangle<#1,#2> corners at #3 #4 and #5 #6 {%
%
% ** get locations
  \!xone=\!M{#3}\!xunit  \!xtwo=\!M{#5}\!xunit%
  \!yone=\!M{#4}\!yunit  \!ytwo=\!M{#6}\!yunit%
  \ifdim \!xtwo<\!xone
    \!dimenI=\!xone  \!xone=\!xtwo  \!xtwo=\!dimenI
  \fi
  \ifdim \!ytwo<\!yone
    \!dimenI=\!yone  \!yone=\!ytwo  \!ytwo=\!dimenI
  \fi
  \!dimenI=#1\relax  \advance\!xone\!dimenI  \advance\!xtwo\!dimenI
  \!dimenI=#2\relax  \advance\!yone\!dimenI  \advance\!ytwo\!dimenI
  \let\!MRect=\!M%                  ** save current coord/dimen mode
  \!setdimenmode
%
% ** shade rectangle if appropriate
  \!shaderectangle
%
% ** draw horizontal edges
  \!dimenI=.5\linethickness
  \advance \!xone  -\!dimenI%       ** adjust x-location to overlap corners
  \advance \!xtwo   \!dimenI%       ** ditto
  \putrule from {\!xone} {\!yone} to {\!xtwo} {\!yone}
  \putrule from {\!xone} {\!ytwo} to {\!xtwo} {\!ytwo}
%
% ** draw vertical edges
  \advance \!xone   \!dimenI%       ** restore original x-values
  \advance \!xtwo  -\!dimenI%
  \advance \!yone  -\!dimenI%       ** adjust y-location to overlap corners
  \advance \!ytwo   \!dimenI%       ** ditto
  \putrule from {\!xone} {\!yone} to {\!xone} {\!ytwo}
  \putrule from {\!xtwo} {\!yone} to {\!xtwo} {\!ytwo}
%
  \let\!M=\!MRect%                  ** restore coord/dimen mode
  \ignorespaces}

% ** \shaderectangleson
% **   Subsequent rectangles will be shaded according to
% **   the current shading pattern.  Affects \putrectangle, \putbar,
% **   \frame, \sethistograms, and \setbars. See Subsection 7.5 of the manual.
\def\shaderectangleson{%
  \def\!shaderectangle{\!!shaderectangle}%
  \ignorespaces}
% ** \shaderectanglesoff
% **    Suppresses  \shaderectangleson.  The default.
\def\shaderectanglesoff{%
  \def\!shaderectangle{}%
  \ignorespaces}
\shaderectanglesoff

% ** The following internal routine shades the current rectangle, when
% **   \!shaderectangle = \!!shaderectangle .
\def\!!shaderectangle{%
  \!dimenA=\!xtwo  \advance \!dimenA -\!xone
  \!dimenB=\!ytwo  \advance \!dimenB -\!yone
  \ifdim \!dimenA<\!dimenB
    \!startvshade (\!xone,\!yone,\!ytwo)
    \!lshade      (\!xtwo,\!yone,\!ytwo)
  \else
    \!starthshade (\!yone,\!xone,\!xtwo)
    \!lshade      (\!ytwo,\!xone,\!xtwo)
  \fi
  \ignorespaces}

% ** \frame [<SEPARATION>] {TEXT}
% ** Draws a frame of thickness linethickness about the box enclosing
% **   TEXT; the frame is separated from the box by a distance of
% **   SEPARATION.  The result is an hbox with the same baseline as TEXT.
% **   If <SEPARATION> is omitted, you get the effect of <0pt>.
% ** See Subsection 4.2 of the manual.
\def\frame{%
  \!ifnextchar<{\!frame}{\!frame<\!zpt> }}
\long\def\!frame<#1> #2{%
  \beginpicture
    \setcoordinatesystem units <1pt,1pt> point at 0 0
    \put {#2} [Bl] at 0 0
    \!dimenA=#1\relax
    \!dimenB=\!wd \advance \!dimenB \!dimenA
    \!dimenC=\!ht \advance \!dimenC \!dimenA
    \!dimenD=\!dp \advance \!dimenD \!dimenA
    \let\!MFr=\!M
    \!setdimenmode
    \putrectangle corners at {-\!dimenA} {-\!dimenD} and {\!dimenB} {\!dimenC}
    \!setcoordmode
    \let\!M=\!MFr
  \endpicture
  \ignorespaces}

% ** \rectangle <WIDTH> <HEIGHT>
% ** Constructs a rectangle of width WIDTH and heigth HEIGHT.
% ** See Subsection 4.2 of the manual.
\def\rectangle <#1> <#2> {%
  \setbox0=\hbox{}\wd0=#1\ht0=#2\frame {\box0}}

% *********************************************
% ***  CURVES  (Upper level \plot commands) ***
% *********************************************
%
% ** User commands
% **   \plot  DATA  /
% **   \plot  "FILE NAME"
% **   \setquadratic
% **   \setlinear
% **   \sethistograms
% **   \vshade  ...
% **   \hshade  ...
% \plot: multi-purpose command. Draws histograms, bar graphs, piecewise-linear
% or piecewise quadratic curves, depending on the setting of \!drawcurve.
% See Subsections 4.3-4.5, 5.1, 5.2 of the manual.
\def\plot{%
  \!ifnextchar"{\!plotfromfile}{\!drawcurve}}
\def\!plotfromfile"#1"{%
  \expandafter\!drawcurve \input #1 /}
% Command to set piecewise quadratic mode
% See Subsections 5.1, 7.3, and 7.4 of the manual.
\def\setquadratic{%
  \let\!drawcurve=\!qcurve
  \let\!!Shade=\!!qShade
  \let\!!!Shade=\!!!qShade}
% Command to set piecewise linear mode
% See Subsections 5.1, 7.3, and 7.4 of the manual.
\def\setlinear{%
  \let\!drawcurve=\!lcurve
  \let\!!Shade=\!!lShade
  \let\!!!Shade=\!!!lShade}
% Command to set histogram mode
% See Subsection 4.3 of the manual.
\def\sethistograms{%
  \let\!drawcurve=\!hcurve}
% Commands to cycle through list of coordinates in piecewise quadratic
% interpolation mode
\def\!qcurve #1 #2 {%
  \!start (#1,#2)
  \!Qjoin}
\def\!Qjoin#1 #2 #3 #4 {%
  \!qjoin (#1,#2) (#3,#4)             % \!qjoin  is defined in QUADRATIC
  \!ifnextchar/{\!finish}{\!Qjoin}}
% Commands to cycle through list of coordinates in piecewise linear
% interpolation mode
\def\!lcurve #1 #2 {%
  \!start (#1,#2)
  \!Ljoin}
\def\!Ljoin#1 #2 {%
  \!ljoin (#1,#2)                    % \!ljoin  is defined in LINEAR
  \!ifnextchar/{\!finish}{\!Ljoin}}
\def\!finish/{\ignorespaces}
% Command to cycle through list of coordinates in histogram mode
\def\!hcurve #1 #2 {%
  \edef\!hxS{#1}%
  \edef\!hyS{#2}%
  \!hjoin}
\def\!hjoin#1 #2 {%
  \putrectangle corners at {\!hxS} {\!hyS} and {#1} {#2}
  \edef\!hxS{#1}%
  \!ifnextchar/{\!finish}{\!hjoin}}

% \vshade: See Subsection 7.3 of the manual.
\def\vshade #1 #2 #3 {%
  \!startvshade (#1,#2,#3)
  \!Shadewhat}
% \hshade: See Subsection 7.4 of the manual.
\def\hshade #1 #2 #3 {%
  \!starthshade (#1,#2,#3)
  \!Shadewhat}
% Commands to cycle through coordinates and optional "edge effect"
% fields while shading.
\def\!Shadewhat{%
  \futurelet\!nextchar\!Shade}
\def\!Shade{%
  \if <\!nextchar
    \def\!nextShade{\!!Shade}%
  \else
    \if /\!nextchar
      \def\!nextShade{\!finish}%
    \else
      \def\!nextShade{\!!!Shade}%
    \fi
  \fi
  \!nextShade}
\def\!!lShade<#1> #2 #3 #4 {%
  \!lshade <#1> (#2,#3,#4)                 % \!lshade is defined in SHADING
  \!Shadewhat}
\def\!!!lShade#1 #2 #3 {%
  \!lshade (#1,#2,#3)
  \!Shadewhat}
\def\!!qShade<#1> #2 #3 #4 #5 #6 #7 {%
  \!qshade <#1> (#2,#3,#4) (#5,#6,#7)      % \!qshade is defined in SHADING
  \!Shadewhat}
\def\!!!qShade#1 #2 #3 #4 #5 #6 {%
  \!qshade (#1,#2,#3) (#4,#5,#6)
  \!Shadewhat}
% ** Set default interpolation mode
\setlinear

%  ********************************************
%  *** DASHPATTERNS (Sets up dash patterns) ***
%  ********************************************
%  **  User commands:
%  **    \setdashpattern <DIMEN1,DIMEN2,DIMEN3,...>
%  **    \setdots <INTRADOT_DISTANCE>
%  **    \setdotsnear <INTRADOT_DISTANCE> for <ARC LENGTH>
%  **    \setdashes <DASH/SKIP_DISTANCE>
%  **    \setdashesnear <DASH/SKIP_DISTANCE> for <ARC LENGTH>
%  **    \setsolid
%  **    \findlength {CURVE CMDS}

%  **  Internal commands:
%  **    \!dashingon
%  **    \!dashingoff

%  **  Dash patterns are specified by a balanced token list whose complete
%  **    expansion has the form: DIMEN1,DIMEN2,DIMEN3,DIMEN4,... ; this produces
%  **    an arc of length DIMEN1, a skip of length DIMEN2, an arc of length
%  **    DIMEN3, a skip of length DIMEN4, ... .  Any number of DIMEN values may
%  **    be given. The pattern is repeated as many times (perhaps fractional)
%  **    as necessary to draw the curve.
%  **  A dash pattern remains in effect until it is overridden by a call to
%  **    \setdashpattern, or to \setdots, \setdotsnear ... , \setdashes,
%  **    \setdashesnear ... , or \setsolid.
%  **  Solid lines are the default.


%  **  \def\setdashpattern <DIMEN1,DIMEN2,DIMEN3,...>
%  **  The following routine converts a balanced list of tokens whose
%  **  complete expansion has the form  DIMEN1,DIMEN2, ... , DIMENk  into
%  **  three list macros that are used in drawing dashed rules and curves:
%  **    !Flist:   \!Rule{DIMEN1}\!Skip{DIMEN2}\!Rule{DIMEN3}\!Skip{DIMEN4} ...
%  **    !Blist:   ...\!Skip{DIMEN4}\!Rule{DIMEN3}\!Skip{DIMEN2}\!Rule{DIMEN1}
%  **    !UDlist:  \\{DIMEN1}\\{DIMEN2}\\{DIMEN3}\\{DIMEN4} ...;
%  **  calculates \!leaderlength := DIMEN1 + ... + DIMENk; and
%  **  sets the curve drawing routines to dash mode.
%  **  Those lists are used by the curve drawing routines.
%  **  Dimenj ... may be given as an explicit dimension (e.g., 5pt), or
%  **  as an expression involving a dimension register (e.g., -2.5\dimen0).
%  **  See Subsection 6.2 of the manual
\def\setdashpattern <#1>{%
  \def\!Flist{}\def\!Blist{}\def\!UDlist{}%
  \!countA=0
  \!ecfor\!item:=#1\do{%
    \!dimenA=\!item\relax
    \expandafter\!rightappend\the\!dimenA\withCS{\\}\to\!UDlist%
    \advance\!countA  1
    \ifodd\!countA
      \expandafter\!rightappend\the\!dimenA\withCS{\!Rule}\to\!Flist%
      \expandafter\!leftappend\the\!dimenA\withCS{\!Rule}\to\!Blist%
    \else
      \expandafter\!rightappend\the\!dimenA\withCS{\!Skip}\to\!Flist%
      \expandafter\!leftappend\the\!dimenA\withCS{\!Skip}\to\!Blist%
    \fi}%
  \!leaderlength=\!zpt
  \def\!Rule##1{\advance\!leaderlength  ##1}%
  \def\!Skip##1{\advance\!leaderlength  ##1}%
  \!Flist%
  \ifdim\!leaderlength>\!zpt
  \else
    \def\!Flist{\!Skip{24in}}\def\!Blist{\!Skip{24in}}\ignorespaces
    \def\!UDlist{\\{\!zpt}\\{24in}}\ignorespaces
    \!leaderlength=24in
  \fi
  \!dashingon}


%  **  \!dashingon  -- puts the curve drawing routines into dash mode
%  **  \!dashingoff -- puts the curve drawing routines into solid mode
%  **  These are internal commands, invoked by \setdashpattern and \setsolid
\def\!dashingon{%
  \def\!advancedashing{\!!advancedashing}%
  \def\!drawlinearsegment{\!lineardashed}%
  \def\!puthline{\!putdashedhline}%
  \def\!putvline{\!putdashedvline}%
%  \def\!putsline{\!putdashedsline}%
  \ignorespaces}%
\def\!dashingoff{%
  \def\!advancedashing{\relax}%
  \def\!drawlinearsegment{\!linearsolid}%
  \def\!puthline{\!putsolidhline}%
  \def\!putvline{\!putsolidvline}%
%  \def\!putsline{\!putsolidsline}%
  \ignorespaces}


%  **  \setdots <LENGTH>  --  sets up a dot/skip pattern where dot (actually
%  **    the current plotsymbol) is plunked down once for every LENGTH
%  **    traveled along the curve.  LENGTH defaults to 5pt.
%  **    See Subsection 6.1 of the manual.
\def\setdots{%
  \!ifnextchar<{\!setdots}{\!setdots<5pt>}}
\def\!setdots<#1>{%
  \!dimenB=#1\advance\!dimenB -\plotsymbolspacing
  \ifdim\!dimenB<\!zpt
    \!dimenB=\!zpt
  \fi
\setdashpattern <\plotsymbolspacing,\!dimenB>}

% ** \setdotsnear <LENGTH> for <ARC LENGTH>
% ** sets up a dot pattern where the dots are approximately LENGTH apart,
% ** the total length of the pattern is ARC LENGTH, and the pattern
% ** begins and ends with a dot. See Subsection 6.3 of the manual.
\def\setdotsnear <#1> for <#2>{%
  \!dimenB=#2\relax  \advance\!dimenB -.05pt
  \!dimenC=#1\relax  \!countA=\!dimenC
  \!dimenD=\!dimenB  \advance\!dimenD .5\!dimenC  \!countB=\!dimenD
  \divide \!countB  \!countA
  \ifnum 1>\!countB
    \!countB=1
  \fi
  \divide\!dimenB  \!countB
  \setdots <\!dimenB>}

%  **  \setdashes <LENGTH>  --  sets up a dash/skip pattern where the dash
%  **    and the skip are each of length LENGTH (the dash is formed by
%  **    plunking down the current plotsymbol over an arc of length LENGTH
%  **    and so may actually be longer than LENGTH.  LENGTH defaults to 5pt.
%  **    See Subsection 6.1 of the manual.
\def\setdashes{%
  \!ifnextchar<{\!setdashes}{\!setdashes<5pt>}}
\def\!setdashes<#1>{\setdashpattern <#1,#1>}

% ** \setdashesnear ...
% ** Like \setdotsnear; the pattern begins and ends with a dash.
% ** See Subsection 6.3 of the manual.
\def\setdashesnear <#1> for <#2>{%
  \!dimenB=#2\relax
  \!dimenC=#1\relax  \!countA=\!dimenC
  \!dimenD=\!dimenB  \advance\!dimenD .5\!dimenC  \!countB=\!dimenD
  \divide \!countB  \!countA
  \ifodd \!countB
  \else
    \advance \!countB  1
  \fi
  \divide\!dimenB  \!countB
  \setdashes <\!dimenB>}

%  **  \setsolid  --  puts the curve drawing routines in "solid line" mode,
%  **    the default mode.  See Subsection 6.1 of the manual.
\def\setsolid{%
  \def\!Flist{\!Rule{24in}}\def\!Blist{\!Rule{24in}}%
  \def\!UDlist{\\{24in}\\{\!zpt}}%
  \!dashingoff}
\setsolid
%  **  \findlength {CURVE CMDS}
%  **  PiCTeX executes the \start, \ljoin, and \qjoin cmds comprising
%  **  CURVE CMDS without plotting anything, but stashes the length
%  **  of the phantom curve away in \totalarclength.
%  **  See Subsection 6.3 of the manual.
\def\findlength#1{%
  \begingroup
    \setdashpattern <0pt, \maxdimen>
    \setplotsymbol ({})
    \dontsavelinesandcurves
    #1%
  \endgroup
  \ignorespaces}

% *************************************************************
% *** DIVISION  (Does long division of dimension registers) ***
% *************************************************************

% ** User command:
% **   \Divide {DIVIDEND} by {DIVISOR} forming {RESULT}

% ** Internal command
% **   \!divide{DIVIDEND}{DIVISOR}{RESULT}

% **  \!divide DIVIDEND [by] DIVISOR [to get] ANSWER
% **  Divides the dimension DIVIDEND by the dimension DIVISOR, placing the
% **  quotient in the dimension register ANSWER.  Values are understood to
% **  be in points.  E.g.  12.5pt/1.4pt=8.92857pt.
% **  Quotient is accurate to 1/65536pt=2**[-16]pt
% **  |DIVISOR| should be < 2048pt (about 28 inches).
\def\!divide#1#2#3{%
  \!dimenB=#1%                      **  dimB  holds current remainder (r)
  \!dimenC=#2%                      **  dimC  holds divisor (d)
  \!dimenD=\!dimenB%                **  dimD  holds quotient q=r/d for this
  \divide \!dimenD \!dimenC%        **    step, in units of scaled pts
  \!dimenA=\!dimenD%                **  dimA  eventually holds answer (a)
  \multiply\!dimenD \!dimenC%       **  r <-- r - dq
  \advance\!dimenB -\!dimenD%       **  First step complete. Have integer part
%                                   **  of a, and corresponding remainder.
  \!dimenD=\!dimenC%                **  Temporarily use dimD to hold |d|
    \ifdim\!dimenD<\!zpt \!dimenD=-\!dimenD
  \fi
  \ifdim\!dimenD<64pt%              **  Branch on the magnitude of |d|
    \!divstep[\!tfs]\!divstep[\!tfs]%
  \else
    \!!divide
  \fi
  #3=\!dimenA\ignorespaces}
% **  The following code handles divisors  d  with
% **    (1)  .88in =  64pt <= d <  256pt =  3.54in
% **    (2) 3.54in = 256pt <= d < 2048pt = 28.34in
% **  Anything bigger than that may result in an overflow condition.
% **  For our purposes, we should never even see case (2).
\def\!!divide{%
  \ifdim\!dimenD<256pt
    \!divstep[64]\!divstep[32]\!divstep[32]%
  \else
    \!divstep[8]\!divstep[8]\!divstep[8]\!divstep[8]\!divstep[8]%
    \!dimenA=2\!dimenA
  \fi}


% **  The following macro does the real long division work.
\def\!divstep[#1]{%                 **  #1 = "B"
  \!dimenB=#1\!dimenB%              **  r <-- B*r
  \!dimenD=\!dimenB%                **  dimD  holds quotient q=r/d for this
    \divide \!dimenD by \!dimenC%   **    step, in units of scaled pts
  \!dimenA=#1\!dimenA%              **  a <-- B*a + q
    \advance\!dimenA by \!dimenD%
  \multiply\!dimenD by \!dimenC%    **  r <-- r - dq
    \advance\!dimenB by -\!dimenD}

% **  \Divide:  See Subsection 9.3 of the manual.
\def\Divide <#1> by <#2> forming <#3> {%
  \!divide{#1}{#2}{#3}}

% *********************************************
% *** ELLIPSES (Draws ellipses and circles) ***
% *********************************************

% ** User commands
% **   \ellipticalarc  axes ratio A:B  DEGREES degrees from XSTART YSTART
% **      center at XCENTER YCENTER
% **   \circulararc DEGREES degrees from XSTART YSTART
% **      center at XCENTER YCENTER

% ** Internal command
% **   \!sinandcos{32*ANGLE in radians}{32*SIN}{32*COS}


% **   \ellipticalarc  axes ratio A:B  DEGREES degrees from XSTART YSTART
% **      center at XCENTER YCENTER
% **    Draws a elliptical arc starting at the coordinate point (XSTART,YSTART).
% **    The center of the ellipse of which the arc is a segment is at
% **      (XCENTER,YCENTER).
% **    The arc extends through an angle of DEGREES degrees (may be + or -).
% **    A:B is the ratio of the length of the xaxis to the length of
% **      the yaxis of the ellipse
% **    Sqrt{[(XSTART-XCENTER)/A]**2 + [(YSTART-YCENTER)/B]**2}
% **      must be < 512pt (about 7in).
% **    Doesn't modify the dimensions (ht, dp, wd) of the PiCture under
% **      construction.

% ** \circulararc  --  See Subsection 5.3 of the manual.
\def\circulararc{%
  \ellipticalarc axes ratio 1:1 }
% ** \ellipticalarc  --  See Subsection 5.3 of the manual.
\def\ellipticalarc axes ratio #1:#2 #3 degrees from #4 #5 center at #6 #7 {%
  \!angle=#3pt\relax%                    ** get angle
  \ifdim\!angle>\!zpt
    \def\!sign{}%                        ** counterclockwise
  \else
    \def\!sign{-}\!angle=-\!angle%       ** clockwise
  \fi
  \!xxloc=\!M{#6}\!xunit%                ** convert CENTER to dimension
  \!yyloc=\!M{#7}\!yunit
  \!xxS=\!M{#4}\!xunit%                  ** get STARTing point on rim of ellipse
  \!yyS=\!M{#5}\!yunit
  \advance\!xxS -\!xxloc%                ** make center of ellipse (0,0)
  \advance\!yyS -\!yyloc
  \!divide\!xxS{#1pt}\!xxS %             ** scale point on ellipse to point on
  \!divide\!yyS{#2pt}\!yyS %                 corresponding circle
%
  \let\!MC=\!M%                          ** save current c/d mode
  \!setdimenmode%                        ** go into dimension mode
%
  \!xS=#1\!xxS  \advance\!xS\!xxloc
  \!yS=#2\!yyS  \advance\!yS\!yyloc
  \!start (\!xS,\!yS)%
  \!loop\ifdim\!angle>14.9999pt%         ** draw in major portion of ellipse
    \!rotate(\!xxS,\!yyS)by(\!cos,\!sign\!sin)to(\!xxM,\!yyM)
    \!rotate(\!xxM,\!yyM)by(\!cos,\!sign\!sin)to(\!xxE,\!yyE)
    \!xM=#1\!xxM  \advance\!xM\!xxloc  \!yM=#2\!yyM  \advance\!yM\!yyloc
    \!xE=#1\!xxE  \advance\!xE\!xxloc  \!yE=#2\!yyE  \advance\!yE\!yyloc
    \!qjoin (\!xM,\!yM) (\!xE,\!yE)
    \!xxS=\!xxE  \!yyS=\!yyE
    \advance \!angle -15pt
  \repeat
  \ifdim\!angle>\!zpt%                   ** complete remaining arc, if any
    \!angle=100.53096\!angle%            ** convert angle to radians, divide
    \divide \!angle 360 %                **   by 2, and multiply by 32
    \!sinandcos\!angle\!!sin\!!cos%      ** get 32*sin & 32*cos
    \!rotate(\!xxS,\!yyS)by(\!!cos,\!sign\!!sin)to(\!xxM,\!yyM)
    \!rotate(\!xxM,\!yyM)by(\!!cos,\!sign\!!sin)to(\!xxE,\!yyE)
    \!xM=#1\!xxM  \advance\!xM\!xxloc  \!yM=#2\!yyM  \advance\!yM\!yyloc
    \!xE=#1\!xxE  \advance\!xE\!xxloc  \!yE=#2\!yyE  \advance\!yE\!yyloc
    \!qjoin (\!xM,\!yM) (\!xE,\!yE)
  \fi
%
  \let\!M=\!MC%                          ** restore c/d mode
  \ignorespaces}%                        **   if appropriate


%  ** \!rotate(XREG,YREG)by(32cos,32sin)to(XXREG,YYREG)
%  ** rotates (XREG,YREG) by angle with specfied scaled cos & sin to
%  ** (XXREG,YYREG).  Uses \!dimenA & \!dimenB as scratch registers.
\def\!rotate(#1,#2)by(#3,#4)to(#5,#6){%
  \!dimenA=#3#1\advance \!dimenA -#4#2%   ** Rcos(x+t)=Rcosx*cost - Rsinx*sint
  \!dimenB=#3#2\advance \!dimenB  #4#1%   ** Rsin(x+t)=Rsinx*cost + Rcosx*sint
  \divide \!dimenA 32  \divide \!dimenB 32
  #5=\!dimenA  #6=\!dimenB
  \ignorespaces}
\def\!sin{4.17684}%                       ** 32*sin(pi/24) (pi/24=7.5deg)
\def\!cos{31.72624}%                      ** 32*cos(pi/24)


%  ** \!sinandcos{32*ANGLE in radians}{\SINCS}{\COSCS}
%  **   Computes the 32*sine and 32*cosine of a small ANGLE expressed in
%  **   radians/32 and puts these values in the replacement texts of
%  **   \SINCS and \COSCS
\def\!sinandcos#1#2#3{%
 \!dimenD=#1%                **  angle is expressed in radians/32: 1pt = 1/32rad
 \!dimenA=\!dimenD%          **  dimA will eventually contain 32sin(angle)in pts
 \!dimenB=32pt%              **  dimB will eventually contain 32cos(angle)in pts
 \!removept\!dimenD\!value%  **  get value of 32*angle, without "pt"
 \!dimenC=\!dimenD%          **  holds 32*angle**i/i! in pts
 \!dimenC=\!value\!dimenC \divide\!dimenC by 64 %   ** now 32*angle**2/2
 \advance\!dimenB by -\!dimenC%                     ** 32-32*angle**2/2
 \!dimenC=\!value\!dimenC \divide\!dimenC by 96 %   ** now 32*angle**3/3!
 \advance\!dimenA by -\!dimenC%                     ** now 32*(angle-angle**3/6)
 \!dimenC=\!value\!dimenC \divide\!dimenC by 128 %  ** now 32*angle**4/4!
 \advance\!dimenB by \!dimenC%
 \!removept\!dimenA#2%                              ** set 32*sin(angle)
 \!removept\!dimenB#3%                              ** set 32*cos(angle)
 \ignorespaces}

% *****************************************************************
% ***  RULES  (Draws rules, i.e., horizontal & vertical lines)  ***
% *****************************************************************
% **  User command:
% **    \putrule [<XDIMEN,YDIMEN>] from  XCOORD1 YCOORD1
% **      to  XCOORD2 YCOORD2
% **  Internal commands:
% **    \!puthline [<XDIMEN,YDIMEN>]    (h = horizontal)
% **      Set by dashpat to either: \!putsolidhline  or \!putdashedhline
% **    \!putvline [<XDIMEN,YDIMEN>]    (v = vertical)
% **      Either:  \!putsolidvline  or  \!putdashedvline

% **  \putrule [<XDIMEN,YDIMEN>] from XCOORD1 YCOORD1
% **    to XCOORD2 YCOORD2
% **  Draws a rule -- dashed or solid depending on the current dash pattern --
% **    from (X1,Y1) to (X2,Y2).  Uses TEK's  \hrule & \vrule & \leaders
% **    constructions to handle horizontal & vertical lines efficiently both
% **    in terms of execution time and space in the DVI file.
% **  See Subsection 4.1 of the manual.
\def\putrule#1from #2 #3 to #4 #5 {%
  \!xloc=\!M{#2}\!xunit  \!xxloc=\!M{#4}\!xunit%
  \!yloc=\!M{#3}\!yunit  \!yyloc=\!M{#5}\!yunit%
  \!dxpos=\!xxloc  \advance\!dxpos by -\!xloc
  \!dypos=\!yyloc  \advance\!dypos by -\!yloc
%
  \ifdim\!dypos=\!zpt
    \def\!!Line{\!puthline{#1}}\ignorespaces
  \else
    \ifdim\!dxpos=\!zpt
      \def\!!Line{\!putvline{#1}}\ignorespaces
    \else
       \def\!!Line{}
    \fi
  \fi
  \let\!ML=\!M%           ** save current coord\dimen mode
  \!setdimenmode%         ** express locations in dimens
  \!!Line%
  \let\!M=\!ML%           ** restore previous c/d mode
  \ignorespaces}


% **  \!putsolidhline [<XDIMEN,YDIMEN>]
% **  Place horizontal solid line
\def\!putsolidhline#1{%
  \ifdim\!dxpos>\!zpt
    \put{\!hline\!dxpos}#1[l] at {\!xloc} {\!yloc}
  \else
    \put{\!hline{-\!dxpos}}#1[l] at {\!xxloc} {\!yyloc}
  \fi
  \ignorespaces}

% **  \!putsolidvline [shifted <XDIMEN,YDIMEN>]
% **  Place vertical solid line
\def\!putsolidvline#1{%
  \ifdim\!dypos>\!zpt
    \put{\!vline\!dypos}#1[b] at {\!xloc} {\!yloc}
  \else
    \put{\!vline{-\!dypos}}#1[b] at {\!xxloc} {\!yyloc}
  \fi
  \ignorespaces}

\def\!hline#1{\hbox to #1{\leaders \hrule height\linethickness\hfill}}
\def\!vline#1{\vbox to #1{\leaders \vrule width\linethickness\vfill}}


% **  \!putdashedhline [<XDIMEN,YDIMEN>]
% **  Place dashed horizontal line
\def\!putdashedhline#1{%
  \ifdim\!dxpos>\!zpt
    \!DLsetup\!Flist\!dxpos
    \put{\hbox to \!totalleaderlength{\!hleaders}\!hpartialpattern\!Rtrunc}
      #1[l] at {\!xloc} {\!yloc}
  \else
    \!DLsetup\!Blist{-\!dxpos}
    \put{\!hpartialpattern\!Ltrunc\hbox to \!totalleaderlength{\!hleaders}}
      #1[r] at {\!xloc} {\!yloc}
  \fi
  \ignorespaces}

% **  \!putdashedhline [<XDIMEN,YDIMEN>]
% **  Place dashed vertical line
\def\!putdashedvline#1{%
  \!dypos=-\!dypos%            ** vertical leaders go from top to bottom
  \ifdim\!dypos>\!zpt
    \!DLsetup\!Flist\!dypos
    \put{\vbox{\vbox to \!totalleaderlength{\!vleaders}
      \!vpartialpattern\!Rtrunc}}#1[t] at {\!xloc} {\!yloc}
  \else
    \!DLsetup\!Blist{-\!dypos}
    \put{\vbox{\!vpartialpattern\!Ltrunc
      \vbox to \!totalleaderlength{\!vleaders}}}#1[b] at {\!xloc} {\!yloc}
  \fi
  \ignorespaces}

% **  The rest of the macros in this section are subroutines used by
% **  \!putdashedhline and \!putdashedvline.
\def\!DLsetup#1#2{%            ** Dashed-Line set up
  \let\!RSlist=#1%             ** set !Rule-Skip list
  \!countB=#2%                 ** convert rule length to integer (number of sps)
  \!countA=\!leaderlength%     ** ditto, leaderlength
  \divide\!countB by \!countA% ** number of complete leader units
  \!totalleaderlength=\!countB\!leaderlength
  \!Rresiduallength=#2%
  \advance \!Rresiduallength by -\!totalleaderlength%  \** excess length
  \!Lresiduallength=\!leaderlength
  \advance \!Lresiduallength by -\!Rresiduallength
  \ignorespaces}

\def\!hleaders{%
  \def\!Rule##1{\vrule height\linethickness width##1}%
  \def\!Skip##1{\hskip##1}%
  \leaders\hbox{\!RSlist}\hfill}

\def\!hpartialpattern#1{%
  \!dimenA=\!zpt \!dimenB=\!zpt
  \def\!Rule##1{#1{##1}\vrule height\linethickness width\!dimenD}%
  \def\!Skip##1{#1{##1}\hskip\!dimenD}%
  \!RSlist}

\def\!vleaders{%
  \def\!Rule##1{\hrule width\linethickness height##1}%
  \def\!Skip##1{\vskip##1}%
  \leaders\vbox{\!RSlist}\vfill}

\def\!vpartialpattern#1{%
  \!dimenA=\!zpt \!dimenB=\!zpt
  \def\!Rule##1{#1{##1}\hrule width\linethickness height\!dimenD}%
  \def\!Skip##1{#1{##1}\vskip\!dimenD}%
  \!RSlist}

\def\!Rtrunc#1{\!trunc{#1}>\!Rresiduallength}
\def\!Ltrunc#1{\!trunc{#1}<\!Lresiduallength}

\def\!trunc#1#2#3{%
  \!dimenA=\!dimenB
  \advance\!dimenB by #1%
  \!dimenD=\!dimenB  \ifdim\!dimenD#2#3\!dimenD=#3\fi
  \!dimenC=\!dimenA  \ifdim\!dimenC#2#3\!dimenC=#3\fi
  \advance \!dimenD by -\!dimenC}

%  ****************************************************************
%  ***  LINEAR ARC  (Draws straight lines -- solid and dashed)  ***
%  ****************************************************************

%  **  User commands
%  **    \inboundscheckoff
%  **    \inboundscheckon

%  **  Internal commands
%  **    \!start (XCOORD,YCOORD)
%  **    \!ljoin (XCOORD,YCOORD)
%  **    \!drawlinearsegment  --  set by \dashpat to either
%  **      \!linearsolid  or  \!lineardashed
%  **    \!advancedashing     --  set by \dashpat to either
%  **       \relax  or  \!!advancedashing
%  **    \!plotifinbounds     --  set by \inboundscheck off/on to either
%  **       \!plot  or  \!!plotifinbounds
%  **    \!initinboundscheck  --  set by \inboundscheck off/on to either
%  **       \relax  or  \!!initinboundscheck


%  \plotsymbolspacing  ** distance between consecutive plot positions
%  \!xS                ** starting x
%  \!yS                ** starting y
%  \!xE                ** ending   x
%  \!yE                ** ending   y
%  \!xdiff             ** x_end - x_start
%  \!ydiff             ** y_end - y_start
%  \!distacross        ** how far along curve next point to be plotted is
%  \!arclength         ** approximate length of arc for current interval
%  \!downlength        ** remaining length for "pen" to be down
%  \!uplength          ** length for "pen" to be down
%  \!intervalno        ** counts segments to curve
%  \totalarclength     ** cumulative distance along curve
%  \!npoints           ** approximately  (arc length / plotsymbolspacing)

%  **  Calls -- \!Pythag, \!divide, \!plot


%  **  \!start (XCOORD,YCOORD)
%  **  Sets initial point for linearly (or quadratically) interpolated curve
\def\!start (#1,#2){%
  \!plotxorigin=\!xorigin  \advance \!plotxorigin by \!plotsymbolxshift
  \!plotyorigin=\!yorigin  \advance \!plotyorigin by \!plotsymbolyshift
  \!xS=\!M{#1}\!xunit \!yS=\!M{#2}\!yunit
  \!rotateaboutpivot\!xS\!yS
  \!copylist\!UDlist\to\!!UDlist% **\!UDlist has the form \\{dimen1}\\{dimen2}..
%                                 ** Routine will draw dashed line with pen
%                                 ** down for dimen1, up for dimen2, ...
  \!getnextvalueof\!downlength\from\!!UDlist
  \!distacross=\!zpt%             ** 1st point goes at start of curve
  \!intervalno=0 %                ** initialize interval counter
  \global\totalarclength=\!zpt%   ** initialize distance traveled along curve
  \ignorespaces}

%  **  \!ljoin (XCOORD,YCOORD)
%  **  Draws a straight line starting at the last point specified
%  **    by the most recent \!start, \!ljoin, or \!qjoin, and
%  **    ending at (XCOORD,YCOORD).
\def\!ljoin (#1,#2){%
  \advance\!intervalno by 1
  \!xE=\!M{#1}\!xunit \!yE=\!M{#2}\!yunit
  \!rotateaboutpivot\!xE\!yE
  \!xdiff=\!xE \advance \!xdiff by -\!xS%**  xdiff = xE - xS
  \!ydiff=\!yE \advance \!ydiff by -\!yS%**  ydiff = yE - yS
  \!Pythag\!xdiff\!ydiff\!arclength%     **  arclength = sqrt(xdiff**2+ydiff**2)
  \global\advance \totalarclength by \!arclength%
  \!drawlinearsegment%   ** set by dashpat to \!linearsolid or \!lineardashed
  \!xS=\!xE \!yS=\!yE%   ** shift ending points to starting points
  \ignorespaces}


% **  The following routine is used to draw a "solid" line between (xS,yS)
% **  and (xE,yE).  Points are spaced nearly every  \plotsymbolspacing length
% **  along the line.
\def\!linearsolid{%
  \!npoints=\!arclength
  \!countA=\plotsymbolspacing
  \divide\!npoints by \!countA%      ** now #pts =. arclength/plotsymbolspacing
  \ifnum \!npoints<1
    \!npoints=1
  \fi
  \divide\!xdiff by \!npoints
  \divide\!ydiff by \!npoints
  \!xpos=\!xS \!ypos=\!yS
%
  \loop\ifnum\!npoints>-1
    \!plotifinbounds
    \advance \!xpos by \!xdiff
    \advance \!ypos by \!ydiff
    \advance \!npoints by -1
  \repeat
  \ignorespaces}


% ** The following routine is used to draw a dashed line between (xS,yS)
% ** and (xE,yE). The dash pattern continues from the previous segment.
\def\!lineardashed{%
% **
  \ifdim\!distacross>\!arclength
    \advance \!distacross by -\!arclength  %nothing to plot in this interval
%
  \else
%
    \loop\ifdim\!distacross<\!arclength
%     ** plot point, interpolating linearly in x and y
      \!divide\!distacross\!arclength\!dimenA%  ** dimA = across/arclength
      \!removept\!dimenA\!t%  ** \!t holds value in dimA, without the "pt"
      \!xpos=\!t\!xdiff \advance \!xpos by \!xS
      \!ypos=\!t\!ydiff \advance \!ypos by \!yS
      \!plotifinbounds
      \advance\!distacross by \plotsymbolspacing
      \!advancedashing
    \repeat
%
    \advance \!distacross by -\!arclength%    ** prepare for next interval
  \fi
  \ignorespaces}


\def\!!advancedashing{%
  \advance\!downlength by -\plotsymbolspacing
  \ifdim \!downlength>\!zpt
  \else
    \advance\!distacross by \!downlength
    \!getnextvalueof\!uplength\from\!!UDlist
    \advance\!distacross by \!uplength
    \!getnextvalueof\!downlength\from\!!UDlist
  \fi}


% ** \inboundscheckoff & \inboundscheckon: See Subsection 5.5 of the manual.
\def\inboundscheckoff{%
  \def\!plotifinbounds{\!plot(\!xpos,\!ypos)}%
  \def\!initinboundscheck{\relax}\ignorespaces}
\def\inboundscheckon{%
  \def\!plotifinbounds{\!!plotifinbounds}%
  \def\!initinboundscheck{\!!initinboundscheck}%
  \!initinboundscheck\ignorespaces}
\inboundscheckoff

% ** The following code plots the current point only if it falls in the
% ** current plotarea.  It doesn't matter if the coordinate system has
% ** changed since the plotarea was set up.  However, shifts of the plot
% ** are ignored (how the plotsymbol stands relative to its plot position is
% ** unknown anyway).
\def\!!plotifinbounds{%
  \ifdim \!xpos<\!checkleft
  \else
    \ifdim \!xpos>\!checkright
    \else
      \ifdim \!ypos<\!checkbot
      \else
	 \ifdim \!ypos>\!checktop
	 \else
	   \!plot(\!xpos,\!ypos)
	 \fi
      \fi
    \fi
  \fi}


\def\!!initinboundscheck{%
  \!checkleft=\!arealloc     \advance\!checkleft by \!xorigin
  \!checkright=\!arearloc    \advance\!checkright by \!xorigin
  \!checkbot=\!areabloc      \advance\!checkbot by \!yorigin
  \!checktop=\!areatloc      \advance\!checktop by \!yorigin}

% *********************************
% *** LOGTEN  (Log_10 function) ***
% *********************************
%
% ** \!logten{X}
% ** Calculates log_10 of X.  X and LOG10(X) are in fixed point notation.
% **  X must be positive; it may have an optional `+' sign; any number
% **  of digits may be specified for X.  The absolute error in LOG10(X) is
% **  less than .0001 (probably < .00006).  That's about as good as you
% **  hope for, since TEX only operates to 5 figures after the decimal
% **  point anyway.
%  \!rootten=3.162278pt       **** These are values are set in ALLOCATIONS
%  \!tenAe=2.543275pt  (=A5)
%  \!tenAc=2.773839pt  (=A3)
%  \!tenAa=8.690286pt  (=A1)
\def\!logten#1#2{%
  \expandafter\!!logten#1\!nil
  \!removept\!dimenF#2%
  \ignorespaces}
\def\!!logten#1#2\!nil{%
  \if -#1%
    \!dimenF=\!zpt
    \def\!next{\ignorespaces}%
  \else
    \if +#1%
      \def\!next{\!!logten#2\!nil}%
    \else
      \if .#1%
	\def\!next{\!!logten0.#2\!nil}%
      \else
	\def\!next{\!!!logten#1#2..\!nil}%
      \fi
    \fi
  \fi
  \!next}
\def\!!!logten#1#2.#3.#4\!nil{%
  \!dimenF=1pt %                 ** DimF holds log10 original argument
  \if 0#1%
    \!!logshift#3pt %            ** Argument < 1
  \else %                        ** Argument >= 1
    \!logshift#2/%               ** Shift decimal pt as many places
    \!dimenE=#1.#2#3pt %         **   as there are figures in #2
  \fi %                          ** Now dimE holds revised X want log10 of
  \ifdim \!dimenE<\!rootten%          ** Transform X to XX between sqrt(10)
    \multiply \!dimenE 10 %           **   and 10*sqrt(10)
    \advance  \!dimenF -1pt
  \fi
  \!dimenG=\!dimenE%                  ** dimG <- (XX + 10)
    \advance\!dimenG 10pt
  \advance\!dimenE -10pt %            ** dimE <- (XX - 10)
  \multiply\!dimenE 10 %              ** dimE = 10*(XX-10)
  \!divide\!dimenE\!dimenG\!dimenE%   ** Now dimE=10t==10*(XX-10)/(XX+10)
  \!removept\!dimenE\!t%              ** !t=10t, with "pt" removed
  \!dimenG=\!t\!dimenE%               ** dimG=100t**2
  \!removept\!dimenG\!tt%             ** !tt=100t**2, with "pt" removed
  \!dimenH=\!tt\!tenAe%               ** dimH=10*a5*(10t)**2 /100
    \divide\!dimenH 100
  \advance\!dimenH \!tenAc%           ** ditto + 10*a3
  \!dimenH=\!tt\!dimenH%              ** ditto * (10t)**2 /100
    \divide\!dimenH 100
  \advance\!dimenH \!tenAa%           ** ditto + 10*a1
  \!dimenH=\!t\!dimenH%               ** ditto * 10t / 100
    \divide\!dimenH 100 %             ** Now dimH = log10(XX) - 1
  \advance\!dimenF \!dimenH}%         ** dimF = log10(X)
\def\!logshift#1{%
  \if #1/%
    \def\!next{\ignorespaces}%
  \else
    \advance\!dimenF 1pt
    \def\!next{\!logshift}%
  \fi
  \!next}

 \def\!!logshift#1{%
   \advance\!dimenF -1pt
   \if 0#1%
     \def\!next{\!!logshift}%
   \else
     \if p#1%
       \!dimenF=1pt
       \def\!next{\!dimenE=1p}%
     \else
       \def\!next{\!dimenE=#1.}%
     \fi
   \fi
   \!next}

% ***********************************************************
% *** PICTURES (Basic setups for PiCtures; \put commands) ***
% ***********************************************************

% **  User Commands:
% **    \beginpicture
% **    \endpicture
% **    \endpicturesave <XREG,YREG>
% **    \setcoordinatesystem units <XUNIT,YUNIT> point at XREF YREF
% **    \put {OBJECT} [ORIENTATION] <XSHIFT,YSHIFT> at XCOORD YCOORD
% **    \multiput {OJBECT} [ORIENTATION] <XSHIFT,YSHIFT>) at
% **      XCOORD YCOORD
% **      *NUMBER_OF_TIMES DXCOORD DYCOORD  /
% **    \accountingon
% **    \accountingoff
% **    \stack [ORIENTATION] <LEADING> {LIST OF ITEMS}
% **    \lines [ORIENTATION] {LINES}
% **    \Lines [ORIENTATION] {LINES}
% **    \setdimensionmode
% **    \setcoordinatemode
% **    \Xdistance
% **    \Ydistance

% **  Internal commands:
% **    \!setputobject{OBJECT}{[ORIENTATION]<XSHIFT,YSHIFT>}
% **    \!dimenput{OBJECT}[ORIENTATION]<XSHIFT,YSHIFT>(XDIMEN,YDIMEN)
% **    \!setdimenmode
% **    \!setcoordmode
% **    \!ifdimenmode
% **    \!ifcoordmode


% **  \beginpicture
% **  \endpicture
% **  \endpicturesave <XREG,YREG>
% **    \beginpicture ... \endpicture  creates an hbox.  Objects are
% **    placed in this box using the \put command and the like (see below).
% **    The location of an object is specified in terms of coordinate system(s)
% **    established by \setcoordinatesystem.  Each coordinate system (there
% **    might be just one) specifies the length of 1 horizontal unit, the length
% **    of 1 vertical unit, and the coordinates of a "reference point".  The
% **    reference points of various coordinate systems will be in the same
% **    physical location.  The macros keep track of the size of the objects
% **    and their locations. The resulting hbox is the smallest hbox which
% **    encloses all the objects, and whose TEK reference point is the point
% **    on the left edge of the box closest vertically to the PICTEX reference
% **    point. Using \endpicturesave, you can (globally) save the distance TEK's
% **    reference point is to the right (respectively, up from) PICTEX's
% **    reference point in the dimension register \XREG (respectively \YREG).
% **    You can then \put the picture OBJECT into a larger picture so that its
% **    reference point is at (XCOORD,YCOORD) with the command
% **      \put {picture OBJECT} [Bl] <\XREG, \YREG> at  XCOORD YCOORD
% **  \beginpicture : See Subsection 1.1 of the manual.
\def\beginpicture{%
  \setbox\!picbox=\hbox\bgroup%
  \!xleft=\maxdimen
  \!xright=-\maxdimen
  \!ybot=\maxdimen
  \!ytop=-\maxdimen}

% **  \endpicture : See Subsection 1.1 of the manual.
\def\endpicture{%
  \ifdim\!xleft=\maxdimen%  ** check if nothing was put in picbox
    \!xleft=\!zpt \!xright=\!zpt \!ybot=\!zpt \!ytop=\!zpt
  \fi
  \global\!Xleft=\!xleft \global\!Xright=\!xright
  \global\!Ybot=\!ybot \global\!Ytop=\!ytop
  \egroup%
  \ht\!picbox=\!Ytop  \dp\!picbox=-\!Ybot
  \ifdim\!Ybot>\!zpt
  \else
    \ifdim\!Ytop<\!zpt
      \!Ybot=\!Ytop
    \else
      \!Ybot=\!zpt
    \fi
  \fi
  \hbox{\kern-\!Xleft\lower\!Ybot\box\!picbox\kern\!Xright}}

% **  \endpicturesave : See Subsection 8.4 of the manual.
\def\endpicturesave <#1,#2>{%
  \endpicture \global #1=\!Xleft \global #2=\!Ybot \ignorespaces}


% **   \setcoordinatesystem units <XUNIT,YUNIT>
% **     point at XREF YREF
% **   Each of `units <XUNIT,YUNIT>' and `point at XREF YREF'
% **     are optional.
% **   Unit lengths must be given in dimensions (e.g., <10pt,1in>).
% **     Default unit lengths are 1pt, 1pt, or previous unit lengths.
% **   Reference point is specified in current units (e.g., 3 5 ).
% **     Default reference point is 0 0 , or previous reference point.
% **   Unit lengths and reference points obey TEX's scoping rules.
% **   See Subsection 1.2 of the manual.
\def\setcoordinatesystem{%
  \!ifnextchar{u}{\!getlengths }
    {\!getlengths units <\!xunit,\!yunit>}}
\def\!getlengths units <#1,#2>{%
  \!xunit=#1\relax
  \!yunit=#2\relax
  \!ifcoordmode
    \let\!SCnext=\!SCccheckforRP
  \else
    \let\!SCnext=\!SCdcheckforRP
  \fi
  \!SCnext}
\def\!SCccheckforRP{%
  \!ifnextchar{p}{\!cgetreference }
    {\!cgetreference point at {\!xref} {\!yref} }}
\def\!cgetreference point at #1 #2 {%
  \edef\!xref{#1}\edef\!yref{#2}%
  \!xorigin=\!xref\!xunit  \!yorigin=\!yref\!yunit
  \!initinboundscheck % ** See linear.tex
  \ignorespaces}
\def\!SCdcheckforRP{%
  \!ifnextchar{p}{\!dgetreference}%
    {\ignorespaces}}
\def\!dgetreference point at #1 #2 {%
  \!xorigin=#1\relax  \!yorigin=#2\relax
  \ignorespaces}

%  ** \put {OBJECT} [XY] <XDIMEN,YDIMEN> at (XCOORD,YCOORD)
%  **   `[XY]' and `<XDIMEN,YDIMEN>' are optional.
%  **   First OBJECT is placed in an hbox (the "objectbox") and then a
%  **     "reference point" is assigned to the objectbox as follows:
%  **     [1] first, the reference point is taken to be the center of the box;
%  **     [2] next, centering is overridden by the specifications
%  **           X=l -- reference point along the left edge of the objectbox
%  **           X=r -- reference point along the right edge of the objectbox
%  **           Y=b -- reference point along the bottom edge of the objectbox
%  **           Y=B -- reference point along the Baseline of the objectbox
%  **           Y=t -- reference point along the top edge of the objectbox;
%  **     [3] finally the reference point is shifted left by XDIMEN, down
%  **           by YDIMEN  (both default to 0pt).
%  **   The objectbox is placed within PICBOX with its reference point at
%  **     (XCOORD,YCOORD).
%  **   If OBJECT is a saved box, say  box0, you have to write
%  **     \put{\box0}...   or  \put{\copy0}...
%  **   The objectbox is void after the put.
%  **   See Subsection 2.1 of the manual.
\long\def\put#1#2 at #3 #4 {%
  \!setputobject{#1}{#2}%
  \!xpos=\!M{#3}\!xunit  \!ypos=\!M{#4}\!yunit
  \!rotateaboutpivot\!xpos\!ypos%
  \advance\!xpos -\!xorigin  \advance\!xpos -\!xshift
  \advance\!ypos -\!yorigin  \advance\!ypos -\!yshift
  \kern\!xpos\raise\!ypos\box\!putobject\kern-\!xpos%
  \!doaccounting\ignorespaces}

%  **   \multiput etc.  Like  \put.  The objectbox is not voided until the
%  **     termininating /, and is placed repeatedly with:
%  **     XCOORD YCOORD -- the objectbox is put down with its reference point
%  **       at (XCOORD,YCOORD);
%  **     *N DXCOORD DYCOORD -- each of N times the current
%  **       (xcoord,ycoord) is incremented by (DXCOORD,DYCOORD), and the
%  **       objectbox is put down with its reference point at (xcoord,ycoord)
%  **       (This specification has to follow an XCOORD YCOORD pair)
%  **     See Subsection 2.2 of the manual.
\long\def\multiput #1#2 at {%
  \!setputobject{#1}{#2}%
  \!ifnextchar"{\!putfromfile}{\!multiput}}
\def\!putfromfile"#1"{%
  \expandafter\!multiput \input #1 /}
\def\!multiput{%
  \futurelet\!nextchar\!!multiput}
\def\!!multiput{%
  \if *\!nextchar
    \def\!nextput{\!alsoby}%
  \else
    \if /\!nextchar
      \def\!nextput{\!finishmultiput}%
    \else
      \def\!nextput{\!alsoat}%
    \fi
  \fi
  \!nextput}
\def\!finishmultiput/{%
  \setbox\!putobject=\hbox{}%
  \ignorespaces}

%  **   \!alsoat XCOORD YCOORD
%  **     The objectbox is put down with reference point at XCOORD,YCOORD
\def\!alsoat#1 #2 {%
  \!xpos=\!M{#1}\!xunit  \!ypos=\!M{#2}\!yunit
  \!rotateaboutpivot\!xpos\!ypos%
  \advance\!xpos -\!xorigin  \advance\!xpos -\!xshift
  \advance\!ypos -\!yorigin  \advance\!ypos -\!yshift
  \kern\!xpos\raise\!ypos\copy\!putobject\kern-\!xpos%
  \!doaccounting
  \!multiput}

% **   \!alsoby*N DXCOORD DYCOORD
% **     N times, the current (XCOORD,YCOORD) is advanced by (DXCOORD,DYCOORD),
% **     and the current (shifted, oriented) OBJECT is put down.
\def\!alsoby*#1 #2 #3 {%
  \!dxpos=\!M{#2}\!xunit \!dypos=\!M{#3}\!yunit
  \!rotateonly\!dxpos\!dypos
  \!ntemp=#1%
  \!!loop\ifnum\!ntemp>0
    \advance\!xpos by \!dxpos  \advance\!ypos by \!dypos
    \kern\!xpos\raise\!ypos\copy\!putobject\kern-\!xpos%
    \advance\!ntemp by -1
  \repeat
  \!doaccounting
  \!multiput}

% **  \accountingoff : Suspends PiCTeX's accounting of the aggregate
% **    size of the picture box.
% **  \accounting on : Reinstates accounting.
% **  See Subsection 8.2 of the manual.
\def\accountingon{\def\!doaccounting{\!!doaccounting}\ignorespaces}
\def\accountingoff{\def\!doaccounting{}\ignorespaces}
\accountingon
\def\!!doaccounting{%
  \!xtemp=\!xpos
  \!ytemp=\!ypos
  \ifdim\!xtemp<\!xleft
     \!xleft=\!xtemp
  \fi
  \advance\!xtemp by  \!wd
  \ifdim\!xright<\!xtemp
    \!xright=\!xtemp
  \fi
  \advance\!ytemp by -\!dp
  \ifdim\!ytemp<\!ybot
    \!ybot=\!ytemp
  \fi
  \advance\!ytemp by  \!dp
  \advance\!ytemp by  \!ht
  \ifdim\!ytemp>\!ytop
    \!ytop=\!ytemp
  \fi}

\long\def\!setputobject#1#2{%
  \setbox\!putobject=\hbox{#1}%
  \!ht=\ht\!putobject  \!dp=\dp\!putobject  \!wd=\wd\!putobject
  \wd\!putobject=\!zpt
  \!xshift=.5\!wd   \!yshift=.5\!ht   \advance\!yshift by -.5\!dp
  \edef\!putorientation{#2}%
  \expandafter\!SPOreadA\!putorientation[]\!nil%
  \expandafter\!SPOreadB\!putorientation<\!zpt,\!zpt>\!nil\ignorespaces}

\def\!SPOreadA#1[#2]#3\!nil{\!etfor\!orientation:=#2\do\!SPOreviseshift}

\def\!SPOreadB#1<#2,#3>#4\!nil{\advance\!xshift by -#2\advance\!yshift by -#3}

\def\!SPOreviseshift{%
  \if l\!orientation
    \!xshift=\!zpt
  \else
    \if r\!orientation
      \!xshift=\!wd
    \else
      \if b\!orientation
	\!yshift=-\!dp
      \else
	\if B\!orientation
	  \!yshift=\!zpt
	\else
	  \if t\!orientation
	    \!yshift=\!ht
	  \fi
	\fi
      \fi
    \fi
  \fi}


%  **  \!dimenput{OBJECT} <XDIMEN,YDIMEN> [XY] (XLOC,YLOC)
%  **    This is an internal put routine, similar to \put, except that
%  **    XLOC=distance right from reference point, YLOC=distance up from
%  **    reference point. XLOC and YLOC are dimensions, so this routine
%  **    is completely independent of the current coordinate system.
%  **    This routine does NOT do ROTATIONS.
\long\def\!dimenput#1#2(#3,#4){%
  \!setputobject{#1}{#2}%
  \!xpos=#3\advance\!xpos by -\!xshift
  \!ypos=#4\advance\!ypos by -\!yshift
  \kern\!xpos\raise\!ypos\box\!putobject\kern-\!xpos%
  \!doaccounting\ignorespaces}


%  ** The following macros permit the picture drawing routines to be used
%  ** either in the default "coordinate mode", or in "dimension mode".
%  **   In coordinate mode  \!M(1.5,\!xunit)    expands to  1.5\!xunit
%  **   In dimension  mode  \!M(1.5pt,\!xunit)  expands to  1.5pt
%  ** Dimension mode is useful in coding macros.
%  ** Any special purpose picture macro that sets dimension mode should
%  ** reset coordinate mode before completion.
%  ** See Subsection 9.2 of the manual.
\def\!setdimenmode{%
  \let\!M=\!M!!\ignorespaces}
\def\!setcoordmode{%
  \let\!M=\!M!\ignorespaces}
\def\!ifcoordmode{%
  \ifx \!M \!M!}
\def\!ifdimenmode{%
  \ifx \!M \!M!!}
\def\!M!#1#2{#1#2}
\def\!M!!#1#2{#1}
\!setcoordmode
\let\setdimensionmode=\!setdimenmode
\let\setcoordinatemode=\!setcoordmode
%  ** \Xdistance{XCOORD}, \Ydistance{YCOORD}  are the horizontal and
%  **   vertical distances from the origin (0,0) to the point
%  **   (XCOORD,YCOORD)  in the current coordinate system.
%  ** See Subsection 9.2 of the manual.
\def\Xdistance#1{%
  \!M{#1}\!xunit
  \ignorespaces}
\def\Ydistance#1{%
  \!M{#1}\!yunit
  \ignorespaces}
% ** The following macros -- \stack, \line, and \Lines -- are useful for
% **   annotating PiCtures. They can be used outside the \beginpicture ...
% **   \endpicture environment.
% ** \stack [POSITIONING] <LEADING> {VALUESLIST}
% ** Builds a vertical stack of the values in VALUESLIST. Values in
% ** VALUESLIST are separated by commas.  In the resulting stack, values are
% ** centered by default, and positioned flush left (right) if
% ** POSITIONING = l (r).  Values are separated vertically by LEADING,
% ** which defaults to \stackleading.
% ** See Subsection 2.3 of the manual.
\def\stack{%
  \!ifnextchar[{\!stack}{\!stack[c]}}
\def\!stack[#1]{%
  \let\!lglue=\hfill \let\!rglue=\hfill
  \expandafter\let\csname !#1glue\endcsname=\relax
  \!ifnextchar<{\!!stack}{\!!stack<\stackleading>}}
\def\!!stack<#1>#2{%
  \vbox{\def\!valueslist{}\!ecfor\!value:=#2\do{%
    \expandafter\!rightappend\!value\withCS{\\}\to\!valueslist}%
    \!lop\!valueslist\to\!value
    \let\\=\cr\lineskiplimit=\maxdimen\lineskip=#1%
    \baselineskip=-1000pt\halign{\!lglue##\!rglue\cr \!value\!valueslist\cr}}%
  \ignorespaces}
% ** \lines [POSITIONING] {LINES}
% ** Builds a vertical array of the lines in LINES. Each line in LINES
% ** is terminated by a \cr.  In the resulting array, lines are
% ** centered by default, and positioned flush left (right) if
% ** POSITIONING = l (r).  The lines in the array are subject to TeX's
% ** usual spacing rules: in particular the baselines are ordinarily an equal
% ** distance apart. The baseline of the array is the baseline of the
% ** the bottom line.
% ** See Subsection 2.3 of the manual.
\def\lines{%
  \!ifnextchar[{\!lines}{\!lines[c]}}
\def\!lines[#1]#2{%
  \let\!lglue=\hfill \let\!rglue=\hfill
  \expandafter\let\csname !#1glue\endcsname=\relax
  \vbox{\halign{\!lglue##\!rglue\cr #2\crcr}}%
  \ignorespaces}
% ** \Lines [POSITIONING] {LINES}
% ** Like \lines, but the baseline of the array is the baseline of the
% ** top line.  See Subsection 2.3 of the manual.
\def\Lines{%
  \!ifnextchar[{\!Lines}{\!Lines[c]}}
\def\!Lines[#1]#2{%
  \let\!lglue=\hfill \let\!rglue=\hfill
  \expandafter\let\csname !#1glue\endcsname=\relax
  \vtop{\halign{\!lglue##\!rglue\cr #2\crcr}}%
  \ignorespaces}

% *********************************************
% *** PLOTTING (Things to do with plotting) ***
% *********************************************

% **  User commands
% **    \setplotsymbol ({PLOTSYMBOL} [ORIENTATION] <XSHIFT,YSHIFT>)
% **    \savelinesandcurves on "FILE_NAME"
% **    \dontsavelinesandcurves
% **    \writesavefile {MESSAGE}
% **    \replot {FILE_NAME}

% **  Internal command
% **    \!plot(XDIMEN,YDIMEN)

% **  \setplotsymbol ({PLOTSYMBOL} [ ] < , >)
% **  Save PLOTSYMBOL away in an hbox for use with curve plotting routines
% **  See Subsection 5.2 of the manual.
\def\setplotsymbol(#1#2){%
  \!setputobject{#1}{#2}
  \setbox\!plotsymbol=\box\!putobject%
  \!plotsymbolxshift=\!xshift
  \!plotsymbolyshift=\!yshift
  \ignorespaces}

\setplotsymbol({\fiverm .})%       ** initialize plotsymbol


% **  \!plot is either \!!plot (when no lines and curves are being saved) or
% **                   \!!!plot (when   lines and curves are being saved)

% **  \!!plot(XDIMEN,YDIMEN)
% **  Places the current plotsymbol a horizontal distance=XDIMEN-xorigin
% **    and a vertical distance=YDIMEN-yorigin from the current
% **    reference point.
\def\!!plot(#1,#2){%
  \!dimenA=-\!plotxorigin \advance \!dimenA by #1%    ** over
  \!dimenB=-\!plotyorigin \advance \!dimenB by #2%    ** up
  \kern\!dimenA\raise\!dimenB\copy\!plotsymbol\kern-\!dimenA%
  \ignorespaces}

% **  \!!!plot(XDIMEN,YDIMEN)
% **  Like \!!plot, but also saves the plot location in units of
% **    scaled point, on file `replotfile'
\def\!!!plot(#1,#2){%
  \!dimenA=-\!plotxorigin \advance \!dimenA by #1%    ** over
  \!dimenB=-\!plotyorigin \advance \!dimenB by #2%    ** up
  \kern\!dimenA\raise\!dimenB\copy\!plotsymbol\kern-\!dimenA%
  \!countE=\!dimenA
  \!countF=\!dimenB
  \immediate\write\!replotfile{\the\!countE,\the\!countF.}%
  \ignorespaces}

% ** \savelinesandcurves on "FILE_NAME"
% **   Switch to save locations used for plotting lines and curves
% **   (No advantage in saving locations for solid lines; however
% **   replotting curve locations speeds things up by a factor of about 4.
% ** \dontsavelinesandcurves
% **   Terminates \savelinesandcurves. The default.
% ** See Subsection 5.6 of the manual.
\def\savelinesandcurves on "#1" {%
  \immediate\closeout\!replotfile
  \immediate\openout\!replotfile=#1%
  \let\!plot=\!!!plot}
\def\dontsavelinesandcurves {%
  \let\!plot=\!!plot}
\dontsavelinesandcurves
% ** \writesavefile {MESSAGE}
% ** The message is preceded by a "%", so that it won't interfere
% ** with replotting.
% ** See Subsection 5.6 of the manual.
{\catcode`\%=11\xdef\!Commentsignal{%}}
\def\writesavefile#1 {%
  \immediate\write\!replotfile{\!Commentsignal #1}%
  \ignorespaces}
% ** \replot "FILE_NAME"
% **   Replots the locations saved earlier under \savelinesandcurves
% **   on "FILE_NAME"
% ** See Subsection 5.6 of the manual.
\def\replot"#1" {%
  \expandafter\!replot\input #1 /}
\def\!replot#1,#2. {%
  \!dimenA=#1sp
  \kern\!dimenA\raise#2sp\copy\!plotsymbol\kern-\!dimenA
  \futurelet\!nextchar\!!replot}
\def\!!replot{%
  \if /\!nextchar
    \def\!next{\!finish}%
  \else
    \def\!next{\!replot}%
  \fi
  \!next}
% **************************************************
% ***  PYTHAGORAS  (Euclidean distance function) ***
% **************************************************
% ** User command:
% **   \placehypotenuse for <dimension1> and <dimension2> in <register>
% ** Internal command:
% **   \!Pythag{X}{Y}{Z}
% **     Input X,Y are dimensions, or dimension registers.
% **     Output Z == sqrt(X**2+Y**2) must be a dimension register.
% **     Assumes that |X|+|Y| < 2048pt (about 28in).

% ** Without loss of generality, suppose  x>0, y>0.  Put s = x+y,
% **   z = sqrt(x**2+y**2). Then  z = s*f,  where  f = sqrt(t**2 + (1-t)**2)
% **   = sqrt((1+tau**2)/2), where  t = x/s  and  tau = 2(t-1/2) .

% ** Uses the \!divide macro (which uses registers \!dimenA--\!dimenD.
% ** Uses the \!removept macro   (e.g., 123.45pt --> 123.45)
% ** Uses registers \!dimenE--\!dimenI.
\def\!Pythag#1#2#3{%
  \!dimenE=#1\relax
  \ifdim\!dimenE<\!zpt
    \!dimenE=-\!dimenE
  \fi%                                            ** dimE = |x|
  \!dimenF=#2\relax
  \ifdim\!dimenF<\!zpt
    \!dimenF=-\!dimenF
  \fi%                                            ** dimF = |y|
  \advance \!dimenF by \!dimenE%                  ** dimF = s = |x|+|y|
  \ifdim\!dimenF=\!zpt
    \!dimenG=\!zpt%                               ** dimG = z = sqrt(x**2+y**2)
  \else
    \!divide{8\!dimenE}\!dimenF\!dimenE%          ** now dimE = 8t = (8|x|)/s
    \advance\!dimenE by -4pt%                     ** 8tau = (8t-4)*2
      \!dimenE=2\!dimenE%                         **   (tau = 2*t - 1)
    \!removept\!dimenE\!!t%                       ** 8tau, without "pt"
    \!dimenE=\!!t\!dimenE%                        ** (8tau)**2, in pts
    \advance\!dimenE by 64pt%                     ** u = [64 + (8tau)**2]/2
    \divide \!dimenE by 2%                        **   [u = (8f)**2]
    \!dimenH=7pt%                                 ** initial guess g at sqrt(u)
    \!!Pythag\!!Pythag\!!Pythag%                  ** 3 iterations give sqrt(u)
    \!removept\!dimenH\!!t%                       ** 8f=sqrt(u), without "pt"
    \!dimenG=\!!t\!dimenF%                        ** z = (8f)*s/8
    \divide\!dimenG by 8
  \fi
  #3=\!dimenG
  \ignorespaces}
\def\!!Pythag{%                                   ** Newton-Raphson for sqrt
  \!divide\!dimenE\!dimenH\!dimenI%               ** v = u/g
  \advance\!dimenH by \!dimenI%                   ** g <-- (g + u/g)/2
    \divide\!dimenH by 2}
% **  \placehypotenuse for <XI> and <ETA> in <ZETA>
% **  See Subsection 9.3 of the manual.
\def\placehypotenuse for <#1> and <#2> in <#3> {%
  \!Pythag{#1}{#2}{#3}}

% **********************************************
% *** QUADRATIC ARC  (Draws a quadratic arc) ***
% **********************************************

% **  Internal command
% **    \!qjoin (XCOORD1,YCOORD1) (XCOORD2,YCOORD2)

% **  \!qjoin (XCOORD1,YCOORD1) (XCOORD2,YCOORD2)
% **  Draws an arc starting at the (last) point specified by the most recent
% **  \!qjoin, or \!ljoin, or \!start  and passing through (X_1,Y_1), (X_2,Y_2).
% **  Uses quadratic interpolation in both  x  and  y:
% **    x(t), 0 <= t <= 1, interpolates  x_0, x_1, x_2  at  t=0, .5, 1
% **    y(t), 0 <= t <= 1, interpolates  y_0, y_1, y_2  at  t=0, .5, 1

\def\!qjoin (#1,#2) (#3,#4){%
  \advance\!intervalno by 1
  \!ifcoordmode
    \edef\!xmidpt{#1}\edef\!ymidpt{#2}%
  \else
    \!dimenA=#1\relax \edef\!xmidpt{\the\!dimenA}%
    \!dimenA=#2\relax \edef\!xmidpt{\the\!dimenA}%
  \fi
  \!xM=\!M{#1}\!xunit  \!yM=\!M{#2}\!yunit   \!rotateaboutpivot\!xM\!yM
  \!xE=\!M{#3}\!xunit  \!yE=\!M{#4}\!yunit   \!rotateaboutpivot\!xE\!yE
%
% ** Find coefficients for x(t)=a_x + b_x*t + c_x*t**2
  \!dimenA=\!xM  \advance \!dimenA by -\!xS%   ** dimA = I = xM - xS
  \!dimenB=\!xE  \advance \!dimenB by -\!xM%   ** dimB = II = xE-xM
  \!xB=3\!dimenA \advance \!xB by -\!dimenB%   ** b=3I-II
  \!xC=2\!dimenB \advance \!xC by -2\!dimenA%  ** c=2(II-I)
%
% ** Find coefficients for y(t)=y_x + b_y*t + c_y*t**2
  \!dimenA=\!yM  \advance \!dimenA by -\!yS%
  \!dimenB=\!yE  \advance \!dimenB by -\!yM%
  \!yB=3\!dimenA \advance \!yB by -\!dimenB%
  \!yC=2\!dimenB \advance \!yC by -2\!dimenA%
%
% ** Use Simpson's rule to calculate arc length over [0,1/2]:
% **   arc length = 1/2[1/6 f(0) + 4/6 f(1/4) + 1/6 f(1/2)]
% ** with f(t) = sqrt(x'(t)**2 + y'(t)**2).
  \!xprime=\!xB  \!yprime=\!yB%          ** x'(t) = b + 2ct
  \!dxprime=.5\!xC  \!dyprime=.5\!yC%    ** dt=1/4 ==> dx'(t) = c/2
  \!getf \!midarclength=\!dimenA
  \!getf \advance \!midarclength by 4\!dimenA
  \!getf \advance \!midarclength by \!dimenA
  \divide \!midarclength by 12
%
% ** Get arc length over [0,1].
  \!arclength=\!dimenA
  \!getf \advance \!arclength by 4\!dimenA
  \!getf \advance \!arclength by \!dimenA
  \divide \!arclength by 12%             ** Now have arc length over [1/2,1]
  \advance \!arclength by \!midarclength
  \global\advance \totalarclength by \!arclength
%
%
% ** Check to see if there's anything to plot in this interval
  \ifdim\!distacross>\!arclength
    \advance \!distacross by -\!arclength%   ** nothing
%
  \else
    \!initinverseinterp%  ** initialize for inverse interpolation on arc length
    \loop\ifdim\!distacross<\!arclength%     ** loop over points on arc
      \!inverseinterp%    ** find  t  such that arc length[0,t] = distacross,
%                         **   using inverse quadratic interpolation
%                         ** now evaluate x(t)=(c*t + b)*t + a
      \!xpos=\!t\!xC \advance\!xpos by \!xB
	\!xpos=\!t\!xpos \advance \!xpos by \!xS
%                                             ** evaluate y(t)
      \!ypos=\!t\!yC \advance\!ypos by \!yB
	\!ypos=\!t\!ypos \advance \!ypos by \!yS
      \!plotifinbounds%                       ** plot point if in bounds
      \advance\!distacross \plotsymbolspacing%** advance arc length for next pt
      \!advancedashing%                       ** see "linear"
    \repeat
%
    \advance \!distacross by -\!arclength%    ** prepare for next interval
  \fi
%
  \!xS=\!xE%              ** shift ending points to starting points
  \!yS=\!yE
  \ignorespaces}


% ** \!getf -- Calculates sqrt(x'(t)**2 + y'(t)**2) and advances
% **   x'(t) and y'(t)
\def\!getf{\!Pythag\!xprime\!yprime\!dimenA%
  \advance\!xprime by \!dxprime
  \advance\!yprime by \!dyprime}


% ** \!initinverseinterp -- initializes for inverse quadratic interpolation
% ** of arc length provided  1/3 < midarclength/arclength < 2/3; otherwise
% ** initializes for inverse linear interpolation.
\def\!initinverseinterp{%
  \ifdim\!arclength>\!zpt
    \!divide{8\!midarclength}\!arclength\!dimenE% ** dimE=8w=8r/s, where  r
%                                               **  = midarclength, s=arclength
% **  Test for  w  out of range:  w<1/3  or w>2/3
    \ifdim\!dimenE<\!wmin \!setinverselinear
    \else
      \ifdim\!dimenE>\!wmax \!setinverselinear
      \else%                                    ** w  in range: initialize
	\def\!inverseinterp{\!inversequad}\ignorespaces
%
% **     Calculate the coefficients  \!beta  and  \!gamma  of the quadratic
% **                    t = \!beta*v + \!gamma*v**2
% **     taking the values  t=0, 1/2, 1  at  v=0, w==r/s, 1  respectively:
% **        \!beta = (1/2 - w**2)/[w(1-w)]
% **        \!gamma = 1 - beta.
%
	 \!removept\!dimenE\!Ew%           **  8w, without "pt"
	 \!dimenF=-\!Ew\!dimenE%           **  -(8w)**2
	 \advance\!dimenF by 32pt%         **  32 - (8w)**2
	 \!dimenG=8pt
	 \advance\!dimenG by -\!dimenE%    **  8 - 8w
	 \!dimenG=\!Ew\!dimenG%            **  (8w)*(8-8w)
	 \!divide\!dimenF\!dimenG\!beta%   **  beta = (32-(8w)**2)/(8w(8-8w))
%                                          **       = (1/2 - w**2)/(w(1-w))
	 \!gamma=1pt
	 \advance \!gamma by -\!beta%      **  gamma = 1-beta
      \fi%       ** end of the \ifdim\!dimenE>\!wmax
    \fi%         ** end of the \ifdim\!dimenE<\!wmin
  \fi%           ** end of the \ifdim\!arclength>\!zpt
  \ignorespaces}


% ** For 0 <= t <= 1, let AL(t) = arclength[0,t]/arclength[0,1]; note
% ** AL(0)=0, AL(1/2)=midarclength/arclength, AL(1)=1.  This routine
% ** calculates an approximation to AL^{-1}(distance across/arclength),
% ** using the assumption that AL^{-1} is quadratic.  Specifically,
% ** it finds  t  such that
% **    AL^{-1}(v) =. t = v*(\!beta + \!gamma*v)
% ** where  \!beta  and  \!gamma  are set by \!initinv, and where
% ** v=distance across/arclength
\def\!inversequad{%
  \!divide\!distacross\!arclength\!dimenG%   ** dimG = v = distacross/arclength
  \!removept\!dimenG\!v%                     ** v, without "pt"
  \!dimenG=\!v\!gamma%                       ** gamma*v
  \advance\!dimenG by \!beta%                ** beta + gamma*v
  \!dimenG=\!v\!dimenG%                      ** t = v*(beta + gamma*v)
  \!removept\!dimenG\!t}%                    ** t, without "pt"


% ** When  w <= 1/3  or  w >= 2/3, the following routine writes (using
% ** plain TEK's \wlog command) a warning message on the user's log file,
% ** and initializes for inverse linear interpolation on arc length.
\def\!setinverselinear{%
  \def\!inverseinterp{\!inverselinear}%
  \divide\!dimenE by 8 \!removept\!dimenE\!t
  \!countC=\!intervalno \multiply \!countC 2
  \!countB=\!countC     \advance \!countB -1
  \!countA=\!countB     \advance \!countA -1
  \wlog{\the\!countB th point (\!xmidpt,\!ymidpt) being plotted
    doesn't lie in the}%
  \wlog{ middle third of the arc between the \the\!countA th
    and \the\!countC th points:}%
  \wlog{ [arc length \the\!countA\space to \the\!countB]/[arc length
    \the \!countA\space to \the\!countC]=\!t.}%
  \ignorespaces}

% **  Inverse linear interpolation
\def\!inverselinear{%
  \!divide\!distacross\!arclength\!dimenG
  \!removept\!dimenG\!t}

% **************************************
% **  ROTATIONS  (Handles rotations) ***
% **************************************

% ** User commands
% **   \startrotation [by COS_OF_ANGLE SIN_OF_ANGLE] [about XPIVOT YPIVOT]
% **   \stoprotation
% **   \startrotation [by COS_OF_ANGLE SIN_OF_ANGLE] [about XPIVOT YPIVOT]
% ** Future (XCOORD,YCOORD)'s will be rotated about (XPIVOT,YPIVOT)
% ** by the angle with the give COS and SIN. Both fields are optional.
% ** [COS,SIN] defaults to previous value, or (1,0).
% ** (XPIVOT,YPIVOT) defaults to previous value, or (0,0)
% ** You can't change the coordinate system in the scope of a rotation.
% ** See Subsection 9.1 of the manual.
\def\startrotation{%
  \let\!rotateaboutpivot=\!!rotateaboutpivot
  \let\!rotateonly=\!!rotateonly
  \!ifnextchar{b}{\!getsincos }%
    {\!getsincos by {\!cosrotationangle} {\!sinrotationangle} }}
\def\!getsincos by #1 #2 {%
  \edef\!cosrotationangle{#1}%
  \edef\!sinrotationangle{#2}%
  \!ifcoordmode
    \let\!ROnext=\!ccheckforpivot
  \else
    \let\!ROnext=\!dcheckforpivot
  \fi
  \!ROnext}
\def\!ccheckforpivot{%
  \!ifnextchar{a}{\!cgetpivot}%
    {\!cgetpivot about {\!xpivotcoord} {\!ypivotcoord} }}
\def\!cgetpivot about #1 #2 {%
  \edef\!xpivotcoord{#1}%
  \edef\!ypivotcoord{#2}%
  \!xpivot=#1\!xunit  \!ypivot=#2\!yunit
  \ignorespaces}
\def\!dcheckforpivot{%
  \!ifnextchar{a}{\!dgetpivot}{\ignorespaces}}
\def\!dgetpivot about #1 #2 {%
  \!xpivot=#1\relax  \!ypivot=#2\relax
  \ignorespaces}

% ** Following terminates rotation.
% ** See Subsection 9.1 of the manual.
\def\stoprotation{%
  \let\!rotateaboutpivot=\!!!rotateaboutpivot
  \let\!rotateonly=\!!!rotateonly
  \ignorespaces}

% ** !!rotateaboutpivot{XREG}{YREG}
% ** XREG <-- xpvt + cos(angle)*(XREG-xpvt) - sin(angle)*(YREG-ypvt)
% ** YREG <-- ypvt + cos(angle)*(YREG-ypvt) + sin(angle)*(XREG-xpvt)
% ** XREG,YREG are dimension registers. Can't be \!dimenA to \!dimenD
\def\!!rotateaboutpivot#1#2{%
  \!dimenA=#1\relax  \advance\!dimenA -\!xpivot
  \!dimenB=#2\relax  \advance\!dimenB -\!ypivot
  \!dimenC=\!cosrotationangle\!dimenA
    \advance \!dimenC -\!sinrotationangle\!dimenB
  \!dimenD=\!cosrotationangle\!dimenB
    \advance \!dimenD  \!sinrotationangle\!dimenA
  \advance\!dimenC \!xpivot  \advance\!dimenD \!ypivot
  #1=\!dimenC  #2=\!dimenD
  \ignorespaces}
% ** \!!rotateonly{XREG}{YREG}
% ** Like \!!rotateaboutpivot, but with a pivot of  (0,0)
\def\!!rotateonly#1#2{%
  \!dimenA=#1\relax  \!dimenB=#2\relax
  \!dimenC=\!cosrotationangle\!dimenA
    \advance \!dimenC -\!rotsign\!sinrotationangle\!dimenB
  \!dimenD=\!cosrotationangle\!dimenB
    \advance \!dimenD  \!rotsign\!sinrotationangle\!dimenA
  #1=\!dimenC  #2=\!dimenD
  \ignorespaces}
\def\!rotsign{}
\def\!!!rotateaboutpivot#1#2{\relax}
\def\!!!rotateonly#1#2{\relax}
\stoprotation
\def\!reverserotateonly#1#2{%
  \def\!rotsign{-}%
  \!rotateonly{#1}{#2}%
  \def\!rotsign{}%
  \ignorespaces}

% **********************************
% *** SHADING  (Handles shading) ***
% **********************************
% **  User commands
% **    \setshadegrid [span <SPAN>] [point at XSHADE YSHADE]
% **    \setshadesymbol [<LS, RS, BS, TS>] ({SHADESYMBOL}
% **      <XDIMEN,YDIMEN> [ORIENTATION])
% **  Internal commands:
% **    \!startvshade  (xS,ybS,ytS)
% **    \!starthshade  (yS,xlS,xrS)
% **    \!lshade [<LS,RS,BS,TS>]
% **       ** when shading vertically:
% **       [the region from (xS,ybS,ytS) to] (xE,ybE,ytE)
% **       ** when shading horizontally:
% **       [the region from (yS,xlS,xrS) to] (yE,xlE,xrE)
% **    \!qshade [<LS,RS,BS,TS>]
% **       ** when shading vertically:
% **       [the region from (xS,ybS,ytS) to] (xM,ybM,ytM)  (xE,ybE,ytE)
% **       ** when shading horizontally:
% **       [the region from (yS,xlS,xrS) to] (yM,xlM,xrM)  (yE,xlE,xrE)
% **    \!lattice{ANCHOR}{SPAN}{LOCATION}{INDEX}{LATTICE LOCATION}
% **    \!override{NOMINAL DIMEN}{REPLACEMENT DIMEN}{DIMEN}

% **  The shading routine can operate either in a "vertical mode" or a
% **  "horizontal mode".  In vertical mode, the region to be shaded is specified
% **  in the form
% **                 {(x,y): xl <= x <= xr  &  yb(x) <= y <= yt(x)}
% **  where  yb  and  yt  are functions of  x.  In horizontal mode, the region
% **  is specified in the form
% **                 {(x,y): yb <= y <= yt  &  xl(y) <= x <= xr(y)}.
% **  The functions  yb  and  yt  may be either both linear or both quadratic;
% **  similarly for  xl  and  xr.  A region with say, piecewise quadratic bottom
% **  and top boundaries, can be shaded by consecutive (vertical) \!qshades,
% **  proceeding from left to right.  Similarly, a region with piecewise
% **  quadratic left and right boundaries can be shaded by consecutive
% **  (horizontal) \!qshades, proceeding from bottom to top.  More complex
% **  regions can be shaded by partitioning them into appropriate subregions,
% **  and shading those.

% **  Shading is accomplished by placing a user-selected shading symbol at
% **  those points of a regular grid which fall within the region to be
% **  shaded.  This region can be "shrunk" so that a largish shading symbol
% **  will not extend outside it.  Shrinking is accomplished by specifying
% **  shrinkages for the left, right, bottom, and top boundaries, in a manner
% **  discussed further below.
% **  \shades and \!joins MUST NOT be intermingled.  Finish drawing a curve
% **  before starting to shade a region, and finish shading a region before
% **  starting to draw a curve.

% **  \setshadegrid [span <SPAN>] [point at XSHADE YSHADE]
% **  The shading symbol is placed down on the points of a grid centered
% **  at the coordinate point (XSHADE,YSHADE).  The grid points are of the
% **  form (j*SPAN,k*SPAN), with  j+k  even.  SPAN is specified
% **  as a dimension.
% **  (XSHADE,YSHADE) defaults to previous (XSHADE,YSHADE) (or (0,0) if none)
% **  SPAN defaults to previous span (or 5pt if none)
% **  See Subsection 7.2 of the manual.
\def\setshadegrid{%
  \!ifnextchar{s}{\!getspan }
    {\!getspan span <\!dshade>}}
\def\!getspan span <#1>{%
  \!dshade=#1\relax
  \!ifcoordmode
    \let\!GRnext=\!GRccheckforAP
  \else
    \let\!GRnext=\!GRdcheckforAP
  \fi
  \!GRnext}
\def\!GRccheckforAP{%
  \!ifnextchar{p}{\!cgetanchor }
    {\!cgetanchor point at {\!xshadesave} {\!yshadesave} }}
\def\!cgetanchor point at #1 #2 {%
  \edef\!xshadesave{#1}\edef\!yshadesave{#2}%
  \!xshade=\!xshadesave\!xunit  \!yshade=\!yshadesave\!yunit
  \ignorespaces}
\def\!GRdcheckforAP{%
  \!ifnextchar{p}{\!dgetanchor}%
    {\ignorespaces}}
\def\!dgetanchor point at #1 #2 {%
  \!xshade=#1\relax  \!yshade=#2\relax
  \ignorespaces}
% **  \setshadesymbol  [<LS, RS, BS, TS>] ({SHADESYMBOL}
% **    <XDIMEN,YDIMEN> [ORIENTATION])
% **  Saves SHADESYMBOL away in an hbox for use with shading routines.
% **  A shade symbol will not be plotted if its plot position comes within
% **    distance LS of the left boundary,  RS of the right boundary,  TS of the
% **    top boundary,  BS of the bottom boundary.  These parameters have
% **    default values that should work in most cases (see below).
% **    To override a default value, specify the replacement value
% **    in the appropriate subfield of the shrinkages field.
% **    0pt may be coded as  "z" (without the quotes).  To accept a
% **    default value, leave the field empty.  Thus
% **      [,z,,5pt]  sets  LS=default, RS=0pt, BS=default, TS=5pt .
% **    Skipping the shrinkages field accepts all the defaults.
% **  See Subsection 7.1 of the manual.
\def\setshadesymbol{%
  \!ifnextchar<{\!setshadesymbol}{\!setshadesymbol<,,,> }}
\def\!setshadesymbol <#1,#2,#3,#4> (#5#6){%
% **  set the shadesymbol
  \!setputobject{#5}{#6}%
  \setbox\!shadesymbol=\box\!putobject%
  \!shadesymbolxshift=\!xshift \!shadesymbolyshift=\!yshift
%
% **  set the shrinkages
  \!dimenA=\!xshift \advance\!dimenA \!smidge% ** default LS = xshift - smidge
  \!override\!dimenA{#1}\!lshrinkage%
  \!dimenA=\!wd \advance \!dimenA -\!xshift%   ** default RS = width - xshift
    \advance\!dimenA \!smidge%                                  - smidge
    \!override\!dimenA{#2}\!rshrinkage
  \!dimenA=\!dp \advance \!dimenA \!yshift%    ** default BS = depth + yshift
    \advance\!dimenA \!smidge%                                  - smidge
    \!override\!dimenA{#3}\!bshrinkage
  \!dimenA=\!ht \advance \!dimenA -\!yshift%   ** default TS = height - yshift
    \advance\!dimenA \!smidge%                                  - smidge
    \!override\!dimenA{#4}\!tshrinkage
  \ignorespaces}
\def\!smidge{-.2pt}%
% ** \!override{NOMINAL DIMEN}{REPLACEMENT DIMEN}{DIMEN}
% ** Overrides the NOMINAL DIMEN by the REPLACEMENT DIMEN to produce DIMEN,
% ** according to the following rules:
% **   REPLACEMENT DIMEN empty: DIMEN <-- NOMINAL DIMEN
% **   REPLACEMENT DIMEN z:     DIMEN <-- 0pt
% **   otherwise:               DIMEN <-- REPLACEMENT DIMEN
% ** DIMEN must be a dimension register
\def\!override#1#2#3{%
  \edef\!!override{#2}%
  \ifx \!!override\empty
    #3=#1\relax
  \else
    \if z\!!override
      #3=\!zpt
    \else
      \ifx \!!override\!blankz
	#3=\!zpt
      \else
	#3=#2\relax
      \fi
    \fi
  \fi
  \ignorespaces}
\def\!blankz{ z}
\setshadesymbol ({\fiverm .})%       ** initialize plotsymbol
%                                    ** \fivesy ^^B  is a small cross

% ** \!startvshade [at] (xS,ybS,ytS)
% ** Initiates vertical shading mode
\def\!startvshade#1(#2,#3,#4){%
  \let\!!xunit=\!xunit%
  \let\!!yunit=\!yunit%
  \let\!!xshade=\!xshade%
  \let\!!yshade=\!yshade%
  \def\!getshrinkages{\!vgetshrinkages}%
  \let\!setshadelocation=\!vsetshadelocation%
  \!xS=\!M{#2}\!!xunit
  \!ybS=\!M{#3}\!!yunit
  \!ytS=\!M{#4}\!!yunit
  \!shadexorigin=\!xorigin  \advance \!shadexorigin \!shadesymbolxshift
  \!shadeyorigin=\!yorigin  \advance \!shadeyorigin \!shadesymbolyshift
  \ignorespaces}

% ** \!starthshade [at] (yS,xlS,xrS)
% ** Initiates horizontal shading mode
\def\!starthshade#1(#2,#3,#4){%
  \let\!!xunit=\!yunit%
  \let\!!yunit=\!xunit%
  \let\!!xshade=\!yshade%
  \let\!!yshade=\!xshade%
  \def\!getshrinkages{\!hgetshrinkages}%
  \let\!setshadelocation=\!hsetshadelocation%
  \!xS=\!M{#2}\!!xunit
  \!ybS=\!M{#3}\!!yunit
  \!ytS=\!M{#4}\!!yunit
  \!shadexorigin=\!xorigin  \advance \!shadexorigin \!shadesymbolxshift
  \!shadeyorigin=\!yorigin  \advance \!shadeyorigin \!shadesymbolyshift
  \ignorespaces}

% **  \!lattice{ANCHOR}{SPAN}{LOCATION}{INDEX}{LATTICE LOCATION}
% **  Consider the lattice with points  ANCHOR + j*SPAN. This routine determines
% **  the index  k  of the smallest lattice point >= LOCATION, and sets
% **  LATTICE LOCATION = ANCHOR + k*SPAN.
% **  INDEX is assumed to be a count register, LATTICE LOCATION a dimen reg.
\def\!lattice#1#2#3#4#5{%
  \!dimenA=#1%                        ** dimA = ANCHOR
  \!dimenB=#2%                        ** dimB = SPAN  (assumed > 0pt)
  \!countB=\!dimenB%                  ** ctB  = SPAN, as a count
%
% ** Determine index of smallest lattice point >= LOCATION
  \!dimenC=#3%                        ** dimC = LOCATION
  \advance\!dimenC -\!dimenA%         ** now dimC = LOCATION-ANCHOR
  \!countA=\!dimenC%                  ** ctA = above, as a count
  \divide\!countA \!countB%           ** now ctA = desired index, if dimC <= 0
  \ifdim\!dimenC>\!zpt
    \!dimenD=\!countA\!dimenB%        ** (tentative k)*span
    \ifdim\!dimenD<\!dimenC%          ** if this is false, ctA = desired index
      \advance\!countA 1 %            ** if true, have to add 1
    \fi
  \fi
%
  \!dimenC=\!countA\!dimenB%          ** lattice location = anchor + ctA*span
    \advance\!dimenC \!dimenA
  #4=\!countA%                        ** the desired index
  #5=\!dimenC%                        ** corresponding lattice location
  \ignorespaces}

% ** \!qshade [with shrinkages] [[LS,RS,BS,TS]]
% ***** during vertical shading:
% **    [the region from (xS,ybS,ytS) to] (xM,ybM,ytM) [and] (xE,ybE,ytE)
% ** Shades the region {(x,y): xS <= x <= xE, yb(x) <= y <= yt(x)}, where
% **   yb is the quadratic thru (xS,ybS) & (xM,ybM) & (xE,ybE)
% **   yt is the quadratic thru (xS,ytS) & (xM,ybM) & (xE,ytE)
% ** xS,ybS,ytS are either given by \!startvshade or carried over
% **   as the ending values of the immediately preceding \!qshade.
% ** For the interpretation of LS, RS, BS, & TS, see \setshadesymbol. The
% **   values set there can be overridden, for the course of this \!qshade
% **   only, in the same manner as overrides are specified for
% **   \setshadesymbol.
% ***** during horizontal shading:
% **    [the region from (yS,xlS,xrS) to] (yM,xlM,xrM) [and] (yE,xlE,xrE)
\def\!qshade#1(#2,#3,#4)#5(#6,#7,#8){%
  \!xM=\!M{#2}\!!xunit
  \!ybM=\!M{#3}\!!yunit
  \!ytM=\!M{#4}\!!yunit
  \!xE=\!M{#6}\!!xunit
  \!ybE=\!M{#7}\!!yunit
  \!ytE=\!M{#8}\!!yunit
  \!getcoeffs\!xS\!ybS\!xM\!ybM\!xE\!ybE\!ybB\!ybC%**Get coefficients B & C for
  \!getcoeffs\!xS\!ytS\!xM\!ytM\!xE\!ytE\!ytB\!ytC%**y=y0 + B(x-X0) + C(x-X0)**2
  \def\!getylimits{\!qgetylimits}%
  \!shade{#1}\ignorespaces}

% ** \!lshade ... (xE,ybE,ytE)
% ** This is like \!qshade, but the top and bottom boundaries are linear,
% ** rather than quadratic.
\def\!lshade#1(#2,#3,#4){%
  \!xE=\!M{#2}\!!xunit
  \!ybE=\!M{#3}\!!yunit
  \!ytE=\!M{#4}\!!yunit
  \!dimenE=\!xE  \advance \!dimenE -\!xS%   ** xE-xS
  \!dimenC=\!ytE \advance \!dimenC -\!ytS%  ** ytE-ytS
  \!divide\!dimenC\!dimenE\!ytB%            ** ytB = (ytE-ytS)/(xE-xS)
  \!dimenC=\!ybE \advance \!dimenC -\!ybS%  ** ybE-ybS
  \!divide\!dimenC\!dimenE\!ybB%            ** ybB = (ybE-ybS)/(xE-xS)
  \def\!getylimits{\!lgetylimits}%
  \!shade{#1}\ignorespaces}

% **  \!getcoeffs{X0}{Y0}{X1}{Y1}{X2}{Y2}{B}{C}
% **  Finds  B  and  C  such that the quadratic  y = Y0 + B(x-X0) + C(x-X0)**2
% **  passes through (X1,Y1) and (X2,Y2):  when X0=0=Y0, the formulas are:
% **                   B = S1 - X1*C,   C = (S2-S1)/X2
% **  with
% **                 S1 = Y1/X1,   S2 = (Y2-Y1)/(X2-X1).
\def\!getcoeffs#1#2#3#4#5#6#7#8{%
  \!dimenC=#4\advance \!dimenC -#2%            ** dimC=Y1-Y0
  \!dimenE=#3\advance \!dimenE -#1%            ** dimE=X1-X0
  \!divide\!dimenC\!dimenE\!dimenF%            ** dimF=S1
  \!dimenC=#6\advance \!dimenC -#4%            ** dimC=Y2-Y1
  \!dimenH=#5\advance \!dimenH -#3%            ** dimH=X2-X1
  \!divide\!dimenC\!dimenH\!dimenG%            ** dimG=S2
  \advance\!dimenG -\!dimenF%                  ** dimG=S2-S1
  \advance \!dimenH \!dimenE%                  ** dimH=X2-X0
  \!divide\!dimenG\!dimenH#8%                  ** C=(S2-S1)/(X2-X0)
  \!removept#8\!t%                             ** C, without "pt"
  #7=-\!t\!dimenE%                             ** -C*(X1-X0)
  \advance #7\!dimenF%                         ** B=S1-C*(X1-X0)
  \ignorespaces}


\def\!shade#1{%
% ** Get LS,RS,BS,TS for this panel
  \!getshrinkages#1<,,,>\!nil% %       ** now effective LS=dimE, RS=dimF,
%                                      **   BS=dimG, TS=dimH
  \advance \!dimenE \!xS%              ** now dimE=xS+LS
  \!lattice\!!xshade\!dshade\!dimenE%  ** set parity=index of left-mst x-lattice
    \!parity\!xpos%                    **   point >= xS+LS, xpos=its location
  \!dimenF=-\!dimenF%                  ** set dimF=xE-RS
    \advance\!dimenF \!xE
%
  \!loop\!not{\ifdim\!xpos>\!dimenF}%  ** loop over x-lattice points <= xE-RS
    \!shadecolumn%
    \advance\!xpos \!dshade%           ** move over to next column
    \advance\!parity 1%                ** increase index of x-point
  \repeat
%
  \!xS=\!xE%                           ** shift ending values to starting values
  \!ybS=\!ybE
  \!ytS=\!ytE
  \ignorespaces}


\def\!vgetshrinkages#1<#2,#3,#4,#5>#6\!nil{%
  \!override\!lshrinkage{#2}\!dimenE
  \!override\!rshrinkage{#3}\!dimenF
  \!override\!bshrinkage{#4}\!dimenG
  \!override\!tshrinkage{#5}\!dimenH
  \ignorespaces}
\def\!hgetshrinkages#1<#2,#3,#4,#5>#6\!nil{%
  \!override\!lshrinkage{#2}\!dimenG
  \!override\!rshrinkage{#3}\!dimenH
  \!override\!bshrinkage{#4}\!dimenE
  \!override\!tshrinkage{#5}\!dimenF
  \ignorespaces}


\def\!shadecolumn{%
  \!dxpos=\!xpos
  \advance\!dxpos -\!xS%            ** dx = x - xS
  \!removept\!dxpos\!dx%            ** ditto, without "pt"
  \!getylimits%                     ** get top and bottom y-values
  \advance\!ytpos -\!dimenH%        ** less TS
  \advance\!ybpos \!dimenG%         ** plus BS
  \!yloc=\!!yshade%                 ** get anchor point for this column
  \ifodd\!parity
     \advance\!yloc \!dshade
  \fi
  \!lattice\!yloc{2\!dshade}\!ybpos%
    \!countA\!ypos%                 ** ypos=smallest y point for this column
  \!dimenA=-\!shadexorigin \advance \!dimenA \!xpos%      ** over
  \loop\!not{\ifdim\!ypos>\!ytpos}% ** loop over ypos <= yt(t)
    \!setshadelocation%             ** vmode: xloc=xpos, yloc=ypos
%                                   ** hmode: xloc=ypos, yloc=xpos
    \!rotateaboutpivot\!xloc\!yloc%
    \!dimenA=-\!shadexorigin \advance \!dimenA \!xloc%    ** over
    \!dimenB=-\!shadeyorigin \advance \!dimenB \!yloc%    ** up
    \kern\!dimenA \raise\!dimenB\copy\!shadesymbol \kern-\!dimenA
    \advance\!ypos 2\!dshade
  \repeat
  \ignorespaces}

\def\!qgetylimits{%
  \!dimenA=\!dx\!ytC
  \advance\!dimenA \!ytB%         ** yt(t)=ytS + dx*(Bt + dx*Ct)
  \!ytpos=\!dx\!dimenA
  \advance\!ytpos \!ytS
  \!dimenA=\!dx\!ybC
  \advance\!dimenA \!ybB%         ** yb(t)=ybS + dx*(Bb + dx*Cb)
  \!ybpos=\!dx\!dimenA
  \advance\!ybpos \!ybS}

\def\!lgetylimits{%
  \!ytpos=\!dx\!ytB%              ** yt(t)=ytS + dx*Bt
  \advance\!ytpos \!ytS
  \!ybpos=\!dx\!ybB%              ** yb(t)=ybS + dx*Bb
  \advance\!ybpos \!ybS}

\def\!vsetshadelocation{%         ** vmode: xloc=xpos, yloc=ypos
  \!xloc=\!xpos
  \!yloc=\!ypos}
\def\!hsetshadelocation{%         ** hmode: xloc=ypos, yloc=xpos
  \!xloc=\!ypos
  \!yloc=\!xpos}

% **************************************
% *** TICKS  (Draws ticks on graphs) ***
% **************************************
% ** User commands
% **   \ticksout
% **   \ticksin
% **   \gridlines
% **   \nogridlines
% **   \loggedticks
% **   \unloggesticks
% ** See Subsection 3.4 of the manual
% ** The following is an option of the \axis command
% **   ticks
% **     [in] [out]
% **     [long] [short] [length <LENGTH>]
% **     [width <WIDTH>]
% **     [andacross] [butnotacross]
% **     [logged] [unlogged]
% **     [unlabeled] [numbered] [withvalues VALUE1 VALUE2 ... VALUEk / ]
% **     [quantity Q] [at LOC1 LOC2 ... LOCk / ] [from LOC1 to LOC2 by
% **       LOC_INCREMENT]
% ** See Subsection 3.2 of the manual for the rules.
% ** The various options of the  tick  field are processed by the
% ** \!nextkeyword  command defined below.
% ** For example, `\!nextkeyword short '  expands to  `\!ticksshort',
% ** while `\!nextkeyword withvalues' expands to `\!tickswithvalues'.
\def\!axisticks {%
  \def\!nextkeyword##1 {%
    \expandafter\ifx\csname !ticks##1\endcsname \relax
      \def\!next{\!fixkeyword{##1}}%
    \else
      \def\!next{\csname !ticks##1\endcsname}%
    \fi
    \!next}%
  \!axissetup
    \def\!axissetup{\relax}%
  \edef\!ticksinoutsign{\!ticksinoutSign}%
  \!ticklength=\longticklength
  \!tickwidth=\linethickness
  \!gridlinestatus
  \!setticktransform
  \!maketick
  \!tickcase=0
  \def\!LTlist{}%
  \!nextkeyword}
\def\ticksout{%
  \def\!ticksinoutSign{+}}
\def\ticksin{%
  \def\!ticksinoutSign{-}}
\ticksout
\def\gridlines{%
  \def\!gridlinestatus{\!gridlinestootrue}}
\def\nogridlines{%
  \def\!gridlinestatus{\!gridlinestoofalse}}
\nogridlines
\def\loggedticks{%
  \def\!setticktransform{\let\!ticktransform=\!logten}}
\def\unloggedticks{%
  \def\!setticktransform{\let\!ticktransform=\!donothing}}
\def\!donothing#1#2{\def#2{#1}}
\unloggedticks
% ** \!ticks/ : terminates read of tick options
\expandafter\def\csname !ticks/\endcsname{%
  \!not {\ifx \!LTlist\empty}
    \!placetickvalues
  \fi
  \def\!tickvalueslist{}%
  \def\!LTlist{}%
  \expandafter\csname !axis/\endcsname}
\def\!maketick{%
  \setbox\!boxA=\hbox{%
    \beginpicture
      \!setdimenmode
      \setcoordinatesystem point at {\!zpt} {\!zpt}
      \linethickness=\!tickwidth
      \ifdim\!ticklength>\!zpt
	\putrule from {\!zpt} {\!zpt} to
	  {\!ticksinoutsign\!tickxsign\!ticklength}
	  {\!ticksinoutsign\!tickysign\!ticklength}
      \fi
      \if!gridlinestoo
	\putrule from {\!zpt} {\!zpt} to
	  {-\!tickxsign\!xaxislength} {-\!tickysign\!yaxislength}
      \fi
    \endpicturesave <\!Xsave,\!Ysave>}%
    \wd\!boxA=\!zpt}

\def\!ticksin{%
  \def\!ticksinoutsign{-}%
  \!maketick
  \!nextkeyword}
\def\!ticksout{%
  \def\!ticksinoutsign{+}%
  \!maketick
  \!nextkeyword}
\def\!tickslength<#1> {%
  \!ticklength=#1\relax
  \!maketick
  \!nextkeyword}
\def\!tickslong{%
  \!tickslength<\longticklength> }
\def\!ticksshort{%
  \!tickslength<\shortticklength> }
\def\!tickswidth<#1> {%
  \!tickwidth=#1\relax
  \!maketick
  \!nextkeyword}
\def\!ticksandacross{%
  \!gridlinestootrue
  \!maketick
  \!nextkeyword}
\def\!ticksbutnotacross{%
  \!gridlinestoofalse
  \!maketick
  \!nextkeyword}
\def\!tickslogged{%
  \let\!ticktransform=\!logten
  \!nextkeyword}
\def\!ticksunlogged{%
  \let\!ticktransform=\!donothing
  \!nextkeyword}
\def\!ticksunlabeled{%
  \!tickcase=0
  \!nextkeyword}
\def\!ticksnumbered{%
  \!tickcase=1
  \!nextkeyword}
\def\!tickswithvalues#1/ {%
  \edef\!tickvalueslist{#1! /}%
  \!tickcase=2
  \!nextkeyword}
\def\!ticksquantity#1 {%
  \ifnum #1>1
    \!updatetickoffset
    \!countA=#1\relax
    \advance \!countA -1
    \!ticklocationincr=\!axisLength
      \divide \!ticklocationincr \!countA
    \!ticklocation=\!axisstart
    \loop \!not{\ifdim \!ticklocation>\!axisend}
      \!placetick\!ticklocation
      \ifcase\!tickcase
	  \relax %  Case 0: no labels
	\or
	  \relax %  Case 1: numbered -- not available here
	\or
	  \expandafter\!gettickvaluefrom\!tickvalueslist
	  \edef\!tickfield{{\the\!ticklocation}{\!value}}%
	  \expandafter\!listaddon\expandafter{\!tickfield}\!LTlist%
      \fi
      \advance \!ticklocation \!ticklocationincr
    \repeat
  \fi
  \!nextkeyword}
\def\!ticksat#1 {%
  \!updatetickoffset
  \edef\!Loc{#1}%
  \if /\!Loc
    \def\next{\!nextkeyword}%
  \else
    \!ticksincommon
    \def\next{\!ticksat}%
  \fi
  \next}

\def\!ticksfrom#1 to #2 by #3 {%
  \!updatetickoffset
  \edef\!arg{#3}%
  \expandafter\!separate\!arg\!nil
  \!scalefactor=1
  \expandafter\!countfigures\!arg/
  \edef\!arg{#1}%
  \!scaleup\!arg by\!scalefactor to\!countE
  \edef\!arg{#2}%
  \!scaleup\!arg by\!scalefactor to\!countF
  \edef\!arg{#3}%
  \!scaleup\!arg by\!scalefactor to\!countG
  \loop \!not{\ifnum\!countE>\!countF}
    \ifnum\!scalefactor=1
      \edef\!Loc{\the\!countE}%
    \else
      \!scaledown\!countE by\!scalefactor to\!Loc
    \fi
    \!ticksincommon
    \advance \!countE \!countG
  \repeat
  \!nextkeyword}
\def\!updatetickoffset{%
  \!dimenA=\!ticksinoutsign\!ticklength
  \ifdim \!dimenA>\!offset
    \!offset=\!dimenA
  \fi}
\def\!placetick#1{%
  \if!xswitch
    \!xpos=#1\relax
    \!ypos=\!axisylevel
  \else
    \!xpos=\!axisxlevel
    \!ypos=#1\relax
  \fi
  \advance\!xpos \!Xsave
  \advance\!ypos \!Ysave
  \kern\!xpos\raise\!ypos\copy\!boxA\kern-\!xpos
  \ignorespaces}
\def\!gettickvaluefrom#1 #2 /{%
  \edef\!value{#1}%
  \edef\!tickvalueslist{#2 /}%
  \ifx \!tickvalueslist\!endtickvaluelist
    \!tickcase=0
  \fi}
\def\!endtickvaluelist{! /}
\def\!ticksincommon{%
  \!ticktransform\!Loc\!t
  \!ticklocation=\!t\!!unit
  \advance\!ticklocation -\!!origin
  \!placetick\!ticklocation
  \ifcase\!tickcase
    \relax % Case 0: no labels
  \or %      Case 1: numbered
    \ifdim\!ticklocation<-\!!origin
      \edef\!Loc{$\!Loc$}%
    \fi
    \edef\!tickfield{{\the\!ticklocation}{\!Loc}}%
    \expandafter\!listaddon\expandafter{\!tickfield}\!LTlist%
  \or %      Case 2: labeled
    \expandafter\!gettickvaluefrom\!tickvalueslist
    \edef\!tickfield{{\the\!ticklocation}{\!value}}%
    \expandafter\!listaddon\expandafter{\!tickfield}\!LTlist%
  \fi}
\def\!separate#1\!nil{%
  \!ifnextchar{-}{\!!separate}{\!!!separate}#1\!nil}
\def\!!separate-#1\!nil{%
  \def\!sign{-}%
  \!!!!separate#1..\!nil}
\def\!!!separate#1\!nil{%
  \def\!sign{+}%
  \!!!!separate#1..\!nil}
\def\!!!!separate#1.#2.#3\!nil{%
  \def\!arg{#1}%
  \ifx\!arg\!empty
    \!countA=0
  \else
    \!countA=\!arg
  \fi
  \def\!arg{#2}%
  \ifx\!arg\!empty
    \!countB=0
  \else
    \!countB=\!arg
  \fi}

\def\!countfigures#1{%
  \if #1/%
    \def\!next{\ignorespaces}%
  \else
    \multiply\!scalefactor 10
    \def\!next{\!countfigures}%
  \fi
  \!next}
\def\!scaleup#1by#2to#3{%
  \expandafter\!separate#1\!nil
  \multiply\!countA #2\relax
  \advance\!countA \!countB
  \if -\!sign
    \!countA=-\!countA
  \fi
  #3=\!countA
  \ignorespaces}
\def\!scaledown#1by#2to#3{%
  \!countA=#1\relax%                          ** get original #
  \ifnum \!countA<0 %                         ** take abs value,
    \def\!sign{-}%                            **   remember sign
    \!countA=-\!countA
  \else
    \def\!sign{}%
  \fi
  \!countB=\!countA%                          ** copy |#|
  \divide\!countB #2\relax%                   ** integer part (|#|/sf)
  \!countC=\!countB%                          ** get sf * (|#|/sf)
    \multiply\!countC #2\relax
  \advance \!countA -\!countC%                ** ctA is now remainder
  \edef#3{\!sign\the\!countB.}%               ** +- integerpart.
  \!countC=\!countA %                         ** Tack on proper number
  \ifnum\!countC=0 %                          **   of zeros after .
    \!countC=1
  \fi
  \multiply\!countC 10
  \!loop \ifnum #2>\!countC
    \edef#3{#3\!zero}%
    \multiply\!countC 10
  \repeat
  \edef#3{#3\the\!countA}%                    ** Add on rest of remainder
  \ignorespaces}
\def\!placetickvalues{%
  \advance\!offset \tickstovaluesleading
  \if!xswitch
    \setbox\!boxA=\hbox{%
      \def\\##1##2{%
	\!dimenput {##2} [B] (##1,\!axisylevel)}%
      \beginpicture
	\!LTlist
      \endpicturesave <\!Xsave,\!Ysave>}%
    \!dimenA=\!axisylevel
      \advance\!dimenA -\!Ysave
      \advance\!dimenA \!tickysign\!offset
      \if -\!tickysign
	\advance\!dimenA -\ht\!boxA
      \else
	\advance\!dimenA  \dp\!boxA
      \fi
    \advance\!offset \ht\!boxA
      \advance\!offset \dp\!boxA
    \!dimenput {\box\!boxA} [Bl] <\!Xsave,\!Ysave> (\!zpt,\!dimenA)
  \else
    \setbox\!boxA=\hbox{%
      \def\\##1##2{%
	\!dimenput {##2} [r] (\!axisxlevel,##1)}%
      \beginpicture
	\!LTlist
      \endpicturesave <\!Xsave,\!Ysave>}%
    \!dimenA=\!axisxlevel
      \advance\!dimenA -\!Xsave
      \advance\!dimenA \!tickxsign\!offset
      \if -\!tickxsign
	\advance\!dimenA -\wd\!boxA
      \fi
    \advance\!offset \wd\!boxA
    \!dimenput {\box\!boxA} [Bl] <\!Xsave,\!Ysave> (\!dimenA,\!zpt)
  \fi}

\normalgraphs
\catcode`!=12 %  *****  THIS MUST NEVER BE OMITTED
% \input{postpictex.tex}
% From heath@ncsa.uiuc.edu Tue Sep 29 07:19:27 1992
% This is postpictex.tex  Version 1.1  9/10/87
% To use the PiCTeX macros under LaTeX, you first need to \input the
% file prepictex.tex, then the main corpus of PiCTeX macros (pictex.tex),
% and finally this file.  Do not \input the file latexpicobjs.tex.

\catcode`@=11 \catcode`!=11

% Save meanings of PiCTeX keywords that duplicate LaTeX keywords
\let\!pictexendpicture=\endpicture
\let\!pictexframe=\frame
\let\!pictexlinethickness=\linethickness
\let\!pictexmultiput=\multiput
\let\!pictexput=\put
% Redefine the PiCTeX \beginpicture macro
\def\beginpicture{%
  \setbox\!picbox=\hbox\bgroup%
  \let\endpicture=\!pictexendpicture
  \let\frame=\!pictexframe
  \let\linethickness=\!pictexlinethickness
  \let\multiput=\!pictexmultiput
  \let\put=\!pictexput
  \let\input=\@@input   % \@@input is LaTeX's saved version of TeX's primitive
  \!xleft=\maxdimen
  \!xright=-\maxdimen
  \!ybot=\maxdimen
  \!ytop=-\maxdimen}
% Reestablish LaTeX's meaning of \frame. This makes
% PiCTeX's meaning of \frame available only inside a PiCture.
\let\frame=\!latexframe
% Make PiCTeX's meaning of \frame available everywhere in the
% guise of \pictexframe
\let\pictexframe=\!pictexframe
% Now do the same for \linethickness
\let\linethickness=\!latexlinethickness
\let\pictexlinethickness=\!pictexlinethickness
% Reset LaTeX's default meaning of \\
\let\\=\@normalcr
\catcode`@=12 \catcode`!=12

% \oddsidemargin=.25in
% \evensidemargin=.25in
% \textwidth=6.0in
% \topmargin=0.in
% \textheight=8.5in
% \newcommand{\bmat}{\left[ \begin{array}}
% \newcommand{\emat}{\end{array} \right]}
% \newcommand{\tr}{{\rm tr}}
% \newcommand{\ignore}[1]{}
% \newcommand{\diag}{\mbox{\rm diag }}
% \begin{document}
%\section{Parallel direct methods for sparse matrices}
% \label{secSparse}
\subsection{Cholesky factorization}
\label{subsecCholesky}
In this section we discuss parallel algorithms for solving sparse
systems of linear equations by direct methods.  Paradoxically, sparse
matrix factorization offers additional opportunities for exploiting
parallelism beyond those available with dense matrices, yet it is
usually more difficult to attain good efficiency in the sparse case.
We examine both sides of this paradox: the additional parallelism
induced by sparsity, and the difficulty in achieving high efficiency in
spite of it.  We will see that regularity and locality play a similar
role in determining performance in the sparse case as they do for dense
matrices.

We couch most of our discussion in terms of the Cholesky factorization,
$A = L L^T$, where $A$ is symmetric positive definite (SPD) and $L$ is
lower triangular with positive diagonal entries.  We focus on Cholesky
factorization primarily because this allows us to discuss parallelism
in relative isolation, without the additional complications of pivoting
for numerical stability.  Most of the lessons learned are also
applicable to other matrix factorizations, such as {\it LU\/} and {\it QR}.
We do
not try to give an exhaustive survey of research in this area, which is
currently very active, instead referring the reader to existing
surveys, such as \cite{heathngpeyton91}.
Our main point in the current
discussion is to explain how the sparse case differs from the dense
case, and examine the performance implications of those differences.

We begin by considering the main features of sparse Cholesky
factorization that affect its performance on serial machines.
Algorithm~\ref{alg_ColumnCholeskyFactorization}
gives a standard, column-oriented formulation in which the
Cholesky factor $L$ overwrites the initial matrix $A$, and only the
lower triangle is accessed:

\begin{algorithm}\label{alg_ColumnCholeskyFactorization}
Column Oriented Cholesky factorization
\begin{tabbing}
jnk \= jnk \= jnk \= jnk \= jnk \kill
\> for $j = 1, \; n$ \\
\> \> for $k = 1, \; j-1$ \\
\> \> \> for $i = j, \; n \;\;\;\;\;\; \{ {\rm cmod}(j,k) \}$ \\
\> \> \> \> $a_{ij} = a_{ij} - a_{ik} \cdot a_{jk}$ \\
\> \> $a_{jj} = \sqrt{a_{jj}}$ \\
\> \> for $k = j+1, \; n \;\;\;\;\;\; \{ {\rm cdiv}(j) \}$ \\
\> \> \> $a_{kj} = a_{kj} / a_{jj}$
\end{tabbing}
\end{algorithm}

The outer loop in Algorithm~\ref{alg_ColumnCholeskyFactorization}
is over successive columns of $A$.  The
the current column (indexed by $j$) is modified by a multiple of each
prior column (indexed by $k$); we refer to such an operation as
${\rm cmod}(j,k)$.  The computation performed by the inner loop (indexed by
$i$) is a {\tt saxpy}.  After all its modifications have been
completed, column $j$ is then scaled by the reciprocal of the square
root of its diagonal element; we refer to this operation as ${\rm cdiv}(j)$.
As usual, this is but one of the 3!  ways of ordering the triple-nested
loop that embodies the factorization.

The inner loop in Algorithm~\ref{alg_ColumnCholeskyFactorization}
has no effect, and thus may as well be
skipped, if $a_{jk} = 0$.  For a dense matrix $A$, such an event is too
unlikely to offer significant advantage.  The fundamental difference
with a sparse matrix is that $a_{jk}$ is in fact very often zero, and
computational efficiency demands that we recognize this situation and
take advantage of it.  Another way of expressing this condition is
that column $j$ of the Cholesky factor $L$ does not depend on prior
column $k$ if $\ell_{jk} = 0$, which not only provides a computational
shortcut, but also suggests an additional source of parallelism that
we will explore in detail later.

\subsection{Sparse matrices}
\label{subsecSparseMatrices}

Thus far we have not said what we mean by a `sparse' matrix.  A good
operational definition is that a matrix is sparse if it contains enough
zero entries to be worth taking advantage of them to reduce both the
storage and work required in solving a linear system.  Ideally, we
would like to store and operate on only the nonzero entries of the
matrix, but such a policy is not necessarily a clear win in either
storage or work.  The difficulty is that sparse data structures include
more overhead (to store indices as well as numerical values of nonzero
matrix entries) than the simple arrays used for dense matrices, and
arithmetic operations on the data stored in them usually cannot be
performed as rapidly either (due to indirect addressing of operands).
There is therefore a tradeoff in memory requirements between sparse and
dense representations and a tradeoff in performance between the
algorithms that use them.  For this reason, a practical requirement for
a family of matrices to be `usefully' sparse is that they have only
${\cal O}(n)$ nonzero entries, i.e.\ a (small) constant number of nonzeros
per row or column, independent of the matrix dimension.  For example,
most matrices arising from finite difference or finite element
discretizations of PDEs satisfy this condition.  In addition to the
number of nonzeros, their particular locations, or pattern, in the
matrix also has a major effect on how well sparsity can be exploited.
Sparsity arising from physical problems usually exhibits some
systematic pattern that can be exploited effectively, whereas the same
number of nonzeros located randomly might offer relatively little
advantage.

In Algorithm~\ref{alg_ColumnCholeskyFactorization}, 
the modification of a given column of the matrix by a
prior column not only changes the existing nonzero entries in the
target column, but may also introduce new nonzero entries in the target
column.  Thus, the Cholesky factor $L$ may have additional nonzeros,
called {\em fill}, in locations that were zero in the original matrix
$A$.  In determining the storage requirements and computational work,
these new nonzeros that the matrix gains during the factorization are
equally as important as the nonzeros with which the matrix starts out.

The amount of such fill is dramatically affected by the order in which
the columns of the matrix are processed.  For example, if the first
column of the matrix $A$ is completely dense, then all of the remaining
columns, no matter how sparse they start out, will completely fill in
with nonzeros during the factorization.  On the other hand, if a single
such dense column is permuted (symmetrically) to become the last column
in the matrix, then it will cause no fill at all.  Thus, a critical
part of the solution process for sparse systems is to determine an
ordering for the rows and columns of the input matrix that limits fill
to preserve sparsity.  Unfortunately, finding an ordering that
minimizes fill is a very hard combinatorial problem (NP-complete), but
heuristics are available that do a good job of reducing, if not exactly
minimizing, fill.  These techniques include minimum degree, nested
dissection, and various schemes for reducing the bandwidth or profile
of a matrix (see, e.g., \cite{dufferismanreid86,georgeliu81}).
for details
on these and many other concepts used in sparse matrix computations).

One of the key advantages of SPD matrices is that such a sparsity
preserving ordering can be selected in advance of the numeric
factorization, independent of the particular values of the nonzero
entries: only the pattern of the nonzeros matters, not their numerical
values.  This would not be the case, in general, if we also had to take
into account pivoting for numerical stability, which obviously would
require knowledge of the nonzero values, and would introduce a
potential conflict between preserving sparsity and preserving
stability.  For the SPD case, once the ordering is selected, the
locations of all fill elements in $L$ can be anticipated prior to the
numeric factorization, and thus an efficient static data structure can
be set up in advance to accommodate them (this process is usually
called {\em symbolic factorization}).  This feature also stands in
contrast to general sparse linear systems, which usually require
dynamic data structures to accommodate fill entries as they occur,
since their locations depend on numerical information that becomes
known only as the numeric factorization process unfolds.  Thus, modern
algorithms and software for solving sparse SPD systems include a
symbolic preprocessing phase in which a sparsity-preserving ordering is
computed and a static data structure is set up for storing the entries
of $L$ before any floating point computation takes place.

We introduce some concepts and notation that will be useful in our
subsequent discussion of parallel sparse Cholesky factorization.  An
important tool in understanding the combinatorial aspects of sparse
Cholesky factorization is the notion of the {\em graph\/} of a symmetric
$n \times n$ matrix $A$, which is an undirected graph having $n$
vertices, with an edge between two vertices $i$ and $j$ if the
corresponding entry $a_{ij}$ of the matrix is nonzero.  We denote the
graph of $A$ by $G(A)$.  The structural effect of the factorization
process can then be described by observing that the elimination of a
variable adds fill edges to the corresponding graph so that the
neighbors of the eliminated vertex become a clique (i.e.\ a fully
connected subgraph).  We also define the {\em filled graph}, denoted by
$F(A)$, as having an edge between vertices $i$ and $j$, with $i > j$,
if $\ell_{ij} \neq 0$ in the Cholesky factor $L$ (i.e.\ $F(A)$ is
simply $G(A)$ with all fill edges added).

We use the notation $M_{i*}$ to denote the $i$th row, and $M_{*j}$ to
denote the $j$th column, of a matrix $M$.  For a given sparse matrix
$M$, we define
\[
{\rm Struct}(M_{i*}) = \{ k<i \mid m_{ik} \neq 0 \}
\]
and
\[
{\rm Struct}(M_{*j}) = \{ k>j \mid m_{kj} \neq 0 \}.
\]
In other words, ${\rm Struct}(M_{i*})$ is the sparsity structure of row $i$
of the strict lower triangle of $M$, while ${\rm Struct}(M_{*j})$ is the
sparsity structure of column $j$ of the strict lower triangle of $M$.
For the Cholesky factor $L$, we define the {\em parent\/} function as
follows:
\[
{\rm parent}(j) = \left\{
\begin{array}{ll}
\min{ \{ i \in {\rm Struct}(L_{*j}) \} }, & {\rm if} \;\;
{\rm Struct}(L_{*j}) \neq \emptyset , \\
j & {\rm otherwise}.
\end{array} \right.
\]
Thus, ${\rm parent}(j)$ is the row index of the first offdiagonal nonzero in
column $j$ of $L$, if any, and has the value $j$ otherwise.  Using the
parent function, we define the {\em elimination tree\/} as a graph having
$n$ vertices, with an edge between vertices $i$ and $j$, for $i > j$,
if $i = {\rm parent}(j)$.  If the matrix is irreducible, then the elimination
tree is indeed a single tree with its root at vertex $n$ (otherwise it is
more accurately termed an {\em elimination forest}).  The elimination
tree, which we denote by $T(A)$, is a spanning tree for the filled
graph $F(A)$.  The many uses of the elimination tree in analyzing and
organizing sparse Cholesky factorization are surveyed in \cite{liu90}.
We will illustrate these concepts pictorially in several examples
below.

\subsection{Sparse factorization}
\label{subsecSparseFactorization}

There are three basic types of algorithms for Cholesky factorization,
depending on which of the three indices is placed in the outer loop:
\begin{enumerate}
  \item {\em Row-Cholesky:}
    Taking $i$ in the outer loop, successive rows of $L$ are computed one
    by one, with the inner loops solving a triangular system for each new
    row in terms of the previously computed rows.
  \item {\em Column-Cholesky:}
    Taking $j$ in the outer loop, successive columns of $L$ are computed
    one by one, with the inner loops computing a matrix--vector product that
    gives the effect of previously computed columns on the column currently
    being computed.
  \item {\em Submatrix-Cholesky:}
    Taking $k$ in the outer loop, successive columns of $L$ are computed
    one by one, with the inner loops applying the current column as a
    rank-1 update to the remaining unreduced submatrix.
\end{enumerate}

These three families of algorithms have markedly different memory
reference patterns in terms of which parts of the matrix are accessed
and modified at each stage of the factorization, as illustrated in
Figure~\ref{cholesky_fig}, and each has its advantages and
disadvantages in a given context.

% \hspace*{-64pt}\beginpicture
\begin{figure}

\beginpicture

\setcoordinatesystem
\setplotarea x from 0 to 350, y from 0 to 140
\setplotsymbol ({\rule{.4pt}{.4pt}})
\setshadesymbol ({\rule{.4pt}{.4pt}})
\setlinear
%\putrectangle corners at 0 0 and 275 140

%%%%%%%%%%%%%%
% row-Cholesky
%%%%%%%%%%%%%%

\put {\beginpicture

%        lower trapezoid -- no shading

	 \plot  0 0   0 30   45 30   75 0   0 0  /

%        row j -- dense shading

	 \plot  0 30   0 34   41 34   45 30   0 30  /
	 \setshadegrid span <.7pt>
	 \vshade   0 30 34  <,z,,>  41 30 34  <z,,,>  45 30 30 /

%        upper triangle -- sparse shading

	 \plot  0 34   0 75   41 34   0 34   /
	 \setshadegrid span <1.4pt>
	 \vshade   0 34 75   41 34 34 /

      \endpicture} [lb] at 75 50

\put {row-Cholesky} at 112.5 40

%%%%%%%%%%%%%%%%%
% column-Cholesky
%%%%%%%%%%%%%%%%%

\put {\beginpicture

%        left rectangle -- sparse shading

	 \plot   0 0   0 34   41 34   41 0   0 0  /
	 \setshadegrid span <1.4pt>
	 \vshade  0 0 34   41 0 34  /

%        upper triangle -- no shading

	 \plot   0 34   41 34   0 75   0 34  /

%        column j -- dense shading

	 \plot   41 0   41 34   45 30   45 0   41 0  /
	 \setshadegrid span <.7pt>
	 \hshade   0 41 45  <,,,z>  30 41 45  <,,z,>  34 41 41 /

%        right-lower triangle -- noshading

	 \plot   45 0   75 0   45 30   45 0  /

      \endpicture} at 175 50

\put {column-Cholesky} at 212.5 40

%%%%%%%%%%%%%%%%%%%%
% submatrix-Cholesky
%%%%%%%%%%%%%%%%%%%%

\put {\beginpicture

%   left trapezpoid -- no shading

    \plot   0 0   0 75   41 34   41 0   0 0  /

%   column j -- dense shading

    \plot   41 0   41 34   45 30   45 0   41 0  /
    \setshadegrid span <1.4pt>
    \hshade   0 41 45  <,,,z>  30 41 45  <,,z,>  34 41 41 /

%   right-lower triangle -- dense shading

    \plot   45 0   75 0   45 30   45 0  /
    \setshadegrid span <.7pt>
    \vshade   44 0 30   75 0 0  /

    \endpicture} at 275 50

\put {submatrix-Cholesky} at 312.5 40

%%%%%%%%
% legend
%%%%%%%%

\put {\beginpicture

    \put {\beginpicture

%            legend box for used columns -- dense shading

	     \plot   0 0   0 12   20 12   20 0   0 0  /
	     \setshadegrid span <.7pt>
	     \vshade   0 0 12   20 0 12  /
	     \put {modified} [lb] at 25 0

	  \endpicture} [lb] at 0 0

    \put {\beginpicture

%            legend box for accessed columns -- sparse shading

	     \plot   0 0   0 12   20 12   20 0   0 0  /
	     \setshadegrid span <1.4pt>
	     \vshade   0 0 12   20 0 12  /
	     \put {used for modification} [lb] at 25 0

	  \endpicture} [lb] at 0 20

      \endpicture} [t] at 225 20

\endpicture

\caption{Three forms of Cholesky factorization.}
\label{cholesky_fig}
\end{figure}

For sparse Cholesky factorization, row-Cholesky is seldom used
for a number of reasons, including the difficulty in providing
a row-oriented data structure that can be accessed efficiently
during the factorization, and the difficulty in vectorizing or
parallelizing the triangular solutions required.  We will therefore
focus our attention on the column-oriented methods, column-Cholesky
and submatrix-Cholesky.  Expressed in terms of the column operations
${\rm cmod}$ and ${\rm cdiv}$ and the ${\rm Struct}$ notation defined earlier, 
sparse column-Cholesky can be stated as follows:

\begin{algorithm}\label{alg_SparseColumnCholesky}
Sparse column-Cholesky factorization
\begin{tabbing}
jnk \= jnk \= jnk \= jnk \= jnk \kill
\> \> for $j = 1, \; n$ \\
\> \> \> for\ $k \in {\rm Struct}(L_{j*})$ \\
\> \> \> \> ${\rm cmod}(j,k)$ \\
\> \> \> ${\rm cdiv}(j)$ \\
\end{tabbing}
\end{algorithm}

In column-Cholesky, a given column $j$ of $A$ remains unchanged until
the outer loop index reaches that value of $j$.  At that point column
$j$ is updated by a nonzero multiple of each column $k < j$ of $L$ for
which $\ell_{jk} \neq 0$.  After all column modifications have been
applied to column $j$, the diagonal entry $\ell_{jj}$ is computed and
used to scale the completely updated column to obtain the remaining
nonzero entries of $L_{*j}$.  Column-Cholesky is sometimes said to be a
`left-looking' algorithm, since at each stage it accesses needed
columns to the left of the current column in the matrix.  It can also
be viewed as a `demand-driven' algorithm, since the inner products
that affect a given column are not accumulated until actually needed to
modify and complete that column.  For this reason, Ortega \cite{ortega88}
terms column-Cholesky a `delayed-update' algorithm.  It is also
sometimes referred to as a `fan-in' algorithm, since the basic
operation is to combine the effects of multiple previous columns on a
single target column.  The column-Cholesky algorithm is the most
commonly used method in commercially available sparse matrix packages.

Similarly, sparse submatrix-Cholesky can be expressed as follows.

\begin{algorithm}\label{alg_SparseSubmatrixCholesky}
Sparse submatrix-Cholesky factorization
\begin{tabbing}
jnk \= jnk \= jnk \= jnk \= jnk \kill
\> \> for $k = 1, \; n$ \\
\> \> \> ${\rm cdiv}(k)$ \\
\> \> \> for\ $j \in {\rm Struct}(L_{*k})$ \\
\> \> \> \> ${\rm cmod}(j,k)$ \\
\end{tabbing}
\end{algorithm}

In submatrix-Cholesky, as soon as column $k$ has been computed, its
effects on all subsequent columns are computed immediately.  Thus,
submatrix-Cholesky is sometimes said to be a `right-looking'
algorithm, since at each stage columns to the right of the current
column are modified.  It can also be viewed as a `data-driven'
algorithm, since each new column is used as soon as it is completed to
make all modifications to all the subsequent columns it affects.
For this reason, Ortega \cite{ortega88} terms submatrix-Cholesky an
`immediate-update' algorithm.  It is also sometimes referred to as a
`fan-out' algorithm, since the basic operation is for a single column
to affect multiple subsequent columns.  We will see that these
characterizations of the column-Cholesky and submatrix-Cholesky
algorithms have important implications for parallel implementations.

We note that many variations and hybrid implementations
that lie somewhere between pure column-Cholesky and pure
submatrix-Cholesky are possible.  Perhaps the most important of these are the
multi-frontal methods (see, e.g., \cite{dufferismanreid86}), in which
updating operations are accumulated in and propagated through a series
of {\em front matrices\/} until finally being incorporated into the
ultimate target columns.  Multifrontal methods have a number of
attractive advantages, most of which accrue from the localization of
memory references in the front matrices, thereby facilitating the
effective use of memory hierarchies, including cache, virtual memory
with paging, or explicit out-of-core solutions (the latter was the
original motivation for these methods \cite{irons70}. 
In addition,
since the front matrices are essentially dense, the operations on them
can be done using optimized kernels, such as the BLAS, to take
advantage of vectorization or any other available architectural
features.  For example, such techniques have been used to attain very
high performance for sparse factorization on conventional vector
supercomputers \cite{ashcraftgrimesetal87}
and on RISC workstations \cite{rothberggupta89}.

\subsection{Parallelism in sparse factorization}
\label{subsecSparseParallelism}

We now examine in greater detail the opportunities for parallelism in
sparse Cholesky factorization and various algorithms for exploiting
it.  One of the most important issues in designing any parallel
algorithm is selecting an appropriate level of {\em granularity}, by
which we mean the size of the computational subtasks that are assigned
to individual processors.  The optimal choice of task size depends on
the tradeoff between communication costs and the load balance across
processors.  We follow Liu \cite{liu86} in identifying three potential
levels of granularity in a parallel implementation of Cholesky
factorization:
\begin{enumerate}
  \item
    {\em fine-grain}, in which each task consists of only one or two
    floating point operations, such as a multiply-add pair,
  \item
    {\em medium-grain}, in which each task is a single column
    operation, such as ${\rm cmod}$ or ${\rm cdiv}$,
  \item
    {\em large-grain}, in which each task is the computation of an
    entire group of columns in a subtree of the elimination tree.
\end{enumerate}

Fine-grain parallelism, at the level of individual floating point
operations, is available in either the dense or sparse case.  It can be
exploited effectively by a vector processing unit or a systolic array,
but would incur far too much communication overhead to be exploited
profitably on most current generation parallel computers.  In
particular, the communication latency of these machines is too great
for such frequent communication of small messages to be feasible.

Medium-grain parallelism, at the level of operations on entire columns,
is also available in either the dense or the sparse case.  This level
of granularity accounts for essentially all of the parallel speedup in
dense factorization on current generation parallel machines, and it is
an extremely important source of parallelism for sparse factorization
as well.  This parallelism is due primarily to the fact that many
${\rm cmod}$ operations can be computed \newline                %new
simultaneously by different                                     %new
processors.  For many problems, such a level of granularity provides a
good balance between communication and computation, but scaling up to
very large problems and/or very large numbers of processors may
necessitate that the tasks be further broken up into chunks based on a
two-dimensional partitioning of the columns.  One must keep in mind,
however, that in the sparse case an entire column operation may require
only a few floating point operations involving the sparsely populated
nonzero elements in the column.  For a matrix of order $n$ having a
planar graph, for example, the largest embedded dense submatrix to be
factored is roughly of order $\sqrt n$, and thus a sparse problem must
be extremely large before a two-dimensional partitioning becomes
essential.

Large-grain parallelism, at the level of subtrees of the elimination
tree, is available only in the sparse case.  If $T_i$ and $T_j$ are
disjoint subtrees of the elimination tree, with neither root node a
descendant of the other, then all of the columns corresponding to nodes
in $T_i$ can be computed completely independently of the columns
corresponding to nodes in $T_j$, and {\em vice versa}, and hence these
computations can be done simultaneously by separate processors with no
communication between them.  For example, each leaf node of the
elimination tree corresponds to a column of $L$ that depends on no
prior columns, and hence all of the leaf node columns can be completed
immediately merely by performing the corresponding ${\rm cdiv}$ operation on
each of them.  Furthermore, all such ${\rm cdiv}$ operations can be performed
simultaneously by separate processors (assuming enough processors are
available).  By contrast, in the dense case all ${\rm cdiv}$ operations must
be performed sequentially (at least at this level of granularity),
since there is never more than one leaf node at any given time.

We see from this discussion that the elimination tree serves to
characterize the parallelism that is unique to sparse factorization.
In particular, the height of the elimination tree gives a rough measure
of the parallel computation time, and the width of the elimination tree
gives a rough measure of the degree or multiplicity of large-grain
parallelism.  These measures are only very rough, however, since the
medium level parallelism also plays a major role in determining overall
performance.  Still, we can see that short, bushy elimination trees are
more advantageous then tall, slender ones in terms of the large-grain
parallelism available.  And just as the fill in the Cholesky factor is
very sensitive to the ordering of the matrix, so is the structure of
the elimination tree.  This suggests that we should choose an ordering
to enhance parallelism, and indeed this is possible (see, e.g.,
\cite{jesskees82,lewispeytonpothen89,liu89})
but such an objective may
conflict to some degree with preservation of sparsity.  Roughly
speaking, sparsity and parallelism are largely compatible, since the
large-grain parallelism is due to sparsity in the first place.
However, these two criteria are by no means coincident, as we will see
by example below.

We now illustrate these concepts using a series of simple examples.
Figure~\ref{tridiagonal_fig} shows a small one-dimensional mesh with
a `natural' ordering of the nodes, the nonzero patterns of the
corresponding tridiagonal matrix $A$ and its Cholesky factor $L$, and
the resulting elimination tree $T(A)$.  On the positive side, the
Cholesky factor suffers no fill at all and the total work required for
the factorization is minimal.  However, we see that the elimination
tree is simply a chain, and therefore there is no large-grain
parallelism available.  Each column of $L$ depends on the immediately
preceding one, and thus they must be computed sequentially.  This
behavior is typical of orderings that minimize the bandwidth of a
sparse matrix:  they tend to inhibit rather than enhance large-grain
parallelism in the factorization.  (As previously discussed in Section
\ref{sec_BandGaussElim}, there is in fact little parallelism of any kind to
be exploited in solving a tridiagonal system in this natural order.
The ${\rm cmod}$ operations involve only a couple of flops each, so that even
the `medium-grain' tasks are actually rather small in this case.)
\begin{figure}
	    \setlength{\unitlength}{9pt} %$%$
	\begin{picture}(40,15)(1.5,0)
	    \setlength{\unitlength}{9pt}
	\put(3,0){
	\begin{picture}(3,15)(0,0)
	\setlength{\unitlength}{9pt} \scriptsize
		\put(1,0){\circle{1}}
		\put(1,2){\circle{1}}
		\put(1,4){\circle{1}}
		\put(1,6){\circle{1}}
		\put(1,8){\circle{1}}
		\put(1,10){\circle{1}}
		\put(1,12){\circle{1}}
		\put(1,0){\makebox(0,0){1}}
		\put(1,2){\makebox(0,0){2}}
		\put(1,4){\makebox(0,0){3}}
		\put(1,6){\makebox(0,0){4}}
		\put(1,8){\makebox(0,0){5}}
		\put(1,10){\makebox(0,0){6}}
		\put(1,12){\makebox(0,0){7}}
		\put(1,0.5){\line(0,1){1}}
		\put(1,2.5){\line(0,1){1}}
		\put(1,4.5){\line(0,1){1}}
		\put(1,6.5){\line(0,1){1}}
		\put(1,8.5){\line(0,1){1}}
		\put(1,10.5){\line(0,1){1}}
		\put(1,14){\makebox(0,0){$G(A)$}}
	\end{picture}
	}
	\put(9,2){
	\begin{picture}(8,12)(0,0)
	    \setlength{\unitlength}{9pt}
	    \put(1,7){\makebox(0,0){$\times$}}
	    \put(1,6){\makebox(0,0){$\times$}}
	    \put(2,7){\makebox(0,0){$\times$}}
	    \put(2,6){\makebox(0,0){$\times$}}
	    \put(2,5){\makebox(0,0){$\times$}}
	    \put(3,6){\makebox(0,0){$\times$}}
	    \put(3,5){\makebox(0,0){$\times$}}
	    \put(3,4){\makebox(0,0){$\times$}}
	    \put(4,5){\makebox(0,0){$\times$}}
	    \put(4,4){\makebox(0,0){$\times$}}
	    \put(4,3){\makebox(0,0){$\times$}}
	    \put(5,4){\makebox(0,0){$\times$}}
	    \put(5,3){\makebox(0,0){$\times$}}
	    \put(5,2){\makebox(0,0){$\times$}}
	    \put(6,3){\makebox(0,0){$\times$}}
	    \put(6,2){\makebox(0,0){$\times$}}
	    \put(6,1){\makebox(0,0){$\times$}}
	    \put(7,2){\makebox(0,0){$\times$}}
	    \put(7,1){\makebox(0,0){$\times$}}
	    \put(4,10){\makebox(0,0){$A$}}
% left bracket
	    \thicklines
	    \put(0,0){\line(0,1){8}}
	    \put(0,0){\line(1,0){0.5}}
	    \put(0,8){\line(1,0){0.5}}
% right bracket
	    \put(8,0){\line(0,1){8}}
	    \put(8,0){\line(-1,0){0.5}}
	    \put(8,8){\line(-1,0){0.5}}
	    \thinlines
	\end{picture}
	}
	\put(22,2){
	\begin{picture}(8,12)(0,0)
	    \setlength{\unitlength}{9pt}
	    \put(1,7){\makebox(0,0){$\times$}}
	    \put(1,6){\makebox(0,0){$\times$}}
	    \put(2,6){\makebox(0,0){$\times$}}
	    \put(2,5){\makebox(0,0){$\times$}}
	    \put(3,5){\makebox(0,0){$\times$}}
	    \put(3,4){\makebox(0,0){$\times$}}
	    \put(4,4){\makebox(0,0){$\times$}}
	    \put(4,3){\makebox(0,0){$\times$}}
	    \put(5,3){\makebox(0,0){$\times$}}
	    \put(5,2){\makebox(0,0){$\times$}}
	    \put(6,2){\makebox(0,0){$\times$}}
	    \put(6,1){\makebox(0,0){$\times$}}
	    \put(7,1){\makebox(0,0){$\times$}}
	    \put(4,10){\makebox(0,0){$L$}}
% left bracket
	    \thicklines
	    \put(0,0){\line(0,1){8}}
	    \put(0,0){\line(1,0){0.5}}
	    \put(0,8){\line(1,0){0.5}}
% right bracket
	    \put(8,0){\line(0,1){8}}
	    \put(8,0){\line(-1,0){0.5}}
	    \put(8,8){\line(-1,0){0.5}}
	    \thinlines
	\end{picture}
	}
	\put(34,0){
	\begin{picture}(8,12)(0,0)
	\setlength{\unitlength}{9pt} \scriptsize
		\put(4,0){\circle{1}}
		\put(4,2){\circle{1}}
		\put(4,4){\circle{1}}
		\put(4,6){\circle{1}}
		\put(4,8){\circle{1}}
		\put(4,10){\circle{1}}
		\put(4,12){\circle{1}}
		\put(4,0){\makebox(0,0){1}}
		\put(4,2){\makebox(0,0){2}}
		\put(4,4){\makebox(0,0){3}}
		\put(4,6){\makebox(0,0){4}}
		\put(4,8){\makebox(0,0){5}}
		\put(4,10){\makebox(0,0){6}}
		\put(4,12){\makebox(0,0){7}}
		\put(4,0.5){\line(0,1){1}}
		\put(4,2.5){\line(0,1){1}}
		\put(4,4.5){\line(0,1){1}}
		\put(4,6.5){\line(0,1){1}}
		\put(4,8.5){\line(0,1){1}}
		\put(4,10.5){\line(0,1){1}}
		\put(4,14){\makebox(0,0){$T(A)$}}
	\end{picture}
	}
	\end{picture}
\caption[]{One-dimensional grid and corresponding
	tridiagonal matrix (left), with Cholesky factor
	and elimination tree (right).}
\label{tridiagonal_fig}
\end{figure}

Figure~\ref{md_fig} shows the same one-dimensional mesh with the nodes
reordered by a minimum degree algorithm.  Minimum degree is the most
effective general purpose heuristic known for limiting fill in sparse
factorization \cite{georgeliu89}. In its simplest form, this algorithm
begins by selecting a node of minimum degree (i.e.\ one having fewest
incident edges) in $G(A)$ and numbering it first.  The selected node is
then deleted and new edges are added, if necessary, to make its former
neighbors into a clique.  The process is then repeated on the updated
graph, and so on, until all nodes have been numbered.  We see in
Figure~\ref{md_fig} that $L$ suffers no fill in the new ordering, and
the elimination tree now shows some large-grain parallelism.  In
particular, columns 1 and 2 can be computed simultaneously, then
columns 3 and 4, and so on.  This twofold parallelism reduces the tree
height (roughly the parallel completion time) by approximately a factor
of two.
\begin{figure}
	    \setlength{\unitlength}{9pt}
	\begin{picture}(40,15)(2.1,0)
	    \setlength{\unitlength}{9pt}
	\put(3,0){
	\begin{picture}(3,15)(0,0)
	\setlength{\unitlength}{9pt} \scriptsize
		\put(1,0){\circle{1}}
		\put(1,2){\circle{1}}
		\put(1,4){\circle{1}}
		\put(1,6){\circle{1}}
		\put(1,8){\circle{1}}
		\put(1,10){\circle{1}}
		\put(1,12){\circle{1}}
		\put(1,0){\makebox(0,0){1}}
		\put(1,2){\makebox(0,0){3}}
		\put(1,4){\makebox(0,0){5}}
		\put(1,6){\makebox(0,0){7}}
		\put(1,8){\makebox(0,0){6}}
		\put(1,10){\makebox(0,0){4}}
		\put(1,12){\makebox(0,0){2}}
		\put(1,0.5){\line(0,1){1}}
		\put(1,2.5){\line(0,1){1}}
		\put(1,4.5){\line(0,1){1}}
		\put(1,6.5){\line(0,1){1}}
		\put(1,8.5){\line(0,1){1}}
		\put(1,10.5){\line(0,1){1}}
		\put(1,14){\makebox(0,0){$G(A)$}}
	\end{picture}
	}
	\put(9,2){
	\begin{picture}(8,12)(0,0)
	    \setlength{\unitlength}{9pt}
	    \put(1,7){\makebox(0,0){$\times$}}
	    \put(1,5){\makebox(0,0){$\times$}}
	    \put(2,6){\makebox(0,0){$\times$}}
	    \put(2,4){\makebox(0,0){$\times$}}
	    \put(3,7){\makebox(0,0){$\times$}}
	    \put(3,5){\makebox(0,0){$\times$}}
	    \put(3,3){\makebox(0,0){$\times$}}
	    \put(4,6){\makebox(0,0){$\times$}}
	    \put(4,4){\makebox(0,0){$\times$}}
	    \put(4,2){\makebox(0,0){$\times$}}
	    \put(5,5){\makebox(0,0){$\times$}}
	    \put(5,3){\makebox(0,0){$\times$}}
	    \put(5,1){\makebox(0,0){$\times$}}
	    \put(6,4){\makebox(0,0){$\times$}}
	    \put(6,2){\makebox(0,0){$\times$}}
	    \put(6,1){\makebox(0,0){$\times$}}
	    \put(7,3){\makebox(0,0){$\times$}}
	    \put(7,2){\makebox(0,0){$\times$}}
	    \put(7,1){\makebox(0,0){$\times$}}
	    \put(4,10){\makebox(0,0){$A$}}
% left bracket
	    \thicklines
	    \put(0,0){\line(0,1){8}}
	    \put(0,0){\line(1,0){0.5}}
	    \put(0,8){\line(1,0){0.5}}
% right bracket
	    \put(8,0){\line(0,1){8}}
	    \put(8,0){\line(-1,0){0.5}}
	    \put(8,8){\line(-1,0){0.5}}
	    \thinlines
	\end{picture}
	}
	\put(22,2){
	\begin{picture}(8,12)(0,0)
	    \setlength{\unitlength}{9pt}
	    \put(1,7){\makebox(0,0){$\times$}}
	    \put(1,5){\makebox(0,0){$\times$}}
	    \put(2,6){\makebox(0,0){$\times$}}
	    \put(2,4){\makebox(0,0){$\times$}}
	    \put(3,5){\makebox(0,0){$\times$}}
	    \put(3,3){\makebox(0,0){$\times$}}
	    \put(4,4){\makebox(0,0){$\times$}}
	    \put(4,2){\makebox(0,0){$\times$}}
	    \put(5,3){\makebox(0,0){$\times$}}
	    \put(5,1){\makebox(0,0){$\times$}}
	    \put(6,2){\makebox(0,0){$\times$}}
	    \put(6,1){\makebox(0,0){$\times$}}
	    \put(7,1){\makebox(0,0){$\times$}}
	    \put(4,10){\makebox(0,0){$L$}}
% left bracket
	    \thicklines
	    \put(0,0){\line(0,1){8}}
	    \put(0,0){\line(1,0){0.5}}
	    \put(0,8){\line(1,0){0.5}}
% right bracket
	    \put(8,0){\line(0,1){8}}
	    \put(8,0){\line(-1,0){0.5}}
	    \put(8,8){\line(-1,0){0.5}}
	    \thinlines
	\end{picture}
	}
	\put(34,0){
	\begin{picture}(8,12)(0,0)
	\setlength{\unitlength}{9pt} \scriptsize
		\put(2,2){\circle{1}}
		\put(2,5){\circle{1}}
		\put(2,8){\circle{1}}
		\put(6,2){\circle{1}}
		\put(6,5){\circle{1}}
		\put(6,8){\circle{1}}
		\put(4,11){\circle{1}}
		\put(2,2){\makebox(0,0){1}}
		\put(2,5){\makebox(0,0){3}}
		\put(2,8){\makebox(0,0){5}}
		\put(6,2){\makebox(0,0){2}}
		\put(6,5){\makebox(0,0){4}}
		\put(6,8){\makebox(0,0){6}}
		\put(4,11){\makebox(0,0){7}}
		\put(2,2.5){\line(0,1){2}}
		\put(2,5.5){\line(0,1){2}}
		\put(6,2.5){\line(0,1){2}}
		\put(6,5.5){\line(0,1){2}}
		\put(2.2773,8.4160){\line(2,3){1.4142}}
		\put(5.7227,8.4160){\line(-2,3){1.4142}}
		\put(4,14){\makebox(0,0){$T(A)$}}
	\end{picture}
	}
	\end{picture}
\caption[]{Graph and matrix reordered by minimum degree
	(left), with corresponding Cholesky factor and
	elimination tree (right).}
\label{md_fig}
\end{figure}

At any stage of the minimum degree algorithm, there may be more than
one node with the same minimum degree, and the quality of the ordering
produced may be affected by the tie breaking strategy used.  In the
example of Figure~\ref{md_fig}, we have deliberately broken ties in the
most favorable way (with respect to parallelism); the least favorable
tie breaking would have reproduced the original ordering of
Figure~\ref{tridiagonal_fig}, resulting in no parallelism.  Breaking
ties randomly (which in general is about all one can do) could produce
anything in between these two extremes, yielding an elimination tree
that reveals some large-grain parallelism, but which is taller and less
well balanced than our example in Figure~\ref{md_fig}.  Again, this is
typical of minimum degree orderings.  In view of this property, Liu
\cite{liu89} has developed an interesting strategy for further
reordering of an initial minimum degree ordering that preserves fill
while reducing the height of the elimination tree.

Figure~\ref{nd_fig} shows the same mesh again, this time ordered by
nested dissection, a divide-and-conquer strategy \cite{george73}.
Let $S$ be a set of nodes, called a {\em separator}, whose removal,
along with all edges incident upon nodes in $S$, disconnects $G(A)$
into two remaining subgraphs.  The nodes in each of the two remaining
subgraphs are numbered contiguously and the nodes in the separator $S$
are numbered last.  This procedure is then applied recursively to split
each of the remaining subgraphs, and so on, until all nodes have been
numbered.  If sufficiently small separators can be found, then nested
dissection tends to do a good job of limiting fill, and if the pieces
into which the graph is split are of about the same size, then the
elimination tree tends to be well balanced.  We see in
Figure~\ref{nd_fig} that for our example, with this ordering, the
Cholesky factor $L$ suffers fill in two matrix entries (indicated by
$+$), but the elimination tree now shows a fourfold large-grain
parallelism, and its height has been reduced further.  This behavior is
again typical of nested dissection orderings: they tend to be somewhat
less successful at limiting fill than minimum degree, but their
divide-and-conquer nature tends to identify parallelism more
systematically and produce better balanced elimination trees.
\begin{figure}
	    \setlength{\unitlength}{9pt}
	\begin{picture}(40,15)(1.85,0)
	    \setlength{\unitlength}{9pt}
	\put(2,0){
	\begin{picture}(3,15)(0,0)
	\setlength{\unitlength}{9pt} \scriptsize
		\put(1,0){\circle{1}}
		\put(1,2){\circle{1}}
		\put(1,4){\circle{1}}
		\put(1,6){\circle{1}}
		\put(1,8){\circle{1}}
		\put(1,10){\circle{1}}
		\put(1,12){\circle{1}}
		\put(1,0){\makebox(0,0){1}}
		\put(1,2){\makebox(0,0){3}}
		\put(1,4){\makebox(0,0){2}}
		\put(1,6){\makebox(0,0){7}}
		\put(1,8){\makebox(0,0){5}}
		\put(1,10){\makebox(0,0){6}}
		\put(1,12){\makebox(0,0){4}}
		\put(1,0.5){\line(0,1){1}}
		\put(1,2.5){\line(0,1){1}}
		\put(1,4.5){\line(0,1){1}}
		\put(1,6.5){\line(0,1){1}}
		\put(1,8.5){\line(0,1){1}}
		\put(1,10.5){\line(0,1){1}}
		\put(1,14){\makebox(0,0){$G(A)$}}
	\end{picture}
	}
	\put(8,2){
	\begin{picture}(8,12)(0,0)
	    \setlength{\unitlength}{9pt}
	    \put(1,7){\makebox(0,0){$\times$}}
	    \put(1,5){\makebox(0,0){$\times$}}
	    \put(2,6){\makebox(0,0){$\times$}}
	    \put(2,5){\makebox(0,0){$\times$}}
	    \put(2,1){\makebox(0,0){$\times$}}
	    \put(3,7){\makebox(0,0){$\times$}}
	    \put(3,6){\makebox(0,0){$\times$}}
	    \put(3,5){\makebox(0,0){$\times$}}
	    \put(4,4){\makebox(0,0){$\times$}}
	    \put(4,2){\makebox(0,0){$\times$}}
	    \put(5,3){\makebox(0,0){$\times$}}
	    \put(5,2){\makebox(0,0){$\times$}}
	    \put(5,1){\makebox(0,0){$\times$}}
	    \put(6,4){\makebox(0,0){$\times$}}
	    \put(6,3){\makebox(0,0){$\times$}}
	    \put(6,2){\makebox(0,0){$\times$}}
	    \put(7,6){\makebox(0,0){$\times$}}
	    \put(7,3){\makebox(0,0){$\times$}}
	    \put(7,1){\makebox(0,0){$\times$}}
	    \put(4,10){\makebox(0,0){$A$}}
% left bracket
	    \thicklines
	    \put(0,0){\line(0,1){8}}
	    \put(0,0){\line(1,0){0.5}}
	    \put(0,8){\line(1,0){0.5}}
% right bracket
	    \put(8,0){\line(0,1){8}}
	    \put(8,0){\line(-1,0){0.5}}
	    \put(8,8){\line(-1,0){0.5}}
	    \thinlines
	\end{picture}
	}
	\put(21,2){
	\begin{picture}(8,12)(0,0)
	    \setlength{\unitlength}{9pt}
	    \put(1,7){\makebox(0,0){$\times$}}
	    \put(1,5){\makebox(0,0){$\times$}}
	    \put(2,6){\makebox(0,0){$\times$}}
	    \put(2,5){\makebox(0,0){$\times$}}
	    \put(2,1){\makebox(0,0){$\times$}}
	    \put(3,5){\makebox(0,0){$\times$}}
	    \put(4,4){\makebox(0,0){$\times$}}
	    \put(4,2){\makebox(0,0){$\times$}}
	    \put(5,3){\makebox(0,0){$\times$}}
	    \put(5,2){\makebox(0,0){$\times$}}
	    \put(5,1){\makebox(0,0){$\times$}}
	    \put(6,2){\makebox(0,0){$\times$}}
	    \put(7,1){\makebox(0,0){$\times$}}
	    \put(3,1){\makebox(0,0){$+$}}
	    \put(6,1){\makebox(0,0){$+$}}
	    \put(4,10){\makebox(0,0){$L$}}
% left bracket
	    \thicklines
	    \put(0,0){\line(0,1){8}}
	    \put(0,0){\line(1,0){0.5}}
	    \put(0,8){\line(1,0){0.5}}
% right bracket
	    \put(8,0){\line(0,1){8}}
	    \put(8,0){\line(-1,0){0.5}}
	    \put(8,8){\line(-1,0){0.5}}
	    \thinlines
	\end{picture}
	}
	\put(33,0){
	\begin{picture}(8,12)(0,0)
	\setlength{\unitlength}{9pt} \scriptsize
		\put(1,3){\circle{1}}
		\put(3,3){\circle{1}}
		\put(5,3){\circle{1}}
		\put(7,3){\circle{1}}
		\put(2,6){\circle{1}}
		\put(6,6){\circle{1}}
		\put(4,9){\circle{1}}
		\put(1,3){\makebox(0,0){1}}
		\put(3,3){\makebox(0,0){2}}
		\put(5,3){\makebox(0,0){4}}
		\put(7,3){\makebox(0,0){5}}
		\put(2,6){\makebox(0,0){3}}
		\put(6,6){\makebox(0,0){6}}
		\put(4,9){\makebox(0,0){7}}
		\put(1.1581,3.4743){\line(1,3){0.6838}}
		\put(2.8419,3.4743){\line(-1,3){0.6838}}
		\put(5.1581,3.4743){\line(1,3){0.6838}}
		\put(6.8419,3.4743){\line(-1,3){0.6838}}
		\put(2.2773,6.4160){\line(2,3){1.4453}}
		\put(5.7227,6.4160){\line(-2,3){1.4453}}
		\put(4,14){\makebox(0,0){$T(A)$}}
	\end{picture}
	}
	\end{picture}
\caption[]{Graph and matrix reordered by nested dissection
	(left), with corresponding Cholesky factor and
	elimination tree (right).}
\label{nd_fig}
\end{figure}

Finally, Figure~\ref{oered_fig} shows the same problem reordered by
odd--even reduction.  This is not a general purpose strategy for sparse
matrices, but it is often used to enhance parallelism in tridiagonal
and related systems, so we illustrate it for the sake of comparison
with more general purpose methods.  In odd--even reduction (see, e.g.,
\cite{dufferismanreid86}), odd node numbers come before even node
numbers, and then this same renumbering is applied recursively within
each resulting subset, and so on until all nodes are numbered.
Although the resulting nonzero pattern of $A$ looks superficially
different, we can see from the elimination tree that this method is
essentially equivalent to nested dissection for this type of problem.
\begin{figure}
	    \setlength{\unitlength}{9pt}
	\begin{picture}(40,15)(1.85,0)
	    \setlength{\unitlength}{9pt}
	\put(2,0){
	\begin{picture}(3,15)(0,0)
	\setlength{\unitlength}{9pt} \scriptsize
		\put(1,0){\circle{1}}
		\put(1,2){\circle{1}}
		\put(1,4){\circle{1}}
		\put(1,6){\circle{1}}
		\put(1,8){\circle{1}}
		\put(1,10){\circle{1}}
		\put(1,12){\circle{1}}
		\put(1,0){\makebox(0,0){1}}
		\put(1,2){\makebox(0,0){5}}
		\put(1,4){\makebox(0,0){3}}
		\put(1,6){\makebox(0,0){7}}
		\put(1,8){\makebox(0,0){4}}
		\put(1,10){\makebox(0,0){6}}
		\put(1,12){\makebox(0,0){2}}
		\put(1,0.5){\line(0,1){1}}
		\put(1,2.5){\line(0,1){1}}
		\put(1,4.5){\line(0,1){1}}
		\put(1,6.5){\line(0,1){1}}
		\put(1,8.5){\line(0,1){1}}
		\put(1,10.5){\line(0,1){1}}
		\put(1,14){\makebox(0,0){$G(A)$}}
	\end{picture}
	}
	\put(8,2){
	\begin{picture}(8,12)(0,0)
	    \setlength{\unitlength}{9pt}
	    \put(1,7){\makebox(0,0){$\times$}}
	    \put(1,3){\makebox(0,0){$\times$}}
	    \put(2,6){\makebox(0,0){$\times$}}
	    \put(2,2){\makebox(0,0){$\times$}}
	    \put(3,5){\makebox(0,0){$\times$}}
	    \put(3,3){\makebox(0,0){$\times$}}
	    \put(3,1){\makebox(0,0){$\times$}}
	    \put(4,4){\makebox(0,0){$\times$}}
	    \put(4,2){\makebox(0,0){$\times$}}
	    \put(4,1){\makebox(0,0){$\times$}}
	    \put(5,7){\makebox(0,0){$\times$}}
	    \put(5,5){\makebox(0,0){$\times$}}
	    \put(5,3){\makebox(0,0){$\times$}}
	    \put(6,6){\makebox(0,0){$\times$}}
	    \put(6,4){\makebox(0,0){$\times$}}
	    \put(6,2){\makebox(0,0){$\times$}}
	    \put(7,5){\makebox(0,0){$\times$}}
	    \put(7,4){\makebox(0,0){$\times$}}
	    \put(7,1){\makebox(0,0){$\times$}}
	    \put(4,10){\makebox(0,0){$A$}}
% left bracket
	    \thicklines
	    \put(0,0){\line(0,1){8}}
	    \put(0,0){\line(1,0){0.5}}
	    \put(0,8){\line(1,0){0.5}}
% right bracket
	    \put(8,0){\line(0,1){8}}
	    \put(8,0){\line(-1,0){0.5}}
	    \put(8,8){\line(-1,0){0.5}}
	    \thinlines
	\end{picture}
	}
	\put(21,2){
	\begin{picture}(8,12)(0,0)
	    \setlength{\unitlength}{9pt}
	    \put(1,7){\makebox(0,0){$\times$}}
	    \put(1,3){\makebox(0,0){$\times$}}
	    \put(2,6){\makebox(0,0){$\times$}}
	    \put(2,2){\makebox(0,0){$\times$}}
	    \put(3,5){\makebox(0,0){$\times$}}
	    \put(3,3){\makebox(0,0){$\times$}}
	    \put(3,1){\makebox(0,0){$\times$}}
	    \put(4,4){\makebox(0,0){$\times$}}
	    \put(4,2){\makebox(0,0){$\times$}}
	    \put(4,1){\makebox(0,0){$\times$}}
	    \put(5,3){\makebox(0,0){$\times$}}
	    \put(6,2){\makebox(0,0){$\times$}}
	    \put(7,1){\makebox(0,0){$\times$}}
	    \put(5,1){\makebox(0,0){$+$}}
	    \put(6,1){\makebox(0,0){$+$}}
	    \put(4,10){\makebox(0,0){$L$}}
% left bracket
	    \thicklines
	    \put(0,0){\line(0,1){8}}
	    \put(0,0){\line(1,0){0.5}}
	    \put(0,8){\line(1,0){0.5}}
% right bracket
	    \put(8,0){\line(0,1){8}}
	    \put(8,0){\line(-1,0){0.5}}
	    \put(8,8){\line(-1,0){0.5}}
	    \thinlines
	\end{picture}
	}
	\put(33,0){
	\begin{picture}(8,12)(0,0)
	\setlength{\unitlength}{9pt} \scriptsize
		\put(1,3){\circle{1}}
		\put(3,3){\circle{1}}
		\put(5,3){\circle{1}}
		\put(7,3){\circle{1}}
		\put(2,6){\circle{1}}
		\put(6,6){\circle{1}}
		\put(4,9){\circle{1}}
		\put(1,3){\makebox(0,0){1}}
		\put(3,3){\makebox(0,0){3}}
		\put(5,3){\makebox(0,0){2}}
		\put(7,3){\makebox(0,0){4}}
		\put(2,6){\makebox(0,0){5}}
		\put(6,6){\makebox(0,0){6}}
		\put(4,9){\makebox(0,0){7}}
		\put(1.1581,3.4743){\line(1,3){0.6838}}
		\put(2.8419,3.4743){\line(-1,3){0.6838}}
		\put(5.1581,3.4743){\line(1,3){0.6838}}
		\put(6.8419,3.4743){\line(-1,3){0.6838}}
		\put(2.2773,6.4160){\line(2,3){1.4453}}
		\put(5.7227,6.4160){\line(-2,3){1.4453}}
		\put(4,14){\makebox(0,0){$T(A)$}}
	\end{picture}
	}
	\end{picture}
\caption[]{Graph and matrix reordered by odd--even reduction
	(left), with corresponding Cholesky factor and
	elimination tree (right).}
\label{oered_fig}
\end{figure}

\subsection{Parallel algorithms for sparse factorization}
\label{subsecParallelAlgorithms}

Having developed some understanding of the sources of parallelism in
sparse Cholesky factorization, we now consider some algorithms for
exploiting it.  In designing any parallel algorithm, one of the most
important decisions is how tasks are to be assigned to processors.  In
a shared memory parallel architecture, the tasks can easily be assigned
to processors dynamically by maintaining a common pool of tasks from
which available processors claim work to do.  This approach has the
additional advantage of providing automatic load balancing to whatever
degree is permitted by the chosen task granularity.  An implementation
of this approach for parallel sparse factorization is given in
\cite{georgeheathliung86}.

In a distributed memory environment, communication costs often prohibit
dynamic task assignment or load balancing, and thus we seek a static
mapping of tasks to processors.  In the case of column-oriented
factorization algorithms, this amounts to assigning the columns of the
matrix to processors according to some mapping procedure determined in
advance.  Such an assignment could be made using the block or wrap
mappings, or combinations thereof, often used for dense matrices.
However, such simple mappings risk wasting much of the large-grain
parallelism identified by means of the elimination tree, and may also
incur unnecessary communication.  For example, the leaf nodes of the
elimination tree can be processed in parallel if they are assigned to
different processors, but the latter is not necessarily ensured by a
simple block or wrap mapping.

A better approach for sparse factorization is to preserve locality by
assigning subtrees of the elimination tree to contiguous subsets of
neighboring processors.  A good example of this technique is the
`subtree-to-subcube' mapping often used with hypercube multicomputers
\cite{georgeheathliung89}.
Of course, the same idea applies to other
network topologies, such as submeshes of a larger mesh.  We will assume
that some such mapping is used, and we will comment further on its
implications later.  Whatever the mapping, we will denote the processor
containing column $j$ by ${\rm map}[j]$, or, more generally, if $J$ is a set
of column numbers, ${\rm map}[J]$ will denote the set of processors
containing the given columns.

One of the earliest and simplest parallel algorithms for sparse
Cholesky factorization is the following version of submatrix-Cholesky
\cite{georgeheathliung88}.
Algorithm~\ref{alg_DistFanOutSparseCholesky} runs on each
processor, with each responsible for its own subset, $mycols$, of
columns.

\begin{algorithm}\label{alg_DistFanOutSparseCholesky}
Distributed fan-out sparse Cholesky factorization
\vspace{-.1in}
\begin{tabbing}
jnk \= jnk \= jnk \= jnk \= jnk \= jnk \= jnk \kill
\> \> for $j \in mycols$ \\
\> \> \> if $j$ is a leaf node in $T(A)$ \\
\> \> \> \> ${\rm cdiv}(j)$ \\
\> \> \> \> send $L_{*j}$ to processors in ${\rm map}({\rm Struct}(L_{*j}))$ \\
\> \> \> \> $mycols = mycols - \{j\}$ \\
\> \> while $mycols \neq \emptyset$ \\
\> \> \> receive any column of $L$, say $L_{*k}$ \\
\> \> \> for $j \in mycols \cap {\rm Struct}(L_{*k})$ \\
\> \> \> \> ${\rm cmod}(j,k)$ \\
\> \> \> \> if column $j$ requires no more ${\rm cmod}$s \\
\> \> \> \> \> ${\rm cdiv}(j)$ \\
\> \> \> \> \> send $L_{*j}$ to processors in ${\rm map}({\rm Struct}(L_{*j}))$
 \\
\> \> \> \> \> $mycols = mycols - \{j\}$ \\
\end{tabbing}
\end{algorithm}
% \vfill\eject

\noindent
In Algorithm~\ref{alg_DistFanOutSparseCholesky},
any processor that owns a column of $L$ corresponding
to a leaf node of the elimination tree can complete it immediately
merely by performing the necessary ${\rm cdiv}$ operation, since such a
column depends on no prior columns.  The resulting factor columns are
then broadcast (fanned-out) to all other processors that will need them
to update columns that they own.  The remainder of the algorithm is
then driven by the arrival of factor columns, as each processor goes
into a loop in which it receives and applies successive factor columns,
in whatever order they may arrive, to whatever columns remain to be
processed.  When the modifications of a given column have been
completed, then the ${\rm cdiv}$ operation is done, the resulting factor
column is broadcast as before, and the process continues until all
columns of $L$ have been computed.

Algorithm~\ref{alg_DistFanOutSparseCholesky}
potentially exploits both the large-grain parallelism
characterized by concurrent ${\rm cdiv}$s and the medium-grain parallelism
characterized by concurrent ${\rm cmod}$s, but this data-driven approach also
has a number of drawbacks that severely limit its efficiency.
In particular, performing the column updates one at a time by the
receiving processors results in unnecessarily high communication
frequency and volume, and in a relatively inefficient computational
inner loop.  The communication requirements can be reduced by careful
mapping and by aggregating updating information over subtrees (see,
e.g., \cite{georgeliung89,murice92,zmijewski89}) but even with this
improvement, the fan-out algorithm is usually not competitive with
other algorithms presented later.  The shortcomings of the fan-out
algorithm motivated the formulation of the following fan-in algorithm
for sparse factorization, which is a parallel implementation of
column-Cholesky \cite{ashcrafteisenstatliu90}.

\begin{algorithm}\label{alg_DistFanInSparseCholesky}
Distributed fan-in sparse Cholesky factorization
\begin{tabbing}
jnk \= jnk \= jnk \= jnk \= jnk \= jnk \= jnk \kill
\> \> for $j = 1, \; n$ \\
\> \> \> if $j \in mycols$ or $mycols \cap {\rm Struct}(L_{j*}) \neq \emptyset$
    \\
\> \> \> \> $u = 0$ \\
\> \> \> \> for $k \in mycols \cap {\rm Struct}(L_{j*})$ \\
\> \> \> \> \> $u = u + \ell_{jk}*L_{*k} \;\;\;\;\;\;$ \{aggregate column
update s\} \\
\> \> \> \> if $j \in mycols$ \\
\> \> \> \> \> incorporate $u$ into the factor column $j$ \\
\> \> \> \> \> while any aggregated update column for\\
\> \> \> \> \> \> column $j$ remains, receive in $u$ another\\
\> \> \> \> \> \> aggregated update column for column $j$, and \\   %new
\> \> \> \> \> \> incorporate it into the factor column $j$ \\      %new
\> \> \> \> \> ${\rm cdiv}(j)$ \\                                   %new
\> \> \> \> else \\
\> \> \> \> \> send $u$ to processor ${\rm map}[j]$ \\
\end{tabbing}
\end{algorithm}

Algorithm~\ref{alg_DistFanInSparseCholesky}
takes a demand-driven approach: the updates for a given
column $j$ are not computed until needed to complete that column, and
they are computed by the sending processors rather than the receiving
processor.  As a result, all of a given processor's contributions to
the updating of the column in question can be combined into a single
aggregate update column, which is then transmitted in a single message
to the processor containing the target column.  This approach not only
decreases communication frequency and volume, but it also facilitates a
more efficient computational inner loop.  In particular, no
communication is required to complete the columns corresponding to any
subtree that is assigned entirely to a single processor.  Thus, with an
appropriate locality-preserving and load-balanced subtree mapping,
Algorithm~\ref{alg_DistFanInSparseCholesky}
has a perfectly parallel, communication-free initial phase
that is followed by a second phase in which communication takes place
over increasingly larger subsets of processors as the computation
proceeds up the elimination tree, encountering larger subtrees.  This
perfectly parallel phase, which is due entirely to sparsity, tends to
constitute a larger proportion of the overall computation as the size
of the problem grows for a fixed number of processors, and thus the
algorithm enjoys relatively high efficiencies for sufficiently large
problems.

In the fan-out and fan-in factorization algorithms, the necessary
information flow between columns is mediated by factor columns or
aggregate update columns, respectively.  Another alternative is a {\em
multi-frontal\/} method, in which update information is mediated through a
series of front matrices.  In a sense, this represents an intermediate
strategy, since the effect of each factor column is incorporated
immediately into a front matrix, but its eventual incorporation into
the ultimate target column is delayed until until actually needed.  The
principal computational advantage of multi-frontal methods is that the
frontal matrices are treated as dense matrices, and hence updating
operations on them are much more efficient than the corresponding
operations on sparse data structures that require indirect addressing.
Unfortunately, although the updating computations employ simple dense
arrays, the overall management of the front matrices is relatively
complicated.  As a consequence, multi-frontal methods are difficult to
specify succinctly, so we will not attempt to do so here, but note that
multi-frontal methods have been implemented for both shared-memory
(e.g., \cite{bennermontryweigand87,duff86}) and
distributed-memory
(e.g., \cite{gilbertschreiber92,lucasblanktieman87})
parallel
computers, and are among the most effective methods known for sparse
factorization in all types of computational environments.  For a unified
description and comparison of parallel fan-in, fan-out and multi-frontal
methods, see \cite{ashcrafteisenstatliusherman90}.

In this brief section on parallel direct methods for sparse systems, we
have concentrated on numeric Cholesky factorization for SPD matrices.
We have omitted many other aspects of the computation, even for the SPD
case: computing the ordering in parallel, symbolic factorization, and
triangular solution.  More generally, we have omitted any discussion of
{\it LU\/} factorization for general sparse square matrices or {\it QR\/}
factorization
for sparse rectangular matrices.  Instead we have concentrated on
identifying the major features that distinguish parallel sparse
factorization from the dense case and examining the performance
implications of those differences.


\section{Iterative Methods for Linear Systems}\label{sec_IterativeMethods}

In this section we discuss parallel aspects of iterative
methods for solving large linear systems. For
a good mathematical introduction to a class of successful and popular methods,
the so-called Krylov subspace methods, see \cite{frgona91}.
There are many such methods and new ones are frequently proposed.
Fortunately, they share enough properties that to understand how to
implement them in parallel it suffices to examine carefully
just a few.

For the purposes of parallel implementation there are two classes
of methods: those with short recurrences, i.e.\ methods that
maintain only a very limited number of search direction vectors, and those
with long recurrences.
The first class includes
CG (Conjugate Gradients), CR (Conjugate Residuals), Bi-CG, CGS (CG squared),
QMR (Quasi Minimum Residual), \linebreak          %new
GMRES$(m)$ for small $m$ (Generalized Minimum
Residual),
truncated ORTHOMIN (Orthogonal Minimum Residual), Chebychev iteration,
and so on.
We could further distinguish between methods with fixed iteration parameters
and methods with dynamical parameters, but we will not do so; the
effects of this aspect will be clear from our discussion.
As the archetype for this class we will consider CG;
the parallel implementation issues for this method apply to most
other short recurrence methods.
The second class of methods includes
GMRES, GMRES($m$) with larger $m$, ORTHOMIN,
ORTHODIR (Orthogonal Directions), ORTHORES (Orthogonal Residuals),
and EN (Eirola--Nevanlinna's Rank-1 update method).
We consider GMRES in detail, which is a popular method in this class.

This section is organized as follows.
In Section~\ref{secItKernel} we will discuss the
parallel aspects of important computational kernels in iterative schemes. From
the discussions it should be clear how to combine coarse-grained
and fine-grained approaches, for example when implementing a method on
a parallel machine with vector processors. The implementation for such
machines, in particular those with shared memory, is given much attention
in \cite{DDSV}.
In Section~\ref{secCG}, coarse-grained parallel and data-locality issues
of CG will be discussed, while in Section~\ref{secGMRES} the same
will be done for GMRES.

\subsection{Parallelism in the kernels of iterative methods}\label{secItKernel}

The basic time-consuming computational kernels of iterative schemes are
usually:
\begin{enumerate}
\item inner products,
\item vector updates,
\item matrix--vector products, like $Ap_i$ (for some methods also $A^T p_i$),
\item preconditioning (e.g., solve for $w$ in $Kw=r$).
\end{enumerate}
The inner products can be easily parallelized; each processor computes the
inner product of two segments of each vector (local inner products or LIPs).
On distributed memory machines the LIPs have to be sent to other processors
in order to be reduced to the required global inner product. This step
requires communication. For shared memory machines the inner products
can be computed in parallel without difficulty.
If the distributed memory
system supports overlap of communication with computation,
then we seek opportunities in the algorithm to do so. In the
standard formulation of most iterative schemes this is usually a major
problem.
We will come back to this in the next two sections.
Vector updates are trivially parallelizable: each processor updates its
`own' segment.
The matrix--vector products are often easily parallelized on shared memory
machines by splitting the matrix into strips corresponding to the vector
segments. Each processor takes care of the matrix--vector product of one
strip.

For distributed memory machines there may be a problem if each processor
has only a segment of the vector in its memory. Depending on the bandwidth
of the matrix we may need communication for other elements of the vector,
which may lead to communication problems. However, many sparse
matrix problems are related to a network in which only nearby nodes are
connected. In such a case it seems natural to subdivide the network, or
grid, in suitable blocks and to distribute these blocks over the processors.
When computing $Ap_i$ each processor needs at most the values of $p_i$ at
some nodes in neighboring blocks. If the number of connections to these
neighboring blocks is small compared to the number of internal nodes,
then the communication time can be overlapped with computational work.
For more detailed discussions on implementation aspects on distributed
memory systems, see \cite{stur91} and \cite{pomm92}.

Preconditioning is often the most problematic part in a parallel
environment.
Incomplete decompositions of $A$ form a popular class of preconditionings
in the context of solving discretized PDEs. In this case the preconditioner
$K=LU$, where $L$ and $U$ have a sparsity pattern equal or close to the
sparsity pattern
of the corresponding parts of $A$ ($L$ is lower triangular, $U$ is upper
triangular). For details see \cite{golubvanloan}, \cite{mevo77} and
\cite{mevo81}.
Solving $Kw=r$ leads to solving successively $Lz=r$ and $Uw=z$.
These triangular solves lead to recurrence relations that are not easily
parallelized.
We will now discuss a number of approaches to obtain parallelism in the
preconditioning part.

% \vfill\eject
\begin{enumerate}
\item {\em Reordering the computations}. Depending on the structure of
the matrix a {\em frontal approach\/} may lead to successful parallelism. By
inspecting the dependency graph one can select those elements that can
be computed in parallel. For instance, if a second order PDE is discretized
by the usual five-point star over a rectangular grid, then
the triangular solves can be parallelized if the computation is carried
out along diagonals of the grid, instead of the usual lexicographical
order. For vector computers this leads to a vectorizable preconditioner
(see \cite{asgr88,DDSV,vors89a,vors89b}).
For coarse-grained parallelism this approach is insufficient. By a similar
approach more parallelism can be obtained in three-dimensional situations: the
so-called
{\em hyperplane approach\/} 
\cite{scvo89,vors89a,vors89b}.
The disadvantage is that the data need to be redistributed over the
processors, since the grid points,
which correspond to a hyperplane in the grid, are located
quite irregularly in the array. For shared memory machines this also leads
to reduced performance because of indirect addressing. In general one
concludes that the data dependency approach is not adequate for
obtaining a suitable degree of parallelism.
\item {\em Reordering the unknowns}. One may also use a {\em coloring
scheme\/}
for reordering the unknowns, so that unknowns with the same color are not
explicitly coupled. This means that the triangular solves can be parallelized
for each color. Of course, communication is required for couplings between
groups of different colors.
Simple coloring schemes, like red-black ordering for the five-point
discretized
Poisson operator, seem to have a negative effect on the convergence behavior.
Duff and Meurant \cite{dume88} have carried out numerical experiments for
many different orderings, which show that the numbers of iterations may
increase significantly for other than lexicographical ordering. Some modest
degree of parallelism can be obtained, however, with so-called incomplete
twisted factorizations \cite{DDSV,vors87b,vors89a}.
Multi-color schemes with a large number of colors (e.g., 20 to 100) may
lead to little or no degradation in convergence behavior \cite{doi91}.
but also to less parallelism. Moreover, the ratio of computation
to communication may be more unfavorable.
\item {\em Forced parallelism}. Parallelism can also be forced by simply
neglecting couplings to unknowns residing in other processors.
This is like block Jacobi preconditioning, in which the blocks
may be decomposed in incomplete form \cite{seag86}. Again, this
may not always reduce the overall solution time, since the effects of
increased parallelism are more than undone by an increased number of
iteration steps.
In order to reduce this effect, it is suggested in \cite{raro87} to
construct
incomplete decompositions on slightly overlapping domains. This requires
communication similar to that of matrix--vector products.
In \cite{raro87} results are reported on a six-processor shared
memory system (IBM3090), and speedups close to $6$ have been observed.
\end{enumerate}

The problems with parallelism in the preconditioner have led to searches
for other preconditioners. Often simple diagonal scaling is an adequate
preconditioner and this is trivially parallelizable.
For results on a Connection Machine, see \cite{besa89}.
Often this approach
leads to a significant increase in iteration steps.
Still another approach is to use polynomial preconditioning: $w=p_j(A)r$,
i.e.\ $K^{-1}=p_j(A)$, for some suitable $j$th degree polynomial. This
preconditioner can be implemented by forming only matrix--vector products,
which, depending on the structure of $A$, are easier to parallelize
\cite{saad85}.
For $p_j$ one often selects a Chebychev polynomial,
which requires some information on the spectrum of $A$.

Finally we point out the possibility of using the truncated Neumann series
for the approximate inverse of $A$, or parts of $L$ and $U$. Madsen {\em et
al.} \cite{maro76}
discuss approximate inversion of $A$, which from the
implementation point of view is equivalent to polynomial preconditioning.
In \cite{vors82} the use of truncated Neumann series for removing some of
the recurrences in the triangular solves is discussed. This approach leads
to only fine-grained parallelism (vectorization).

\subsection{Parallelism and data locality in preconditioned CG}\label{secCG}

To use CG to solve $Ax=b$, $A$ must be symmetric and positive definite.
In other short recurrence methods, other properties
of $A$ may be required or desirable, but we will not exploit these
properties explicitly here.

Most often, CG is used
in combination with some kind of preconditioning
\cite{frgona91,golubvanloan,hest54}.
This means that the matrix
$A$ is implicitly multiplied by an approximation
$K^{-1}$ of $A^{-1}$. Usually, $K$ is constructed to be an approximation of
$A$,
and so that $Ky=z$ is easier to solve than $Ax=b$. Unfortunately, a
popular class of preconditioners, those based on incomplete factorizations of
$A$,
are hard to parallelize.  We have
discussed some efforts to obtain more parallelism in the preconditioner
in Section~\ref{secItKernel}.
Here we will assume the preconditioner is chosen
such that the time to solve $Ky=z$ in parallel is comparable with the time
to compute $Ap$.
For CG we also require that the preconditioner $K$ be symmetric positive
definite. We exploit this to implement the preconditioner more
efficiently.

The preconditioned CG algorithm is as follows.
%\pagebreak

\begin{algorithm}\label{alg_PrecondConjugateGradients1}
Preconditioned Conjugate Gradients -- variant 1
\begin{tabbing}{ll}
aaa\=bbb\= \kill
		  \> $x_0$= initial guess; $r_0=b-Ax_0;$ \\
		  \> $p_{-1}=0; \beta_{-1}=0;$ \\
		  \> Solve for $w_0$ in $Kw_0=r_0;$ \\
		  \> $\rho_0=(r_0,w_0)$ \\
		  \> for $i=0,1,2,....$  \\
		     \> \> $p_i=w_i+ \beta_{i-1}p_{i-1}$; \\
		     \> \> $q_i=Ap_i;$ \\
%                    \> \> $\alpha_i=\frac{\rho_i}{(p_i,q_i)}$ \\
		     \> \> $\alpha_i={\rho_i}/{(p_i,q_i)}$ \\
		     \> \> $x_{i+1}=x_i+\alpha_i p_i;$ \\
		     \> \> $r_{i+1}=r_i-\alpha_{i} q_i;$ \\
		     \> \> if $x_{i+1}$ accurate enough then quit; \\
		     \> \> Solve for $w_{i+1}$ in $Kw_{i+1}=r_{i+1}$; \\
		     \> \> $\rho_{i+1}=(r_{i+1},w_{i+1})$;  \\
%                    \> \> $\beta_i=\frac{\rho_{i+1}}{\rho_i}$;\\
		     \> \> $\beta_i={\rho_{i+1}}/{\rho_i}$;\\
		  \> end;
\end{tabbing}
\end{algorithm}

If $A$ or $K$ is not very sparse, most work is done in multiplying
$q_i=Ap_i$ or solving $Kw_{i+1}=r_{i+1}$, and this is where
parallelism is most beneficial. It is also completely dependent on
the structures of $A$ and $K$.

Now we consider parallelizing the rest of the algorithm.
Note that updating $x_{i+1}$ and $r_{i+1}$ can only begin after
completing the inner product for $\alpha_i$. Since on a distributed
memory machine communication is needed for the inner product, we
cannot overlap this communication with useful computation.
The same observation applies to updating $p_i$, which can only begin
after completing the inner product for $\beta_{i-1}$.
Apart from computing $Ap_i$ and solving $Kw_{i+1} = r_{i+1}$, we need to
load $7$ vectors for $10$ vector floating point operations. This means that
for this part of the computation only $10/7$ floating point operation can be
carried out per memory reference on average.

Several authors \cite{chge89,meur84,meur84a,vors86}
have attempted to improve this ratio, and to reduce the number
of synchronization points (the points at which computation must wait for
communication). In Algorithm~\ref{alg_PrecondConjugateGradients1}
there are two such synchronization
points, namely the computation of both inner products.
Meurant \cite{meur84a} (see also \cite{saad85}) has proposed a variant in
which there is only one
synchronization point, however at the cost of possibly reduced numerical
stability, and one additional inner product. In this scheme the ratio between
computations and memory references is about $2$.
We show here yet another variant, proposed by Chronopoulos and Gear
\cite{chge89}.
%\pagebreak

\begin{algorithm}\label{alg_PrecondConjugateGradients2}
Preconditioned conjugate gradients -- variant 2
\begin{tabbing}{ll}
aaa\=bbb\= \kill
		  \> $x_0$= initial guess; $r_0=b-Ax_0;$ \\
		  \> $q_{-1}=p_{-1}=0; \beta_{-1}=0;$ \\
		  \> Solve for $w_0$ in $Kw_0=r_0;$ \\
		  \> $s_0=Aw_0$; \\
		  \> $\rho_0=(r_0,w_0); \mu_0=(s_0,w_0);$ \\
		  \> $\alpha_0=\rho_0/\mu_0$; \\
		  \> for $i=0,1,2,....$  \\
		     \> \> $p_i=w_i+ \beta_{i-1}p_{i-1}$; \\
		     \> \> $q_i=s_i+ \beta_{i-1} q_{i-1}$; \\
		     \> \> $x_{i+1}=x_i+\alpha_i p_i;$ \\
		     \> \> $r_{i+1}=r_i-\alpha_{i} q_i;$ \\
		     \> \> if $x_{i+1}$ accurate enough then quit; \\
		     \> \> Solve for $w_{i+1}$ in $Kw_{i+1}=r_{i+1}$; \\
		     \> \> $s_{i+1}=Aw_{i+1}$; \\
		     \> \> $\rho_{i+1}=(r_{i+1},w_{i+1})$;
			   $\mu_{i+1} =(s_{i+1},w_{i+1})$;\\
%                    \> \> $\beta_i=\frac{\rho_{i+1}}{\rho_i}$;\\
		     \> \> $\beta_i={\rho_{i+1}}/{\rho_i}$;\\
%                    \> \> $\alpha_{i+1}=\frac{\rho_{i+1}}{\mu_{i+1}-\rho_{i+1}
%                          \beta_i/\alpha_i}$; \\
		     \> \> $\alpha_{i+1}={\rho_{i+1}}/({\mu_{i+1}-\rho_{i+1}
			   \beta_i/\alpha_i})$; \\
		  \> end;
\end{tabbing}
\end{algorithm}

\noindent
In this scheme all vectors need be loaded only once per pass of the loop,
which leads to improved data locality.
However, the price is $2n$ extra flops per iteration step.
Chronopoulos and Gear \cite{chge89} claim the method is stable,
based on their numerical experiments.
Instead of two synchronization points, as in the standard version of CG,
we have now only one such synchronization point, as the next loop can be
started only when the inner products at the end of the previous loop have been
completed. Another slight advantage is that these inner products can be
computed in parallel.

Chronopoulos and Gear \cite{chge89} propose to improve further the
data locality and parallelism in CG by combining $s$ successive steps.
Their algorithm is based upon the following property of CG. The residual
vectors $r_0, ... ,r_i$ form an orthogonal basis (assuming exact arithmetic)
for the Krylov subspace spanned by $r_0, Ar_0, ..., A^{i-1}r_0$. Given
$r_0$ through $r_j$, the vectors $r_0, r_1, ..., r_j$, $Ar_j, ... ,
A^{i-j-1}r_j$ also
span this subspace. Chronopoulos and Gear propose to combine
$s$ successive steps by generating $r_j, Ar_j, ... , A^{s-1}r_j$ first, and
then
to orthogonalize and update the current solution with
this blockwise extended subspace. Their approach leads to slightly more
flops than $s$ successive steps of standard CG, and also
one additional matrix--vector product every $s$ steps.
The implementation issues for vector register computers and
distributed memory machines are discussed in great detail in
\cite{chro91}.

The main drawback in this approach is potential numerical
instability: depending on the spectrum of $A$, the set
$r_j, ... , A^{s-1}r_j$ may converge to a vector in the direction
of a dominant eigenvector, or in other words, may become dependent for
large values of $s$. The authors claim success in using
this approach without serious stability problems for small values of $s$.
Nevertheless, it seems that $s$-step CG still has a bad reputation
\cite{saad88}
because of these problems.  However,
a similar approach, suggested by Chronopoulos and Kim \cite{chki90}
for other processes such as GMRES, seems to be more promising. Several
authors have pursued this direction, and we will come back to this in
Section~\ref{secGMRES}.

We consider another variant of CG, in which we may
overlap all communication time with useful computations.
This is just a reorganized version of the original CG scheme,
and is therefore precisely as stable. The key trick
is to delay the updating of the solution vector.
Another advantage
over the previous scheme is that no additional operations are required.
We will assume that the preconditioner $K$ can be written as $K=LL^T$.
Furthermore, $L$ has a block structure,
corresponding to the grid blocks, so that any
communication can again be overlapped with computation.

\begin{algorithm}\label{alg_PrecondConjugateGradients3}
Preconditioned conjugate gradients -- variant 3
\begin{center}
\begin{tabbing}{ll}
aaa\=bbb\=ccc\=abcdefghijklmnopqrstuvwxyzabcdefg\= \kill
		  \> $x_{-1}=x_0$= initial guess; $r_0=b-Ax_0;$ \\
		  \> $p_{-1}=0; \beta_{-1}=0; \alpha_{-1}=0;$\\
		  \> $s=L^{-1}r_0$; \\
		  \> $\rho_0=(s,s)$ \\
		  \> for $i=0,1,2,....$  \\                                     %new
		     \> \> $w_i=L^{-T}s;$                \> \> (0) \\
		     \> \> $p_i=w_i+ \beta_{i-1}p_{i-1}$;\> \> (1) \\
		     \> \> $q_i=Ap_i;$                   \> \> (2) \\
		     \> \> $\gamma =(p_i,q_i);$          \> \> (3) \\
		     \> \> $x_i=x_{i-1}+\alpha_{i-1}p_{i-1};$ \> \> (4) \\
%                    \> \> $\alpha_i=\frac{\rho_i}{\gamma}$;   \> \> (5) \\
		     \> \> $\alpha_i={\rho_i}/{\gamma}$;   \> \> (5) \\
		     \> \> $r_{i+1}=r_i-\alpha_{i} q_i;$ \> \> (6) \\
		     \> \> $s=L^{-1}r_{i+1};$            \> \> (7) \\
		     \> \> $\rho_{i+1}=(s,s);$           \> \> (8) \\
		     \> \> if $r_{i+1}$  small enough                          %new
			     then \> \> (9) \\                                   %new
		     \> \> \> $x_{i+1}=x_i+\alpha_ip_i$     \>  \\
		     \> \> \> quit;                         \>  \\
%                    \> \> $w_{i+1}=L^{-T}s$;            \> \> (10) \\
%                    \> \> $\beta_i=\frac{\rho_{i+1}}{\rho_i}$; \\
		     \> \> $\beta_i={\rho_{i+1}}/{\rho_i}$; \\
		  \> end;                                                      %new
\end{tabbing}
\end{center}
\end{algorithm}

Under the assumptions that we have made, CG can be efficiently parallelized
as follows.
\begin{enumerate}
\item All compute intensive operations can be done in parallel.
Only operations (2), (3), (7), (8), (9), and (0) require communication.
We have assumed that the communication in (2), (7), and (0) can be
overlapped with computation.
\item The communication required for the reduction of the inner product in (3)
can be overlapped with the update for $x_i$ in (4),
(which could in fact have been done in the previous iteration step).
\item The reduction of the inner product in (8) can be overlapped with the
computation in (0). Also step (9) usually requires information such
as the norm of the residual, which can be overlapped with (0).
\item Steps (1), (2), and (3) can be combined: the computation of a segment
of $p_i$ can be followed immediately by the computation of a segment
of $q_i$ in (2), and this can be followed by the computation of a part of
the inner product in (3). This saves on load operations for segments of
$p_i$ and $q_i$.
\item Depending on the structure of $L$, the computation of segments of
$r_{i+1}$ in (6) can be followed by operations in (7), which can be
followed by the computation of parts of the inner product in (8), and
the computation of the norm of $r_{i+1}$, required for (9).
\item The computation of $\beta_i$ can be done as soon as the computation
in (8) has been completed. At that moment, the computation for (1)
can be started if the requested parts of $w_{i}$ have been completed
in (0).
\item If no preconditioner is used, then $w_i=r_i$, and steps (7) and (0)
are skipped. Step (8) is replaced by
$\rho_{i+1}=(r_{i+1},r_{i+1})$. Now we need some computation
to overlap the communication for this inner product. To this end, one
might split the computation in (4) in two parts. The
first part would be computed in parallel with (3), and the
second part with $\rho_{i+1}$.
\end{enumerate}

More recent work on removing synchronization points in CG while  %new
retaining numerical stability appears in 
\cite{dazevedoromine,eij92}.

\subsection{Parallelism and data locality in GMRES}\label{secGMRES}

GMRES, proposed by Saad and Schultz \cite{saad}, is a CG-like
method for solving general nonsingular linear systems $Ax=b$.
GMRES minimizes the residual over the Krylov subspace
${\rm span}[r_0,Ar_0, A^2 r_0 , ... , A^i r_0]$, with $r_0=b-Ax_0$.
This requires, as with CG, the construction of an orthogonal
basis of this space. Since we do not require $A$ to be symmetric,
we need long recurrences: each new vector must be
explicitly orthogonalized against all previously generated basis vectors.
In its most common form GMRES orthogonalizes using Modified Gram--Schmidt
\cite{golubvanloan}.
In order to limit memory
requirements (since all basis vectors must be stored), GMRES is
restarted after each cycle of $m$ iteration steps; this is called
GMRES$(m)$.
A slightly simplified version of GMRES$(m)$ with
preconditioning $K$ is as follows (for details, see \cite{sasc85}):

\begin{algorithm}\label{alg_GMRESm}
GMRES($m$)
\begin{center}
\begin{tabbing}{l}
aaa\=bbb\=ccc\=ddd\=abcdefghijklmnopqrstuvw\= \kill
	     \> $x_0 $ is an initial guess; $r=b-Ax_0$;  \\
	     \> for $j=1,2,....$  \\
		\>\> Solve for $w$ in $Kw=r$;  \\
		\>\> $v_1=w/\|w\|_2$;  \\
%               \>\> $s:=\|r\|_2e_1$;  \\
		\>\> for $i=1,2,...,m$  \\
		\>\>\> Solve for $w$ in $Kw=Av_i$;  \\
		\>\>\> for $k=1,...,i$\>\> \mbox{orthogonalization of $w$}  \\
	     \>\>\>\> $h_{k,i}=(w,v_k)$;\> \mbox{against $v$s, by modified}
	     \\
		\>\>\>\> $w=w-h_{k,i}v_k$;\> \mbox{Gram--Schmidt process}  \\
		   \>\>\> end $k$;  \\
		   \>\>\> $h_{i+1,i}=\|w\|_2$; \\
		   \>\>\> $v_{i+1}=w/h_{i+1,i}$;  \\
%            \>\>\> apply $J_1 ,..., J_{i-1}$ on $(h_{1,i},...,h_{i+1,i})$; \\
%    \>\>\> construct $J_i$, acting on $i$--th and $(i+1)$--st component\\
%    \>\>\> of $h_{.,i}$, such that $(i+1)$--st component of $J_ih_{.,i}$ is
%            0;\\
%                   \>\>\> $s:=J_is$;  \\
%    \>\>\> if $s(i+1)$ is small enough then (UPDATE($\tilde{x},i$); quit);  \\
		 \>\> end $i$; \\
%                \>\> UPDATE($\tilde{x},m$);  \\
		 \>\> Compute $x_m$ using the $h_{k,i}$ and $v_i$;\\
		 \>\> $r=b-Ax_m$; \\
		 \>\> if residual $r$ is small enough then quit \\
		 \>\> else ($x_0:=x_m;$); \\
	     \> end $j$;
\end{tabbing}
\end{center}
\end{algorithm}

Another scheme for GMRES, based upon Householder orthogonalization instead
of modified Gram--Schmidt, has been proposed in \cite{walk88}. For some
applications the additional computation required by Householder
orthogonalization
is compensated by improved numerical properties: the better orthogonality
saves iteration steps.
In \cite{vovu91b}
a variant of GMRES is proposed in which the preconditioner itself
may be an iterative process, which may help to increase parallel
efficiency.

Similar to CG and other iterative schemes, the major computations
are matrix--vector computations (with $A$ and $K$), inner products and
vector updates. All of these operations are easily parallelizable,
although on distributed memory machines the inner products in the
orthogonalization act as
synchronization points. In this part of the algorithm, one new vector,
$K^{-1}Av_j$, is orthogonalized against the previously built orthogonal set
$v_1$, $v_2$, ... , $v_j$. In Algorithm~\ref{alg_GMRESm}, 
this is done using
Level 1 BLAS, which may be quite inefficient. To incorporate Level 2 BLAS
we can do either Householder orthogonalization or classical Gram--Schmidt
twice (which mitigates classical Gram--Schmidt's potential instability
\cite{saad88}.  Both approaches
significantly increase the computational work and do not remove
the synchronization and data-locality problems completely. Note that we
cannot, as in CG, overlap the inner product computation with updating
the approximate solution, since in GMRES this updating can be done only
after completing a cycle of $m$ orthogonalization steps.

The obvious way to extract more parallelism and
data locality is to generate a basis
$v_1$, $Av_1$, ..., $A^mv_1$ for the Krylov subspace first,
and to orthogonalize this set afterwards; this is called
$m$-step GMRES($m$) \cite{chki90}. This approach does not
increase the computational work and, in contrast to CG, the numerical
instability due to generating a possibly near-dependent set is not
necessarily a drawback. One reason is that error cannot build up as in
CG, because the method is restarted every $m$ steps.
In any case, the resulting set, after orthogonalization,
is the basis of some subspace, and the residual is then minimized over that
subspace. If, however, one wants to mimic
standard GMRES($m$) as closely as possible,
one could generate a better (more independent) starting set of basis
vectors $v_1$, $y_2=p_1(A)v_1$, ...,
$y_{m+1}=p_m(A)v_1$, where the $p_j$ are suitable degree $j$
polynomials.
Newton polynomials are suggested in \cite{bahu91} and 
and Chebychev polynomials in \cite{stur91}.

After generating a suitable starting set, we still have to
orthogonalize it. In \cite{stur91}
modified Gram--Schmidt is used while avoiding communication times
that cannot be overlapped. We outline this approach, since it
may be of value for other orthogonalization methods.
Given a basis for the Krylov subspace, we orthogonalize by
\begin{center}
\begin{tabbing}{l}
aaa\=bbb\=ccc\=ddd\=abcdefghijklmnopqrstuvw\= \kill
\> for $k=2, ..., m+1$ :\\
\> \> /* orthogonalize $y_k, ..., y_{m+1}$ w.r.t. $v_{k-1}$ */ \\
\> \> for $j=k, ..., m+1$ \\
\> \> \> $y_j=y_j-(y_j,v_{k-1})v_{k-1} $ \\
\> \> $v_k = y_k/\|y_k\|_2$
\end{tabbing}
\end{center}

In order to overlap the communication costs of the inner products,
we split the $j$-loop into two parts. Then for each $k$ we proceed as
follows.
\begin{tabbing}{l}
aaa\=bbb\=ccc\=ddd\=abcdefghijklmnopqrstuvw\= \kill
\>1.\>compute in parallel the local parts of the inner products for\\
\>  \>the first group\\
\>2.\>assemble the local inner products to global inner products\\
\>3.\>compute in parallel the local parts of the inner products for \\
\>  \>the second group\\
\>4.\>update $y_k$; compute the local inner products required for $\|y_k\|_2$\\
\>5.\>assemble the local inner products of the second group to global\\
\>  \>inner products\\
\>6.\>update the vectors $y_{k+1}, ..., y_{m+1}$\\
\>7.\>compute $v_k=y_k/\|y_k\|_2$
\end{tabbing}

 From this scheme it is obvious that if the length of the vector segments per
processor are not too small, in principle all communication time can be
overlapped by useful computations.

For a $150$ processor MEIKO system, configured as a $15 \times 10$ torus,
de Sturler \cite{stur91} reports speedups of about $120$ for typical
discretized PDE systems with $60,000$ unknowns (i.e.\ $400$ unknowns
per processor). For larger systems, the speedup increases to $150$
(or more if more processors are involved) as expected.
Calvetti {\em et al.} \cite{cape91} report results for an implementation of $m$-step
GMRES, using BLAS2 Householder orthogonalization, for a four-processor
IBM 6000 distributed memory system. For larger linear systems, they observed
speedups close to $2.5$.


\section{Error Bounds for Solving $Ax=b$}\label{sec_ErrorBoundsForAx=b}

How accurately can we expect to solve $Ax=b$? In this section we will
outline the answer to this question, and refer to standard textbooks
for details \cite{golubvanloan,watkinstext,MA221}. The answer depends both on
the matrix $A$, and on the algorithm we use. To derive a useful error
bound, the property of the matrix we need to know is its {\em condition
number} $\kappa (A)$, and the property we require of the algorithm is {\em numerical
stability}.

Library software also exists for evaluating all the error bounds described here,
for dense and band matrices.  We list the LAPACK routines for computing these 
error bounds below.

To define the condition number $\kappa (A)$ of $A$, we need to introduce some more
notation. Let $\hat{x}$ denote solution of the linear system
$(A+E) \hat{x} = b$, where $E$ is a small perturbation of $A$.
We want to measure the difference between $\hat{x} = (A+E)^{-1}b$
and $x = A^{-1}b$.
Let $\|x\|_{\infty} \equiv \max_i |x_i|$; this is called the
{\em infinity-norm of $x$}.
We will derive a bound on
\[
\frac{\|\hat{x} - x\|_{\infty}}{\|\hat{x}\|_{\infty}} \; \; .
\]
When this quantity is significantly less than 1, we say that
$\hat{x}$ is a good approximation to $x$. For example, if
it is $10^{-6}$, we say $\hat{x}$ and $x$ agree to six
decimal places (note that only the largest components of
$\hat{x}$ and $x$ agree to six decimal places; the smaller
components may agree to fewer). Supposing that
$X$ is an $n$-by-$n$ matrix, define
\[
\|X\|_{\infty} \equiv \max_{1 \leq i \leq n} 
\sum_{1 \leq j \leq n} |x_{ij}| \; \; .
\]
$\|X\|_{\infty}$ is called the {\em infinity-norm} of the
matrix $X$.
Note that $\|X\|_{\infty}$ is always within a factor of $n$ of
the absolute value of the largest entry of $X$. Using this
notation, we can now define the condition number of a matrix as
\[
\kappa (A) \equiv \|A\|_{\infty} \cdot \|A^{-1}\|_{\infty} \; \; .
\]

Using $\kappa (A)$, we can bound the
difference between $\hat{x} = (A+E)^{-1}b$ and $x = A^{-1}b$
as follows:
\[
\frac{\|x - \hat{x}\|_{\infty}}{\|\hat{x}\|_{\infty}} \leq
\kappa (A) \cdot \frac{\|E\|_{\infty}}{\|A\|_{\infty}} \; \; .
\]
In other words, to guarantee a small error we need
$\kappa (A)$ not to be too large (one can show it is always 
at least 1), and for $\frac{\|E\|_{\infty}}{\|A\|_{\infty}}$
to be small. When $\kappa (A)$ is not large, we say $A$ is
{\em well conditioned}. If $\kappa (A)$ is large, we say
$A$ is {\em ill conditioned}. (The exact definition of these 
terms depends on the context, but their use is widespread.)
$\kappa (A)$ depends only on the matrix $A$,
not the algorithm we use; we will describe cheap ways to
estimate it below. 

The other property we need to guarantee a small error bound,
$\frac{\|E\|_{\infty}}{\|A\|_{\infty}}$ being small, is 
called {\em numerical stability}, and is a property of the
algorithm. A well designed algorithm will be numerically
stable, and a poor algorithm will not be.  More formally,
we say an algorithm to solve $Ax=b$ is numerically stable
if the solution $\hat{x}$ it computes satisfies
$(A+E) \hat{x} = b$ (or equivalently $\hat{x} = (A+E)^{-1}b$),
where $\frac{\|E\|_{\infty}}{\|A\|_{\infty}}$ is close to
the machine precision $\epsilon$. Recall from 
Chapter [@Computer Architecture] that the machine precision
measures the roundoff error in individual floating point
operations. 
In IEEE standard single precision floating point arithmetic, 
which has 23 bits to store the significant bits of a floating 
point number, $\epsilon = 2^{-24} \approx 10^{-7}$.
In IEEE standard double precision floating point arithmetic, 
which has 52 bits to store the significant bits of a floating 
point number, $\epsilon = 2^{-53} \approx 10^{-16}$.

Gaussian elimination with partial pivoting is {\em almost always}
numerically stable, so the error bound one expects from
solving $Ax=b$ this way is
\[
\frac{\|x - \hat{x}\|_{\infty}}{\|\hat{x}\|_{\infty}} \leq
\kappa (A) \cdot O( \epsilon ) \; \; .
\]
The notation $O( \epsilon )$ means something close to
$\epsilon$. In practice, one sees errors as large
as $\kappa (A) \cdot n \epsilon$ or so, where $n$ is the dimension of $A$.
So for example, if $\kappa (A) \approx 10^4$ and
we compute in IEEE standard
single precision, we can expect to get 3 correct decimal
places in the answer, since $\kappa (A) \epsilon \approx 10^{-3}$.

We can only say that Gaussian elimination with partial pivoting is
almost always numerically stable, because matrices $A$ do exist where 
$\frac{\|E\|_{\infty}}{\|A\|_{\infty}}$ is as large as
$2^n \epsilon$. Since $2^n$ grows very quickly with $n$, the error
bound rapidly becomes enormous (and the actual solution also becomes
poor). These examples are found in numerical analysis textbooks
\cite{golubvanloan,watkinstext,MA221}, but
not in practice. 


Rather than formally proving that Gaussian elimination with
partial pivoting is generally stable, let us instead illustrate
how omitting pivoting can destroy stability.  Let us apply 
Gaussian elimination without pivoting to compute the factorization $A=LU$ of
$A = \left[ \begin{array}{cc}.0001&1\\1&1
\end{array} \right]$. To keep the example simple, we will use
3 decimal digit floating point arithmetic. Note that
$\kappa (A) = \|A\|_{\infty} \cdot \|A^{-1}\|_{\infty} \approx 4$,
so $A$ is well conditioned and we should expect to be able to solve $Ax=b$
accurately. We will use the notation
${\rm fl} (a~op~b)$ to mean the
rounded, floating point result of $a~op~b$, where $op$ is one of the operations $+$, $-$,
$\times$ and $/$.

\begin{eqnarray*}
L&=&\left[ \begin{array}{cc}1&0\\ {\rm fl}(1/10^{-4})&1 \end{array} \right] \
, \ \ {\rm fl}(1/10^{-4}) \ {\rm rounds\ to}\ 10^{4} {\rm \ without\ error} \\
U&=&\left[ \begin{array}{cc}10^{-4}&1 \\ & {\rm fl}(1-10^{4} \cdot 1) \end{array}
\right]\ , \ \ {\rm fl}(1-10^{4} \cdot 1) \ {\rm rounds\ to}\ -10^{4} \\
{\rm so}\ LU&=&\left[ \begin{array}{cc}1&0 \\10^{4}&1 \end{array}
\right]
\left[ \begin{array}{cc}10^{-4}&1\\ &-10^{4} \end{array} \right] =
\left[ \begin{array}{cc}10^{-4}&1\\1&0 \end{array} \right]
\end{eqnarray*}

Note that the original $a_{22}$ has been entirely ``lost'' from the computation by subtracting
$10^4$ from it. We would have gotten the same LU factors whether
$a_{22}$ had been 1, 0, $-2$,
or any number such that ${\rm fl}(a_{22} - 10^4) = -10^4$.
Since the algorithm proceeds to work only with $L$ and $U$, it will
get the same answer for all these different $a_{22}$, which correspond to completely different
$A$ and so completely different $x = A^{-1}b$;  there is no way to guarantee an accurate answer.
This is called {\em numerical instability},
since $L$ and $U$ are {\em not} the exact factors of a matrix $A+E$ close to $A$
(another way to say this is that $\|E\|_{\infty} = \|A-LU\|_{\infty}$ is about as 
large as $\|A\|_{\infty}$).

Let us see what happens when we go on to solve $Ax=[1,2]^T$
for $x$ using this $LU$ factorization.  The correct answer is $x \approx [1,1]^T$.
Instead we get the following. Solving $Ly = [1,2]^T$ yields $y_1 = {\rm fl}(1/1) = 1$ and
$y_2 = {\rm fl}(2-10^4 \cdot 1) = -10^4$; note that the value 2 has been
``lost'' by subtracting $10^4$ from it. Solving $U \hat{x}=y$ yields
$\hat{x}_2 = {\rm fl}((-10^4)/(-10^4)) = 1$ and
$\hat{x}_1 = {\rm fl}((1-1)/10^{-4}) = 0$, a completely erroneous solution.

The intuition behind this is that if intermediate quantities in computing the
product $L \cdot U$ are very large compared to $\|A\|_{\infty}$, then this can lead to
loss of accuracy in the factorization of $A$, because large entries of $L$ and $U$ get
added to small entries of $A$, and these small entries of $A$ get rounded away. If the
intermediate quantities in the product $L \cdot U$ were comparable to those of
$A$, however, we would expect a tiny error $E=A-LU$ in the factorization.
This is what pivoting tries to guarantee.

No matter what algorithm is used to solve $Ax=b$, one can compute
$\frac{\|E\|_{\infty}}{\|A\|_{\infty}}$ very cheaply given the
approximate solution $\hat{x}$. Just use the following formula
\[
\frac{\|E\|_{\infty}}{\|A\|_{\infty}} = \frac{\|A\hat{x}-b\|_{\infty}}
{\|A\|_{\infty} \cdot \| \hat{x} \|_{\infty}} \; \; .
\]
In the case of dense matrices, this formula costs just $O(n^2)$ flops 
to evaluate, must less than the $2n^3/3$ required by Gaussian elimination.
With many iterative methods, $A \hat{x} -b$ is available anyway.
To summarize, one can bound the error in the computed solution $\hat{x}$ by
\[
\frac{\|x - \hat{x}\|_{\infty}}{\|\hat{x}\|_{\infty}} \leq
\kappa (A) \cdot \frac{\|A \hat{x} - b \|_{\infty}}{\|A\|_{\infty} \| \hat{x} \|_{\infty}}
\; \; .
\]

It remains to describe how to estimate the condition number 
$\kappa (A) = \| A \|_{\infty} \cdot \| A^{-1} \|_{\infty}$.
$\|A\|_{\infty}$ is easy to compute from its definition, 
so we concentrate on
$\|A^{-1}\|_{\infty}$.
An obvious approach is to compute $A^{-1}$ explicitly, and
then compute its infinity norm. This approach would cost
twice again as much as solving $Ax=b$ in the first place, and
so is not done. Instead, we settle for approximations to
$\|A^{-1}\|_{\infty}$ which can be computed very cheaply
once the factorization $PA=LU$ has been produced by
Gaussian elimination. In fact, one can usually estimate
$\|A^{-1}\|_{\infty}$ to within a factor of two with just
$O(n^2)$ work, given $PA=LU$. Since computing $PA=LU$ costs
$2n^3/3$ using Gaussian elimination, estimating the
condition number is a negligible and worthwhile extra cost.
Note that computing the condition number to within a factor
of two is more than accurate enough for an error bound; indeed,
an order-of-magnitude estimate is enough to say how many
correct decimal places are in the answer. The algorithm for
estimating $\| A^{-1} \|_{\infty}$ is described in
\cite{hager,higham87,higham90a}, and implemented in
LAPACK. The routines which compute error bounds have
the same names as above, but with an 'x' appended. For example,
the routine that solves $Ax=b$ for general, dense $A$ and
computes error bounds is called {\tt sgesvx}.

{\bf Exercise:} Retrieve the LAPACK routine {\tt sgeevx} from netlib
(see section~\ref{sec_Netlib} below on how to do this). Write a
modified version, which we will call {\tt sgesvn}, which does {\em not}
do pivoting. Run both routines on randomly generated test matrices,
and compare the error bounds they compute. Modify the test matrices
to make the (1,1) entry very small. How do the error bounds compare now? 
Note that LAPACK produces other error bounds besides the one described
here.
$\Box$

\section{Using Netlib to Retrieve Software}\label{sec_Netlib}

A large body of numerical software is freely available 24 hours a day
via an electronic service called {\em Netlib}. In addition to 
LAPACK and LINPACK, there are dozens of other libraries, technical
reports on various parallel computers and software, test data, facilities to
automatically translate Fortran programs to C, bibliographies, names and
addresses of scientists and mathematicians, and so on.
One can communicate with Netlib in one of two ways, by email or (much more 
easily) via an X-window interface called {\em Xnetlib}. Using email,
one sends messages of the form 'send subroutine\_name from library\_name'
or 'send index for library\_name' to the address 'netlib@ornl.gov' or
'netlib@research.att.com'. The message will be automatically read and
the corresponding subroutine mailed back. Xnetlib (which can be obtained
and installed by sending the message 'send xnetlib.shar from xnetlib'
to netlib@ornl.gov) is an X-window interface which lets one
point at and click on subroutines, which are then automatically transferred
back into the user's directory. There are also index search features to help
find the appropriate subroutine.

To get started using netlib, send the one-line message 'send index' to
netlib@ornl.gov. A description of the overall library should be sent to
you within minutes (providing all the intervening networks as well as
netlib server are up).

Here is a brief summary of the contents of netlib. See the index netlib
sends you for more details.

\begin{center}
\begin{tabular}{|ll|}
\multicolumn{2}{c}{Contents of Netlib} \\
\hline
Library & Short Description \\ \hline
a & approximation algorithms \\
alliant & set of programs collected from Alliant users \\
amos & special functions by D. Amos.  = toms/644 \\
apollo & set of programs collected from Apollo users \\
benchmark & various benchmark programs and a summary of timings \\
\hline
bib & bibliographies \\
bihar & Bjorstad's biharmonic solver \\
blas & machine constants, vector and matrix * vector BLAS \\
bmp & Brent's multiple precision package \\
c & another "misc" library, for software written in C \\
\hline
cheney&kincaid - programs from the 1985 text \\
confdb & conference database \\
conformal & conformal mapping \\
contin & continuation, limit points \\
c++ & code in the C++ language \\
\hline
dierckx & Spline fitting on various geometries. \\
domino & communication and scheduling of multiple tasks; Univ. Maryland \\
eispack & matrix eigenvalues and vectors \\
elefunt & Cody and Waite's tests for elementary functions \\
errata & corrections to numerical books \\
\hline
f2c & Fortran to C converter \\
fishpack & separable elliptic PDEs; Swarztrauber and Sweet \\
fitpack & Cline's splines under tension \\
fftpack & Swarztrauber's Fourier transforms \\
fmm & software from the book by Forsythe, Malcolm, and Moler \\
\hline
fn & Fullerton's special functions \\
fortran & single-double precision converter, static debugger \\
fp & floating point arithmetic \\
gcv & Generalized Cross Validation \\
gmat & multi-processing Time Line and State Graph tools, Mark Seager \\
\hline
go & "golden oldies"  gaussq, zeroin, lowess, ... \\
graphics & auto color, ray-tracing benchmark \\
harwell & MA28 sparse linear system \\
hence & Heterogeneous Network Computing Environment \\
hompack & nonlinear equations by homotopy method \\
\hline
\end{tabular}
\end{center}

\begin{center}
\begin{tabular}{|ll|}
\multicolumn{2}{c}{Contents of Netlib (continued)} \\
\hline
Library & Short Description \\ \hline
ieeecss & IEEE / Control Systems Society \\
itpack & iterative linear system solution by Young and Kincaid \\
jakef & automatic differentiation of Fortran subroutines \\
jgraph & tool to create Postscript graphs \\
kincaid&cheney - programs from the 1990 text \\
\hline
lapack & solving the most common problems in numerical linear algebra \\
lanczos & Cullum and Willoughby's Lanczos programs \\
lanz & Large Sparse Symmetric Generalized Eigenproblem, Jones and Patrick \\
laso & Scott's Lanczos program for eigenvalues of sparse matrices \\
linpack & Gaussian elimination, QR, SVD  by Dongarra, Bunch, Moler, Stewart \\
\hline
lp & linear programming \\
machines & short descriptions of various computers \\
matlab & software from the MATLAB user's group \\
microscope & Alfeld and Harris' system for discontinuity checking \\
minpack & nonlinear equations and least squares by More, Garbow, Hillstrom \\
\hline
misc & everything else \\
ml & Standard ML of New Jersey (programming language compiler) \\
mpi & message passing interface (draft specifications) \\
na-digest & archive of mailings to NA distribution list \\
napack & numerical algebra programs \\
\hline
news & Grosse's Netlib News column for na-net, SIAM News, SIGNUM Newsletter \\
numeralgo & algorithms from the new journal "Numerical Algorithms" \\
ode & ordinary differential equations \\
odepack & ordinary differential equations from Hindmarsh \\
odrpack & orthogonal distance regression, Boggs Byrd Donaldson Schnabel \\
\hline
opt & optimization \\
p4 & portable programs for parallel processors \\
paragraph & display of algorithms on message-passing multiprocessor \\
paranoia & Kahan's floating point test \\
parmacs	&  parallel programming macros \\
\hline
pascal & another "misc" library, for software written in Pascal \\
pbwg & parallel benchmark working group  \\
pchip & Hermite cubics  Fritsch+Carlson \\
pdes/madpack & a multigrid package, by Craig Douglas \\
picl & portable instrumented communication library for multiprocessors \\
\hline
\end{tabular}
\end{center}

\begin{center}
\begin{tabular}{|ll|}
\multicolumn{2}{c}{Contents of Netlib (continued)} \\
\hline
Library & Short Description \\ \hline
pltmg & Bank's multigrid code;  too large for ordinary mail \\
polyhedra & Hume's database of geometric solids \\
popi & Digital Darkroom image manipulation software (Holzmann) \\
port & the public subset of PORT library \\
posix & draft standards \\
\hline
pppack & subroutines from de Boor's Practical Guide to Splines \\
pvm & parallel virtual machine \\
pvm3 & parallel virtual machine version 3 \\
quadpack & univariate quadrature by Piessens, de Donker, Kahaner \\
research & miscellanea from AT\&T Bell Labs, Computing Science Research Center \\
\hline
sched & environment for portable parallel algorithms in a Fortran setting. \\
sciport & portable version of Cray SCILIB, by McBride and Lamson \\
sequent & software from the Sequent Users Group \\
slap & Seager + Greenbaum, iterative methods for symmetric and unsymmetric \\
slatec & comprehensive mathematical and statistical software package \\
\hline
sparse & Kundert + Sangiovanni-Vincentelli, C sparse linear algebra \\
sparse&blas - BLAS by indirection \\
sparspak & George + Liu, sparse linear algebra core \\
specfun & transportable special functions \\
spin & simulation and validation of communication protocols, G. Holzmann \\
\hline
stringsearch & string matching \\
toeplitz & linear systems in Toeplitz or circulant form by Garbow \\
toms & Collected Algorithms of the ACM \\
typesetting & typesetting macros and preprocessors \\
uncon/data & optimization test problems \\
\hline
vanhuffel & total least squares, partial SVD  by Van Hufell \\
vfftpk & vectorized FFT; variant of fftpack \\
voronoi & Voronoi diagrams and Delaunay triangulations \\
xnetlib & X windows interface to netlib \\
y12m & sparse linear system  (Aarhus)  \\
\hline
\end{tabular}
\end{center}



\section{Quick Reference Guide to the BLAS}\label{sec_BLASRefGuide}
\pagebreak

{\scriptsize
\vspace*{-.75in}
{\bf Level 1 BLAS}
\begin{verbatim}
                   dim scalar vector   vector   scalars              5-element array
SUBROUTINE _ROTG (                                      A, B, C, S )
SUBROUTINE _ROTMG(                              D1, D2, A, B,        PARAM )
SUBROUTINE _ROT  ( N,         X, INCX, Y, INCY,               C, S )
SUBROUTINE _ROTM ( N,         X, INCX, Y, INCY,                      PARAM )
SUBROUTINE _SWAP ( N,         X, INCX, Y, INCY )
SUBROUTINE _SCAL ( N,  ALPHA, X, INCX )
SUBROUTINE _COPY ( N,         X, INCX, Y, INCY )
SUBROUTINE _AXPY ( N,  ALPHA, X, INCX, Y, INCY )
FUNCTION   _DOT  ( N,         X, INCX, Y, INCY )
FUNCTION   _DOTU ( N,         X, INCX, Y, INCY )
FUNCTION   _DOTC ( N,         X, INCX, Y, INCY )
FUNCTION   __DOT ( N,         X, INCX, Y, INCY )
FUNCTION   _NRM2 ( N,         X, INCX )
FUNCTION   _ASUM ( N,         X, INCX )
FUNCTION   I_AMAX( N,         X, INCX )
\end{verbatim}

{\bf Level 2 BLAS}
\begin{verbatim}
        options            dim   b-width scalar matrix  vector   scalar vector
_GEMV (        TRANS,      M, N,         ALPHA, A, LDA, X, INCX, BETA,  Y, INCY )
_GBMV (        TRANS,      M, N, KL, KU, ALPHA, A, LDA, X, INCX, BETA,  Y, INCY )
_HEMV ( UPLO,                 N,         ALPHA, A, LDA, X, INCX, BETA,  Y, INCY )
_HBMV ( UPLO,                 N, K,      ALPHA, A, LDA, X, INCX, BETA,  Y, INCY )
_HPMV ( UPLO,                 N,         ALPHA, AP,     X, INCX, BETA,  Y, INCY )
_SYMV ( UPLO,                 N,         ALPHA, A, LDA, X, INCX, BETA,  Y, INCY )
_SBMV ( UPLO,                 N, K,      ALPHA, A, LDA, X, INCX, BETA,  Y, INCY )
_SPMV ( UPLO,                 N,         ALPHA, AP,     X, INCX, BETA,  Y, INCY )
_TRMV ( UPLO, TRANS, DIAG,    N,                A, LDA, X, INCX )
_TBMV ( UPLO, TRANS, DIAG,    N, K,             A, LDA, X, INCX )
_TPMV ( UPLO, TRANS, DIAG,    N,                AP,     X, INCX )
_TRSV ( UPLO, TRANS, DIAG,    N,                A, LDA, X, INCX )
_TBSV ( UPLO, TRANS, DIAG,    N, K,             A, LDA, X, INCX )
_TPSV ( UPLO, TRANS, DIAG,    N,                AP,     X, INCX )

        options            dim   scalar vector   vector   matrix
_GER  (                    M, N, ALPHA, X, INCX, Y, INCY, A, LDA )
_GERU (                    M, N, ALPHA, X, INCX, Y, INCY, A, LDA )
_GERC (                    M, N, ALPHA, X, INCX, Y, INCY, A, LDA )
_HER  ( UPLO,                 N, ALPHA, X, INCX,          A, LDA )
_HPR  ( UPLO,                 N, ALPHA, X, INCX,          AP )
_HER2 ( UPLO,                 N, ALPHA, X, INCX, Y, INCY, A, LDA )
_HPR2 ( UPLO,                 N, ALPHA, X, INCX, Y, INCY, AP )
_SYR  ( UPLO,                 N, ALPHA, X, INCX,          A, LDA )
_SPR  ( UPLO,                 N, ALPHA, X, INCX,          AP )
_SYR2 ( UPLO,                 N, ALPHA, X, INCX, Y, INCY, A, LDA )
_SPR2 ( UPLO,                 N, ALPHA, X, INCX, Y, INCY, AP )
\end{verbatim}

{\bf Level 3 BLAS}
\begin{verbatim}
        options                          dim      scalar matrix  matrix  scalar matrix
_GEMM (             TRANSA, TRANSB,      M, N, K, ALPHA, A, LDA, B, LDB, BETA,  C, LDC )
_SYMM ( SIDE, UPLO,                      M, N,    ALPHA, A, LDA, B, LDB, BETA,  C, LDC )
_HEMM ( SIDE, UPLO,                      M, N,    ALPHA, A, LDA, B, LDB, BETA,  C, LDC )
_SYRK (       UPLO, TRANS,                  N, K, ALPHA, A, LDA,         BETA,  C, LDC )
_HERK (       UPLO, TRANS,                  N, K, ALPHA, A, LDA,         BETA,  C, LDC )
_SYR2K(       UPLO, TRANS,                  N, K, ALPHA, A, LDA, B, LDB, BETA,  C, LDC )
_HER2K(       UPLO, TRANS,                  N, K, ALPHA, A, LDA, B, LDB, BETA,  C, LDC )
_TRMM ( SIDE, UPLO, TRANSA,        DIAG, M, N,    ALPHA, A, LDA, B, LDB )
_TRSM ( SIDE, UPLO, TRANSA,        DIAG, M, N,    ALPHA, A, LDA, B, LDB )
\end{verbatim}
\pagebreak

\vspace*{-.75in}
\begin{tabular}{l l}
                        & prefixes \\
Generate plane rotation 
   & S, D \\
Generate modified plane rotation
   & S, D \\
Apply plane rotation
   & S, D \\
Apply modified plane rotation
   & S, D \\
$ x \leftrightarrow y $
   & S, D, C, Z \\
$ x \leftarrow \alpha x $
   & S, D, C, Z, CS, ZD \\
$ y \leftarrow x $
   & S, D, C, Z \\
$ y \leftarrow \alpha x + y $
   & S, D, C, Z \\
$ dot \leftarrow x ^ {T} y $
   & S, D, DS \\
$ dot \leftarrow x ^ {T} y $
   & C, Z \\
$ dot \leftarrow x ^ {H} y $
   & C, Z \\
$ dot \leftarrow \alpha + x ^ {T} y $
   & SDS \\
$ nrm2 \leftarrow || x || _ {2} $
   & S, D, SC, DZ \\
$ asum \leftarrow || re( x ) || _ {1}  + || im( x ) || _ {1} $
   & S, D, SC, DZ \\
$ amax \leftarrow 1^{st} k \ni | re( x _ {k} ) |  + | im( x _ {k} ) | $
   & S, D, C, Z \\
\vspace{.33in}
\verb+            + $ = max( | re( x _ {i} ) |  + | im( x _ {i} ) | ) $ \\


$ y \leftarrow \alpha A x + \beta y , y \leftarrow \alpha A ^ {T} x + \beta y , y \leftarrow \alpha A ^{H} x + \beta y , A - m \times n $
   & S, D, C, Z \\
$ y \leftarrow \alpha A x + \beta y , y \leftarrow \alpha A ^ {T} x + \beta y , y \leftarrow \alpha A ^{H} x + \beta y , A - m \times n $
   & S, D, C, Z \\
$ y \leftarrow \alpha A x + \beta y $
   & C, Z \\
$ y \leftarrow \alpha A x + \beta y $
   & C, Z \\
$ y \leftarrow \alpha A x + \beta y $
   & C, Z \\
$ y \leftarrow \alpha A x + \beta y $
   & S, D \\
$ y \leftarrow \alpha A x + \beta y $
   & S, D \\
$ y \leftarrow \alpha A x + \beta y $
   & S, D \\
$ x \leftarrow A x, x \leftarrow A ^{T} x, x \leftarrow A ^ {H} x $
   & S, D, C, Z \\
$ x \leftarrow A x, x \leftarrow A ^{T} x, x \leftarrow A ^ {H} x $
   & S, D, C, Z \\
$ x \leftarrow A x, x \leftarrow A ^{T} x, x \leftarrow A ^ {H} x $
   & S, D, C, Z \\
$ x \leftarrow A ^{-1} x, x \leftarrow A ^{-T} x, x \leftarrow A ^ {-H} x $
   & S, D, C, Z \\
$ x \leftarrow A ^{-1} x, x \leftarrow A ^{-T} x, x \leftarrow A ^ {-H} x $
   & S, D, C, Z \\
\vspace{.25in}
$ x \leftarrow A ^{-1} x, x \leftarrow A ^{-T} x, x \leftarrow A ^ {-H} x $
   & S, D, C, Z \\


$ A \leftarrow \alpha x y ^{T} + A , A - m \times n $
   & S, D \\
$ A \leftarrow \alpha x y ^{T} + A , A - m \times n $
   & C, Z \\
$ A \leftarrow \alpha x y ^{H} + A , A - m \times n $
   & C, Z \\
$ A \leftarrow \alpha x x ^{H} + A $
   & C, Z \\
$ A \leftarrow \alpha x x ^{H} + A $
   & C, Z \\
$ A \leftarrow \alpha x y ^{H} + y ( \alpha x ) ^ {H} + A $
   & C, Z \\
$ A \leftarrow \alpha x y ^{H} + y ( \alpha x ) ^ {H} + A $
   & C, Z \\
$ A \leftarrow \alpha x x ^{T} + A $
   & S, D \\
$ A \leftarrow \alpha x x ^{T} + A $
   & S, D \\
$ A \leftarrow \alpha x y ^{T} + \alpha y x ^ {T} + A $
   & S, D \\
\vspace{.49in}
$ A \leftarrow \alpha x y ^{T} + \alpha y x ^ {T} + A $
   & S, D \\


$ C \leftarrow \alpha op(A)op(B) + \beta C, op(X) = X, X ^{T}, X ^{H}, C - m \times n $
   & S, D, C, Z \\
$ C \leftarrow \alpha AB + \beta C, C \leftarrow \alpha BA + \beta C, C - m \times n, A = A ^{H} $
   & C, Z \\
$ C \leftarrow \alpha AB + \beta C, C \leftarrow \alpha BA + \beta C, C - m \times n, A = A ^{T} $
   & S, D, C, Z \\
$ C \leftarrow \alpha AA ^{H} + \beta C, C \leftarrow \alpha A ^{H} A + \beta C, C - n \times n $
   & C, Z \\
$ C \leftarrow \alpha AA ^{T} + \beta C, C \leftarrow \alpha A ^{T} A + \beta C, C - n \times n $
   & S, D, C, Z \\
$ C \leftarrow \alpha AB ^{H} + \alpha BA ^{H} + \beta C, C \leftarrow \alpha A ^{H} B + \alpha B ^{H} A + \beta C, C - n \times n $
   & C, Z \\
$ C \leftarrow \alpha AB ^{T} + \alpha BA ^{T} + \beta C, C \leftarrow \alpha A ^{T} B + \alpha B ^{T} A + \beta C, C - n \times n $
   & S, D, C, Z \\
$ B \leftarrow \alpha op(A)B, B \leftarrow \alpha B op(A), op(A) = A, A ^{T}, A ^{H}, B - m \times n $
   & S, D, C, Z \\
$ B \leftarrow \alpha op(A ^{-1} )B, B \leftarrow \alpha B op(A ^{-1} ), op(A) = A, A ^{T}, A ^{H}, B - m \times n $
   & S, D, C, Z \\

\end{tabular}
{\setlength{\textwidth}{4.0in}}
\pagebreak

{\bf Notes}

\vspace{.25in}

{\bf Meaning of prefixes}

\vspace{.25in}

\begin{tabular}{l l l}
S - REAL              &  C - COMPLEX \\
D - DOUBLE PRECISION  &  Z - COMPLEX*16    & (this may not be supported \\
                      &                    & by all machines) \\
\end{tabular}

\vspace{.25in}
\noindent

For the Level 2 BLAS a set of extended-precision routines with the prefixes
ES, ED, EC, EZ may also be available.

\vspace{.25in}

{\bf Level 1 BLAS }

\vspace{.25in}
\noindent

In addition to the listed routines there are two further 
extended-precision dot product routines DQDOTI and DQDOTA.

\vspace{.25in}

{\bf Level 2 and Level 3 BLAS }

\vspace{.25in}

Matrix types

\vspace{.25in}

\begin {tabular}{l l l}
GE - GEneral     &  GB - General Band \\
SY - SYmmetric   &  SB - Symmetric Band    &  SP - Symmetric Packed \\
HE - HErmitian   &  HB - Hermitian Band    &  HP - Hermitian Packed \\
TR - TRiangular  &  TB - Triangular Band   &  TP - Triangular Packed \\
\end {tabular}

\vspace{.25in}

Options

\vspace{.25in}
\noindent

The dummy options arguments are declared as CHARACTER*1 and may be passed
as character strings.

\begin{tabular}{l l}
TRAN\_  &  = `{\bf N}o transpose',  `{\bf T}ranspose',  `{\bf C}onjugate transpose' $( X, X ^{T}, X^{H} )$ \\
UPLO  &  = `{\bf U}pper triangular',  `{\bf L}ower triangular'\\
DIAG  &  = `{\bf N}on-unit triangular',  `{\bf U}nit triangular'\\
SIDE  &  = `{\bf L}eft',  `{\bf R}ight' (A or op(A) on the left, or A or op(A)
on the right) \\
\end{tabular}

\vspace{.25in}
\noindent
\begin{flushleft}
For real matrices, TRANS\_ = `T' and TRANS\_ = `C' have the same meaning.\\
For Hermitian matrices, TRANS\_ = `T' is not allowed.\\
For complex symmetric matrices, TRANS\_ = `H' is not allowed.\\
\end{flushleft}
}

\pagebreak
\bibliographystyle{alpha}
% \bibliography{/home/barad-dur/jwd/users/demmel/papers/biblio}
\bibliography{biblio}

\end{document}

 

